Page 1 of 1
Using Commercial-Off-The-Shelf (COTS) products, the process of creating new systems through COTS software integration can be a risky yet unavoidable practice. Many of the problems associated with COTS integration can be attributed to poor initial program planning and tracking. Northrop Grumman Mission Systems recently re-evaluated a completed project in an effort to prove out a new method for accurately estimating the cost of projects involving a high COTS content. The company had recently completed and delivered a transition project from a major legacy system to a new system based heavily on the integration of COTS hardware and software. The system supports the Air Force’s day-to-day mission of providing command and telemetry data via the Air Force Control Network.
After the project’s completion, a post-mortem analysis was performed and an estimate was generated for size, effort, schedule and cost using parametric modeling methods. The results were reasonable, with estimates within 7% of the actuals. Northrop Grumman Mission Systems engineers knew the project stood out for its high COTS content, and thought they could get the model closer. Therefore, they applied a new analysis method, deeply examining the COTS elements of the project and also utilizing a parametric tool’s quick-size function, which aids in estimating COTS integration. This method was remarkably accurate, generating results within 2% of the actuals.
Besides providing the basis for cost estimates, the parametric model provides a framework for all parties involved to consider costs early in the program formulation stage. These results demonstrate that COTS integration efforts can be accurately estimated; they also offer insights into what parameters should be used to track the status, estimates to completion, and future efforts with major dependencies on COTS products. This case study will illustrate the methods Northrop Grumman’s analysts used, as well as describe the steps taken for a post-project calibration using Galorath’s SEER-SEM tool. The resulting methods should substantially improve the cost estimates for projects with a high COTS content.
Northrop Grumman Mission Systems is a leading global integrator of complex, mission-enabling systems and services for the federal agencies engaged in defense and intelligence activities, as well as federal civilian organizations, state and local governments and commercial clients. One of the Northrop Grumman Mission Systems’ recent projects involved building the software system for a ground station to improve launch and control services of military satellites through increased automation and simplified launch operation (Figure 1). Northrop Grumman Mission Systems’ original proposal was based largely on internal research and development (IRAD) efforts that had been undertaken at the company’s own expense.
For future bid and proposal work, Northrop Grumman wanted to know how accurate their COTS estimates were. The cost of the project was estimated using a parametric tool that has become a standard for software cost prediction and modeling at Northrop Grumman Mission Systems. The project was a success; the job was completed on schedule and met the customer’s expectations. The job stood out for its high COTS content, but since the COTS integration costs were not separately identified or estimated on the project, analysts decided to revisit the estimate to determine whether it could have been improved. A new method of estimating COTS integration costs that had been developed recently—but had not yet been used on an actual project—was evaluated.
Revisiting the Estimate
Northrop Grumman Mission Systems estimators began by establishing a one-page outline of the project that included major milestones such as the authorization to proceed—the DD250 form that signifies the completion of the project, technical reviews, incremental delivery schedule and formal test events. The outline also defined the functional and product organization involved in the project including contractors and subcontractors and included all major elements of cost such as products and subsystems, sites, deliveries and major COTS elements by subsystem.
The next step was analyzing the actual costs by mapping the project work breakdown structure (WBS) into the standard Northrop Grumman Mission Systems model, called the Process Capability Database (PCDB). The PCDB is Northrop Grumman Mission Systems’ central repository for historical data and includes work descriptions, labor, subcontractor and other direct cost data. The actual costs were re-summarized using the standard WBS in order to determine the cost of each subsystem. The development effort was divided into hardware and software efforts. Support efforts such as configuration management, testing and quality assurance were also broken out. Time phased actuals were used to establish headcount by milestone, discipline and subsystem.
The next step was generating a detailed count of the actual code used in the satellite ground station. The configuration management system contained 58 different file types, which is not surprising since COTS integration systems typically have lots of file types. The contents of the project were differentiated based on file type extensions into third generation language (3GL) and fourth generation language (4GL) code, scripts, tables and intermediates. Figure 2 shows that the code created for this project was counted using source lines of code (SLOC), lines of executable code as defined by syntax (and excluding comments, blanks, continuation lines and pre-processor lines).
The code was further broken down into the following categories: 1) new: new code developed from scratch; 2) reuse: source code created elsewhere but used with some re-engineering and testing effort; 3) derivative: multiple source files derived from one original with some modification; and 4) non-product: code that was not part of final delivery such as test scripts and stubs. Product development IPT lead engineers were interviewed to determine the characteristics of the development environment and staff in order to establish the input parameters (the original proposal was not provided for their reference).
Estimating COTS Integration Efforts
Summarizing the COTS integration efforts provided a greater challenge. The first step was sizing the COTS software packages. Analysts spoke to the vendors but in many cases they didn’t know or were unwilling to divulge the size of their code. The parametric cost estimating software’s quick-sizing tool proved to be invaluable in these cases (Figure 3). This tool allowed the user to input whatever information is available to estimate COTS size. In some cases engineers made comparisons between the COTS programs used in the project to well-known products in the database. In other cases, they searched the COTS product documentation for well-known function names and inspected the index for known elements. Based on this information, the quick-sizing tool provided size counts that were later adjusted by the proportion of the product that was actually used in the project. The quick-sizing tool was also used to generate estimates of the integration effort based on its database of existing projects.
This highly detailed information on the actual COTS products, along with the revised COTS integration estimates, were used to create a final estimate. This estimate additionally factored in the fact that one portion of the system architecture was forced to restart with a different COTS product 28% into the schedule, and another product was severely constrained by staffing problems. It also took into consideration the fact that in the original proposal, the script and 4GL were assumed to be only 25% as difficult as 3GL coding, so only one in four lines of code was used in the estimate. For the final estimate, the 4GL and script were corrected and all line counts were used. Finally, the quick-size estimating feature was used to estimate the impact of COTS integration function counts and parameters from IPT interviews.
The results from this final estimation model were remarkably good. Seven of the eight products were significantly closer to reality using the modified modeling methods (Figure 4). The overall effort and schedule predicted were within 2% of the program actuals. Based on these results, Northrop Grumman Mission Systems estimators plan to use this same approach to verify the process and productivity profiles of on-going COTS integration programs that are nearing delivery now. The results will be used to better determine intermediate milestones and help quantify the effects of having to drop a COTS component from the project and write custom software instead. Over the long run, for projects with major dependencies on COTS products, these new methods should provide better insight into what parameters should be used to track project status and completion estimates.
Northrop Grumman Mission Systems
Redondo Beach, CA.
El Segundo, CA.