The RASSP Digest - Vol. 2, 2nd. Qtr. 1995

RASSP Benchmrk Program: Measuring 4x

by G. A. Shaw

1. Introduction

The primary goals established for the benchmark component of the RASSP program are
  1. Evaluate performance of the RASSP design methodology relative to standard practice
    with emphasis on design cycle time, cost, and quality of products.

  2. Identify weaknesses in the design methodology and supporting tools, and recommend
    corrective actions or improvements wherever possible.
Small design problems, nominally 6 months in duration and 5-10K hours of effort, are utilized as the primary vehicle for observing and assessing the performance of the RASSP process. The first two design problems, or benchmarks, are related to the design of a processor for synthetic aperture radar formation on board an unmanned air vehicle [1].

The 4x improvements sought through RASSP are to be measured relative to DoD contractors’ standard practice at the start of the RASSP program (July, 1993). Therefore, a standard practice baseline must be developed for each benchmark as the basis for evaluating RASSP progress toward the principal goals of reduced design cycle time, reduced cost, and improved quality.

2. Some 4x Measurement Challenges

2.1 Limitations of Statistical Characterization

The principal metrics targeted for improvement by RASSP, time and cost required to produce an embedded signal processor, and the quality of the resulting product, are dependent on a multitude of variables. As evidence of this fact, note that commercial parametric cost estimation tools may require the specification of hundreds of parameters in order to estimate the cost and design cycle time associated with a particular hardware and software development effort.

The success of commercial parametric cost estimation tools provides evidence that, given a stable design process and many trials over which to make measurements, it is possible to develop reliable predictors of future performance. However, the RASSP process is continually evolving, and the users of RASSP are continually learning how to efficiently use the process, so that the development of a statistically significant database for calibrating performance of the overall process is not currently feasible. As a consequence, the most significant challenge for the benchmark activity is to develop quantitative metrics for assessing the performance of the evolving RASSP process over the half-dozen or fewer available benchmarks (trials).

The general approach adopted for the benchmarking is to compare performance of the RASSP process to parametric cost estimates for the same design problem (i.e. benchmark). The parametric cost estimates represent a statistically derived standard practice baseline. The result is a comparison of a single RASSP “trial” to an average of many standard practice “trials.”

2.2 Interdependency of RASSP Objectives

A second challenge stems from the fact that some of the goals, such as reduced cycle time and development cost, might be achieved at the expense of other goals, such as improved product quality. For example, developing a design for thirty-year supportability with built-in test (i.e. improved quality and life cycle cost) will add to the development effort (design cycle time and cost) of a processor with otherwise equivalent functionality. Similarly, provision for model year upgradability and reuse of the application software (reduced life cycle cost) will add to the development time and cost of the first version of a processor.

As a consequence, a simple measure of development time or cost for a processor is not sufficient to demonstrate that RASSP is achieving the desired performance goals because it ignores too many other dimensions of the development process and requirements. Instead, one must compare the time and cost achieved using RASSP to the actual or estimated time and cost achievable using standard practice, assuming the same set of processor requirements and objectives, and the same level of expertise on the part of the Development teams.

2.3 Dependence on Experience of Personnel

The expertise of the benchmark execution team, and in particular the prior experience with similar processor designs, has a significant impact on the cost, schedule, and quality of a processor. The importance of the development team experience on a project is illustrated by applying the PRICE-H parametric cost estimation tool to estimate the development cost of a 6U VME form-factor board for SAR processing. Changing the parameter which defines the amount of prior experience in developing similar boards from “significant” to “none” approximately doubles the cost of the board. This implies that without resorting to any process improvements, it is possible to achieve a 2x reduction in cost merely through the choice of individuals assigned to the development.

Clearly these types of dependencies need to be accounted for by developing a baseline standard practice model which reflects the level of expertise and prior experience represented in the RASSP benchmark execution team.

2.4 Maturity of Technology

Contemporary embedded digital processor designs are almost exclusively comprised of integrated circuits, and it is the encapsulation of complexity within the integrated circuit that enables wide availability and application of sophisticated technology such as programmable DSP chips. However, in the 1960s, if one were to measure the cost-effectiveness of developing a digital design exclusively with integrated circuits, the results would suggest that the technology was not cost effective simply because there were not a sufficient number of existing integrated circuits, design tools, and trained designers.

In the same sense, some of the methodologies being explored in the RASSP program, such as top-down VHDL design with virtual prototyping, are not fully mature. Therefore, in assessing cost-effectiveness of these methodologies, the cost of developing infrastructure, such as simulation libraries, and providing training, should be distinguished from the cost of applying the methodology to a given application.

3. ARPA’s Approach

The creation of a benchmarking component in the RASSP program represents a novel approach to assessing progress in a process-oriented development program. While process measurement and evaluation is not novel, there is no prior history from which to extract methodology or best practices for measuring improvement of a complex, evolving, design process such as RASSP. The approach adopted for benchmarking is driven by the challenges outlined above and a desire to address as many of the principal measurement and assessment goals as possible, subject to the resources available for both the execution and the evaluation of the benchmark. Wherever possible, existing measurement tools and process metrics have been chosen to exploit historical performance data, and proven measurement methodologies.

3.1 Benchmark Evaluation Process

Over the life of the RASSP program, the benchmark process involves concurrent activity in three major areas:
  1. Developing benchmark applications and supporting material such as data, written specifications,
    executable requirements, test benches, and suitable performance and complexity metrics.

  2. Evaluating benchmark execution including on-site meetings and observations,
    clarification and correction of requirements, milestone reviews, and interpretation of metrics.

  3. Reporting, including benchmark evaluation reports, conference papers, and briefings.

3.2 Metrics

The principal RASSP performance metrics, such as design cycle time, are measured for a given benchmark application and compared to an estimate for standard practice to provide an indication of the relative performance of the RASSP process. However, coarse-grain metrics, such as design cycle time, provide little insight into where RASSP is improving or failing relative to standard practice. Therefore, for each principal metric, such as design cycle time, supporting metrics are necessary if insight into the benefits and deficiencies of the underlying process is desired. Table 1 provides a sampling of principal and supporting metrics.

Another use of metrics is to normalize application complexity across benchmarks. For a given benchmark, complexity is used to estimate the cost and design cycle time for a standard practice approach using parametric cost modeling. The observed cost and schedule required to complete a benchmark using the RASSP methodology is compared to the estimate of standard practice cost and schedule. Complexity metrics are also needed to assess the improvements in RASSP over successive benchmarks. In order to compare the cost and design cycle over successive benchmarks, normalized complexity numbers, such as source lines of code per person month (SLOC/PM), are used.

Table 1. Principal and Supporting Metrics


3.3 Standard Practice Model

As noted earlier, the principal RASSP metrics, such as design cycle time, depend on a large number of dependent and independent variables, and the scope of the benchmarking effort does not provide for the development of a complicated, parameterized model from first principles.

Three options were therefore considered for the current practice model:
  1. A model based on average cycle times at each phase of the design could be developed for a representative application. However, an average model would not be capable of reflecting the specific conditions of a given benchmark, for example the level of experience of the staff, or the degree of reuse. Another problem with an average model is that it would not necessarily scale to the six-month duration of a benchmark.

  2. A similar application might be identified and the actual cost and schedule compared against the performance on the RASSP benchmark. Several problems arise with this approach including locating a similar application, and accounting for the differences in personnel and technology at the time the similar application was developed.

  3. A detailed parametric cost model could be developed for each benchmark. The model would be specialized to account for the specifics of the benchmark such as the number and experience of personnel, the complexity of the software and hardware, and numerous other variables that affect cost and schedule.
The parametric model approach was selected based on cost-effectiveness, and adaptability to the specific character of each benchmark. Commercially available parametric cost estimation tools provide an estimate of the total cost and development cycle time to develop hardware or software. The parametric estimates rely on input parameters describing the hardware or software technology, the experience of the developers, and the complexity of the problem to develop a cost and development time estimate. An explicit model of the development process is not required, although some tools, such as the PRICE-S and SEER-SEM software estimators, allow mil-standard development processes. Parametric estimators account for the underlying process indirectly by calibration of the estimators through regression on historical data. Thus parametric cost estimation provides a mechanism for developing standard practice estimates for two of the three primary RASSP metrics, namely cost and design cycle time, without requiring an explicit current practice process model.

Parametric cost estimators also include a capability for estimating life cycle cost. Since production quantities of hardware are not planned as part of the benchmarks, and since the duration of the RASSP program is only four years, actual measurement of life cycle performance is not feasible. Estimation methods are therefore the only means assessing life cycle cost and supportability issues.

3.4 Parametric Cost Estimators

A key benchmark task is to measure the cost and design cycle time of the RASSP methodology relative to what would be achievable using industry standard practice at the start of the RASSP program in July of 1993. In order to accomplish this task, multiple, commercial, parametric cost estimation tools are employed for the following reasons:
  1. The fact that a cost estimation tool is commercially available and supported suggests some measure of success has been achieved with the tool in accurately predicting cost and schedule.

  2. Commercial tools incorporate “industry standard” default parameters many of which are updated on an annual basis. Dates in the past or future can be selected to account for the historical or projected impact of technology.

  3. The use of more than one tool to predict cost and schedule helps insure that the estimators are being used correctly.

  4. The widespread availability and use of commercial cost estimation tools makes them a commonly encountered and well understood basis for representing the complexity of a particular application.
The PRICE family of parametric cost estimators, SEER, and REVIC, are all used in developing the standard practice baseline for a given benchmark [2].

4. Status

The first RASSP benchmark, which began in September of 1993, required the development of a VHDL-based virtual prototype for a SAR processor. The second benchmark, which is now underway, requires the Developers to carry the virtual prototype to a physical implementation as a means of assessing the fidelity and value of the virtual prototype model and methodology. At the conclusion of Benchmark-2, the actual time and cost to develop the SAR processor will be compared to an industry standard practice baseline. Expectations are that the effort expended in developing the virtual prototype will lead to significant reductions in the time and cost required to develop the processor hardware.

The VHDL modeling effort on Benchmark-1 has produced a limited database for calibrating parametric cost estimators for VHDL software development. Using lines of code as the principal metric, good agreement has been demonstrated between the parametric cost estimates and the actual VHDL effort. The calibrated parametric cost model can be used to estimate the savings associated with automatic code generation (synthesis of VHDL code), and reuse relative to manual coding of VHDL.

References

[1] B. W. Zuerndorfer and G. A. Shaw, “SAR Processing for RASSP Application,” Proceedings of the First Annual Rassp Conference, August, 1994.

[2] J.C. Anderson, “Predicting the Future with RASSP Benchmarks.” Proceedings of the First Annual RASSP Conference, August, 1994.
The RASSP Digest - Vol. 2, 2nd. Qtr. 1995

95q2/news_5.html