BENCHMARKING IDEA 2

Predict the effectiveness of a general purpose machine using a specially-constructed benchmark program, designed to be representative of a class of applications.

Examples:

• The Gibson mix        (``synthetic benchmarks'')
• Whetstone
• Dhrystone

Problems:

• There's no such thing as an `average' program - typical programs rely heavily on same aspect of performance
• Real programs are bigger than toy benchmarks
• Details of the compiler can severely distort comparisons by optimising parts of the program away
• Heavy reliance on synthetic benchmarks encourages cheating - hacks in the architecture or compiler which improve performance on benchmarks but not real programs
• New architectural techniques can make old attempts to synthesise ``typical'' programs obsolete