The case against BAPCo

ALLEGATIONS AMD has made against benchmark firm BAPCo over its 2002 version of Sysmark and the supposed benefit it gives to the Pentium 4 has caused no small stir in the hardware community. But some sections of the worldwide IT media appear just to have donned blinkers, and said nothing.

If we have the consumer's best interests at heart, then what is a benchmark supposed to represent? Is it what BAPCo claims here, a real world and objective indicator of performance? If this is so, then why has AMD expressed such strong objections against SYSmark 2002, and has backed its claims by releasing a PDF that graphically exposes its evidence. Let's recap what AMD's grievances are, and then address how this affects the consumer.

AMD claims, in a PDF that is available here, that comparing this year's benchmark with last year's one shows Intel's P4 to have gained disproportionately. When compared to Intel's 2 GHz P4A, AMD's Athlon XP 2000+ has degraded by an amazing 20% in Internet Content Creation. At 10%, the degradation in Office Productivity is also very significant (page 4 of PDF).

Having compared SYSmark 2001 to SYSmark 2002 at the task-level, AMD found that BAPCo had removed tasks that had favored AMD, had added tasks that now favor Intel, and had also inflated workload contributions by repeating tasks that favor the chip giant. This pattern was found across applications in Internet Content Creation and Office Productivity (page 8 of PDF). AMD's Athlon XP routinely outperformed comparable P4 processors when using SYSmark 2001. When BAPCo moved the goal posts for SYSmark 2002, no doubt with Intel's blessing, AMD's performance chip is now made to look a loser. Its current flagship processor, the Athlon XP 2600+, now even falls behind Intel's lowly 2 GHz P4A in SYSmark's overall score.

Of course, AMD's analysis raises some serious questions that BAPCo will have to answer. In the meantime, AMD will continue to use SYSmark 2001 until their concerns regarding this year's benchmark suite are resolved. Because AMD's evidence shows SYSmark 2002 to be so far off the scale, it will raise questions about BAPCo's other benchmark suites. This includes its recently released MobileMark 2002, a joint venture benchmark suite with MadOnion. Let's not forget that BAPCo and MadOnion are strategic partners, so this may reflect badly on MadOnion as well.

What is the relationship between BAPCo and Intel? It's cosy. Intel chairs the BAPCo desktop performance committee that is responsible for the makeup of SYSmark. Also, Intel provides software engineers that develop the SYSmark products. Talk about the fox guarding the chicken coop. When you add to this the manipulation that AMD claims, I would go further and add that this can only be seen as collusion between Intel and BAPCo. It should now be clear that the claim that was made in January last year, that BAPCo was just a front for Intel, was right on the mark.

Will AMD's campaign against BAPCo succeed? I believe AMD will be successful because it has grass roots support from the people that really matter. Also, AMD could not afford to make such claims against BAPCo unless they were on very solid ground. The longer BAPCo delays its response to AMD's accusations, the weaker its position becomes. Don't forget that support for SYSmark amongst the hardware community is not exactly solid.

Van Smith, Editor of Van's Hardware Journal, who has been resolute in highlighting the SYSmark issues, has written a commentary on the latest shenanigans. Anand Lal Shimpi, Editor-in-Chief of AnandTech, pulled the benchmark from his Athlon XP 2600+ review and said in this thread, "SYSmark 2002 can no longer be used as a sole measurement of application performance." Johan De Gelas, Senior Editor of Ace's Hardware, who also has issues with SYSmark 2002, has never used it in any of his reviews. He is also critical of MadOnion's CPUMark 2002 because he claims it doesn't prove a thing. Dean Kent, site owner of Real World Technologies, is now satisfied that the proof he required that BAPCo was intentionally trying to show the P4 in a good light has been pretty much satisfied.

If AMD's claims can be independently verified, then the title of the classic spaghetti western film, The Good, The Bad, And The Ugly, may be used to conclude this story. The characters who star in this particular scenario are AMD (the good) for exposing the smoking gun, Intel (the bad) for condoning and contributing to this fraud, and BAPCo (the ugly) for allowing its so called independence to be violated. Also, if BAPCo is the lamb that follows its master's voice, then Intel is surely its shepherd.

The world is still anxiously waiting for BAPCo to respond and to come up with a credible response.

However, it's the poor consumer that gets the raw deal when benchmarks don't truly represent the performance that is claimed. But maybe this is about to change. A small group of PC owners have filed a class action lawsuit against Intel, Gateway, and Hewlett-Packard. The plaintiffs allege that these companies misled them into believing that the Pentium 4 was a superior processor to Intel's own Pentium III and AMD's Athlon. The benchmarks of that era would definitely back up their claims, but would it be enough to convince a jury?

Intel's recently released Pentium 4 Celeron has also received criticism for poor performance, with PC World describing it as an " all bark and no bite" lap dog, which has even worse performance than the PIII based Celeron it supersedes.

When all is said and done, the class action lawsuit against Intel might turn out to be the best thing that could happen to the benchmark industry, exposing the holes and inconsistencies and benefiting the consumer. µ