The Inquirer-Home

Are editors' awards editorially challenged?

Mario asks some very good questions
Tue Oct 26 2004, 00:47
EDITORS' AWARDS are dear to the heart of vendors and gongs awarded by different hardware sites are the order of the day. But there's something wrong with the state of Denmark and that's what I hope to point out in this article.

First, we'd like to say that the Anandtech web site is only cited as an example, and we've given that respected web site a sight of this article before publication.

The thrust of this piece is far from anti-Anandtech. In its reply, it maintained that it offers comparative reviews of boards based on a number of criteria, including those intended for people contemplating buying Intel only or AMD only boards.

But if there is a choice between Intel and AMD boards, it seems clear to me that consumers would be better served by direct comparisons between the two on points where it matters - that is to say on price and performance.

Top honours for pedestrian performance
An Anandtech motherboard review from August - Intel 925X Roundup: Creative Engineering 101 - didn't raise any steam in the article's comments forum, but it certainly raised my blood pressure. A 925X based Asus board, the P5AD2 Premium, had been presented an Anandtech editor's choice gold award for excellence, which I felt wasn't justified at all.

Let's compare the performance of that board to its Athlon 64 based counterpart - the MSI K8N Neo2 Platinum, which also won the same editor's choice gold award. The P4 based platform won only one benchmark test - MPEG-4 encoding. It was last in all the others. When we look at how far in front the Athlon 64 rig was in the benchmarks that it won, the 925X based Asus board clearly didn't merit its award.

Business Winstone - 13.6%
Multimedia Content Creation Winstone - 9.2%
Far Cry - 15.1%
UT2K4 - 21.4%
Aquamark 3 - 4.6%
Final Fantasy XI - 22.0%
Halo - 21.2%
Quake 3 - 11.4%
Wolfenstein - 10.9%
UT2K3 Flyby - 29.9%
UT2K3 Botmatch - 33.0%
X2 - 4.7%
Comanche 4 - 25.6%

As you can see, the Pentium 4 machine was destroyed - the Athlon 64 rig didn't take any prisoners. This was a white wash - men against boys, but Anandtech somehow saw fit to award - and thus give equality to - a patently inferior platform. When one also factors in that gaming constituted 78.6% of the tests, which all went the wrong way for the second tier motherboard, how can this award be justified?

Also, think about the top performing Athlon 64 based motherboards from Asus and Abit, which both won the Anandtech editor's choice silver award. (3). Because both those mainboards also leave the 925X based Asus offering eating dust, don't they merit higher recognition? If the author had rightly based the award on standard configuration performance, the Asus board would have been lucky to have even been considered for a bronze editor's award.

It would be nice to know how two motherboards, which are worlds apart, were awarded the same gold editor's gong. If there is no credible reason which explains this, then these awards are arbitrarily based. If that is the case, then Anandtech is giving the performance challenged Intel a helping hand - it's the chip giant's chipsets and processors which ride on the back of such awards.

Some may be thinking that the processors used in the review were not equally matched - they're clearly different performers. But as the author said in response to that exact question: "The 3.6 is the fastest Intel processor. If you will check our launch reviews you will see the 3.6 outperformed the 3.4EE. We are indeed comparing the best performing Intel - the 3.6 - to the best performing AMD - FX53." (4).

Adding insult to injury
Having had my intelligence insulted, Anandtech went on to rub salt into an open, weeping wound. The 3.6GHz Pentium 4 that was used in the review wasn't available in volume - it had been paper launched. We reported on the same day Anandtech's article was published that Intel's flagship Pentium 4 was as widespread as hen's teeth. Using it in a comparative motherboard review, without indicating its virtual non-availability, further distorts reality.

Is the Intel 925X/915 platform broke?
As the author noted: "Workstation benchmark results with SPECviewperf 7.1.1 have been extremely variable on the 925X/915 chipsets. Results are so inconsistent, with up to 50% variation from one board to another using the same configuration, that Workstation benchmarks will be excluded from 925X/915 reviews until we can discover and fix the inconsistency or we can establish a new suite of Workstation benchmarks." (5).

When a critical component of the Anandtech benchmark suite wasn't reported because of inconsistent results, and more importantly, it didn't know why that was, how could Anandtech make conclusive determinations and dish out awards when it didn't have all the facts?

Because the six tests, which constitute the SPECviewperf 7.1.1 benchmark, had not been included in the review, the number of measurement tests had been reduced by 30% - a not too inconsiderable amount.

Doing Intel's bidding
When one also considers that the same author had rightly praised socket 939 board performance in an earlier review: "It was difficult to resist being a little sensationalist in this 939 roundup and titling the review, "Who needs 925X?" That would have been a fair title, however, since you can clearly see that all of the Socket 939/FX53 boards completely outperform Intel's top 560 on the top 925X motherboard. Even Media Encoding, the last bastion of Intel dominance, is now a dead heat with the new AutoGK benchmark." (6).

And then strangely praised the 925X Asus board as well: "Based on top performance at stock speeds, the wonderful implementation of Intel 925X/ICH6R features, the excellent enhancements to those features such as Stack Cool, Dolby Digital Live encoding, WiFi G networking, dual PCI Express LAN, high-speed 1394b firewire, and the best overclocking abilities of any 925X motherboard, we are pleased to award the AnandTech Gold Editors Choice to the Asus P5AD2 Premium. The P5AD2 hardly qualifies as cheap, but it does deliver excellent value with a standout range of features, enhancements, and performance." (7).

Doesn't the first quote negate the second? Anandtech has to ask itself why it has been so incongruous.

The 925X Asus board is no doubt a top performer in the Intel world. But in the real world that we all live in it's clearly wooden spoon to the Athlon 64 based competition. It's no good having a "wonderful implementation" if the performance doesn't deliver. It's like having the best chassis in Formula One but an engine which isn't up to the job - the car won't win any races. "Excellent value" against what exactly? It seems like Anandtech has swallowed lock, stock, and barrel the Intel line that says: "I think the theme is the overall platform elements"..."the complete platform"..."The message you should walk away with is that this really isn't about one technology or another, it's about delivering a collection of them". As I said in an earlier piece: What good does having "the complete platform" when the engine that drives it is second class?

Anandtech, considered one of the mega hardware review sites, has set a very bad example. The loose cannon remarks, which were used to describe Asus' performance challenged motherboard, will proliferate the Intel line - the chip maker will be laughing all the way to the bank. Anandtech may as well be adjunct to the chip giant's marketing department. Look at how Asus North America is already riding on the poor example set. Its front page hyperlink, which links to the Anandtech review, says: "P5AD2 Premium awarded Editor's Choice in Anandtech's 925x Roundup for Dominating Performance in stock and overclocked speeds". (8). Dominating against what exactly? It certainly didn't dominate against the Athlon 64 based competition.

Others unfortunately suffer fron incongruities
Because of Anandtech's very high profile, and it just happened to be the best example, it was chosen to highlight this problem. But others have the same ailment.

Asus has a growing list of review sites that have already presented the performance challenged motherboard an award. (9 & 10). It's like sheep being herded into a pen. Can anybody guess who the shepherd is? Since we've heavily criticized Anandtech, it's only right that the others in the dock be named.

1. Altom Data, Denmark
2. CHIP, Netherlands
3. Gamestar, Germany
4. APC, Australia
5. CHIP, Germany
6. Maximum PC Magazine, United States
7. NAG (New Age Gaming), South Africa
8. PC Gameaddict, Australia
9. PC Professionell, Germany
10. Gamers Depot, United States
11. PC games, Russia
12. Sudhian Media, United States
13., Netherlands
14. Hot Hardware, United States
15. HWM, Malaysia
16., United States
17. PC Perspective, United States
18., Russia
19., Germany
20. PC Enthusiasts Magazine, United States

Another example of an awards process that is broken is taken from Tom's Hardware Guide. Back in July, it reviewed 16 motherboards based on the Intel 865PE and the 875P chipsets. Three of those boards received its editor's choice award. (11). But based on pedestrian performance against the Athlon 64 based competition, those boards shouldn't have merited an award.

The worst example of hyperbole that I've seen for a 925X based board is taken from PC Perspective. Maybe the tech site should think about renaming itself PC Unreality:

"The ASUS P5AD2 Premium motherboard offers users outstanding performance and incredible features! Overclockers and enthusiasts will feel right at home with this motherboard to start off their new Intel 925X platform system. And because of the board's exceptional qualities, it is the first recipient of the PC Perspective Editor's Choice Award!" (12).

If Microsoft releases its 64-bit desktop OS next year, the 925X motherboards shipping today will be obsolete unless the 64-bit support is just waiting for the chip giant to authorize its use via a BIOS update. So what good does having all those "incredible features" when the critical one for the future - 64-bit capability - might not even be available.

Fixing the broken process
It should be clear to the neutral observer that the editor's awards process is broken. When coming a poor second still guarantees winning gold, which is unconscionable, it is like Ferrari being the runaway winners of the Formula One Championship, only to then have to share its hard earned award with a team whose pedestrian performance didn't merit it.

The following changes to the Anandtech editor's awards for motherboards would make them worthwhile, credible, and covetable. There would be seven motherboard categories - twenty-one gongs in total if all gold, silver, and bronze awards are presented.

Awards for standard configuration performance
1. Top-of-the-line
2. Mid-to-upper-range
3. Best-bang-for-the-buck
4. Entry-level

Awards for other disciplines
1. Best overclocker
2. Best featured
3. Best innovation

These gongs, which would be clearly differentiated, would be awarded quarterly or at some regular interval. Gongs for overclocking, features, and innovation wouldn't be part of the first group evaluation. The reason is that they devalue standard configuration performance - the most important evaluation benchmark. Some editor's awards have clearly lost sight of that.

For the top-of-the-line award, performance would have by far the greatest weighting. For the mid-to-upper-range award, performance would still be very important but price would come into play. The best-bang-for-the-buck award doesn't need an explanation. The entry-level award would be the best motherboard with integrated graphics.

With these awards, Anandtech would cover the whole PC motherboard spectrum. From my perspective, no Netburst platform would meet the performance or price/performance requirement to justify being considered for the first group of awards.

For the other disciplines, however, the Netburst platform could clean sweep those gongs. The 865/875 based motherboards are reckoned to be the best overclockers out there. The 925X based boards are probably the best featured. As for innovation, Asrock's P4 Combo board, which supports both socket 478 and socket 775 processors, would be a worthy candidate for consideration.

Because of the number of awards and the regularity of the awards process, Anandtech wouldn't have to dilute the value of its gongs by awarding the same award to different motherboards. So if a motherboard won the top-of-the-line gold award, it would be the only motherboard that would hold that gong until the next awards process. But there may be instances where one board is so very good, or continues to be improved, that it continues to hold that award.

Now what's been suggested is a general outline, so it can still be further refined. To make the awards process "above board", Anandtech would post the awards process criteria. That way, those who want to can read them. Those who don't like them can raise their ire. Credibility would be brought to bear with the help of the Anandtech community.

Another day, another dog's breakfast
It doesn't inspire confidence when one considers the shortcomings of Anandtech's 925X motherboard review. We've highlighted the failings in detail, so let's bring them together in a list.

Asus award failings
1. Giving equality to a patently inferior platform
2. Awarding it above two better performing Athlon 64 based motherboards
3. Awarding gold status for pedestrian performance
4. Awarding gold status without having all the facts
5. Awarding a potentially obsolete platform - 64-bit support not confirmed
6. Making the Intel 925X platform look much better than it actually is
7. The ill-judged award could deceive the general buying public
8. The ill-judged award may have a detrimental effect on competition, which could keep IT prices higher than they need to be

Review failings
1. Lost sight of the most important criteria - standard configuration performance
2. Use of paper launched 3.6GHz P4 distorted results
3. Leaving out 30% of the tests
4. Not publishing the SPECviewperf 7.1.1 results
5. Not knowing why the 925X boards showed such wide SPECviewperf 7.1.1 variation
6. Stoking the hyperbole fire

1. Editor's awards have been brought into disrepute
2. Intel's 925X platform looks like it can do no wrong
3. Anandtech's reputation has taken another nose dive
4. Intel will be laughing all the way to the bank
5. Others will follow Anandtech's bad example

How will Anandtech respond?
Let's hope it doesn't see this like it initially viewed its earlier "Linux and EM64T; Intel's 64-bit Suggestion" review debacle: "Relax, its just a primer for future articles." I consider this motherboard review to have worse structural failings.

What should Anandtech do? If the tech site's collective judgment agrees with what's been said, which should include its readership as well, then the actions required to remedy the failings are obvious. The most important being the withdrawal of the 925X Asus board award. If Anandtech fails to do that, it will have to justify the failings listed above.

Defending the indefensible
Hardware review sites we've named may feel that their awards process is credible and above board. If so, and you wish to defend your position before the world, I'm sure the editor has prepared a page to deal with those replies. If you feel too embarrassed to defend your case, your silence will seal your guilt.

In light of current performance, if no credible review site would award the Intel P4 processor an editor's choice award for excellence, why is it that when the wooden spoon performer is plugged into a Netburst based motherboard, the platform is elevated to the heights of unreality?

The major performance question that hardware review sites need to answer is this. When one considers that there are two major platforms that compete in the PC market - AMD and Intel - how is it that the top Intel based solution, which is pedestrian when compared to the Athlon 64 based competition, can even qualify to be considered for an award based on standard configuration performance?

Of course, it may now be the case that standard configuration performance is no longer the most important criteria. If that is the case, then why bother testing the motherboards? One may as well just photograph the boards, list the specs, and present the awards.

It could be that I've been totally naive. Maybe I've missed the whole concept and design of the editor's awards process. It might be that editor's awards are just that in name only. Maybe it's just a tool used to keep the motherboard makers happy, which in turn keeps the life blood of the review site flowing - its advertising revenue.

If the previous paragraph is true, then the following conclusions can be arrived at. Some editor's awards are being used to support and maintain the Intel empire. This has the effect of maintaining the status quo, which will stifle competition. This means the price we pay for IT will continue to be higher than it needs to be - consumer and business customers will continue to be the losers. So some hardware review sites which may claim to be their readers' keeper are stabbing them in the back.

Sense and sensibility
An Anandtech motherboard review from June 2002, written by Anand Lal Shimpi, Anandtech editor-in-chief and CEO, made this comment about his editor's choice awards: "Normally on this page we hand out our coveted Editor's Choice awards, including the Editor's Choice Gold award for excellence."(13). The dictionary definition for excellence: "superiority: the quality or state of being outstanding and superior". (14). The Asus 925X based motherboard clearly didn't meet that mark in comparison to the Athlon 64 based competition.

In the future, if Anandtech and other sites continue to present gongs to the undeserving, may I suggest they change two aspects of their awards process. The award wording should be changed to the following: "Editor's choice award for the performance challenged." A picture of a bicycle should be added to the award logo, which would graphically send the right message. Both those changes would clearly differentiate the second class performers.

Hardware sites need to bring credibility to the presentation of awards. Sense and sensibility matter to people spending their hard earned money for products, which is why they read review sites and for that matter review magazines. Let's hope these words are taken in the spirit they're intended. µ


Share this:

blog comments powered by Disqus
Subscribe to INQ newsletters

Sign up for INQbot – a weekly roundup of the best from the INQ

INQ Poll

Heartbleed bug discovered in OpenSSL

Have you reacted to Heartbleed?