Passmark puts joy back into simple benchmarking

---

Product: Performance Test 5.0
Company: Passmark Software
Web: www.passmark.com
Price: $24 USD
Requirements: MS-Windows 98 or higher, all the way up to XP
(32-bit and 64-bit versions available)
Tested version: 32-bit on WinXP Home SP2

---

ONCE UPON A TIME in the dark ages of computing -say early to mid-'90s- testing your PC performance was easy: you booted ms-dos, loaded "si.exe" (Norton Utilities' "System Information") and immediately you saw a (text mode) bar graph showing your cpu's Mhz. That was all one needed to conclude "my PC is faster than yours". Looking back, perhaps "faster" was an exaggeration (grin). Those were the days 40 Mhz 386 cpus from AMD beating Intel's 33 Mhz, or a bit later when Intella shipped thousands of 66 to 100 Mhz 486 computers, which were expensive and also the average on corporate desktops. (warning: long introduction follows, if you are of the impatient type and you want to go straight to the review, scroll down to "Enter Passmark").

The Magical "MS-Office Generated Number"

So I don't know exactly when I started losing interest and stopped paying attention to complex benchmark results, but I think it was around the time when PC Mag introduced its Winstone performance test, or as I put it, the "magic number". It was a time when benchmark programs used by the printed magazines started growing exponentially in siz, and adding Microsoft Office "scripted tests" to the mix.

Suddenly when you opened certain american computing magazines all we saw were a series of PC systems pictured one after the other, and next to each, its "magic number" (the Winstone score), and next to only one the magic phrase: "Winner!". Oh boy, it felt so good not to have to think about things!. ;)

But what exactly did that number mean? Well, it meant that for running that set of test scripts, system "x" was faster than the rest, nothing more, nothing less. Of course, someone who used other applications -which were not part of the benchmarking suite- could have obtained very different results, depending on the design of the applications -if those were disk-intensive, or cpu-internsive, or later in the win95 days, if those applications you ran on a daily basis ran out of memory and made the OS increase the virtual-memory pagefile or not, etcetera, etcetera, etcetera. Even the compiler used to create the application could affect results. I remember reading at the time that Byte found that IBM's own recompilation of Microsoft's Windows 3.1 code to create the WinOS2 subsystem used inside IBM's 32-bit OS/2 2.x - was faster due to their use of Watcom C++ instead of Microsoft's own compiler.So indeed different software can produce different results due to CPU optimization.

Back to present time: the moral of the story is that applications-based benchmarking is always a "bubble scenario" and you can't take a magic number for granted and think that's definitive, specially if the apps used (and the files loaded and saved as part of it) do not match the ones you use daily. Yet, for years, plenty of people simply took the magic "Winstone number" for granted and  proceeded to base their purchase decisions on the Winstone score. I remember having heated arguments with people who insisted on purchasing the "higher winstone" system while it was obvious by looking at the specs that they would have been much better with a lower scoring system but which had better graphics performance (or disk performance, depending on the case).

Speaking of magic numbers, last year, PC Mag decided to dump their own 12-years-old Winstone legacy. In their own words: "With Winstone's long run, from its development in 1992 to its final iterations in 2004, we believe we succeeded. Now we've decided to move on to a new set of tests" (sic) [interesting, "we dump it, because we succeeded" ;)].  Immediately they announced their decision to switch horses to a program with "large industry consensus behind it" and embraced BAPCo's SYSmark 2004. So a few days ago, I visited the "Benchmarks" category in the PCMag.com downloads site  and found that indeed they list BapCo's benchmark tool which, once again, seems to run scripts on a small set of desktop applications (mostly from Microsoft) installed on the system. All this to come up with the already familiar "performance number" based on the time taken by the evaluated system to complete the scripted tasks with the mentioned applications. Let's see the applications listed in the description: Adobe Acrobat 5.0.5, Microsoft Access 2002, Microsoft Excel 2002, Microsoft Internet Explorer 6, Microsoft Outlook 2002, Microsoft PowerPoint 2002, Microsoft Word 2002, Network Associates McAfee VirusScan 7.0, ScanSoft Dragon Naturally Speaking 6, and WinZip Computing WinZip 8.1. Hrmm.... they might have as well named it MicrosoftMark!. I wasn't thrilled... another meaningless number that would depend a lot on Microsoft's office code that I never run concurrently, and most important, comprised of 9 out of 10 applications that I hardly use anymore after having migrated to StarOffice 7 and Mozilla/Firefox two years ago.

"My name is Fernando and I fear fat benchmark programs"

Before someone at the above mentioned company decides to get trigger-happy and flame me: don't misunderstand my point, I think that huge benchmark programs based on MS Office and other huge bloated content creation apps ( Macromedia Dreamweaver MX, Flash MX etc) are ok and they have an "audience" and serve a purpose. I'm just personally not interested in running those tests when I need to measure the performance of two of my systems (or one of my systems with graphics card "a" vs. "b"),  as the results will be based on such an unrealistic set of applications that it wouldn't reflect my daily use, it would just be a meaningless number. In short: I think there's lots of baloney in benchmarking-land, and personally, the only tests that I care about are the synthetic benchmarks of each PC subsystem (disk i/o, cpu, video, etc).

It's an easy way to get a score for every subsystem, then repeat on another PC to compare scores (or on the same system exchanging components to see the impact). There are whole web sites dedicated to discuss if Doom 3 or Halo or the killing-humans-or-beasts-game-of-the-month achieves three or five or ten frames per second more on system "x" vs "y". I wish them all Godspeed, but that doesn't float my boat.

The fact that there are complex benchmark applications like Futuremark's behemoth, Sysmark 2004 -which sells for $399, ships on DVD-Rom and lists among the requirements "7.0 GB Free hard disk space"- means that there are some folks that are actually interested in that kind of NASA-like scientific testing, and I also suppose that they end up taking decisions based on those magic numbers. While controversies do often erupt about the benchmarking programs, or the drivers, or the graphics cards, quite frankly, I'm not interesting in those byzantine discussions, that's what hard core benchmark oriented neo-religious sites focus on.

Enter Passmark

I asked our resident Charlie for suggestions on what simple "2d" graphics benchmark program to use for testing two notebooks with a very similar cpu, but different ATI graphics chipsets, as a future article involves knowing the impact of one notebook having a slower ATI 9550 vs the other's full ATI 9600 on daily "2d" windows GDI graphics operations (opening a window, displaying text, scrolling up and down, simple directx animation, etc), yet, I warned him about my problem with fat-benchmark programs: "But I don't want a bloated piece of **** that forces me to install Microsoft Bob 2.0 Resurrection along with Encarta Encyclopaedia 2006 Preview Edition - The Melinda Gates Selection". So, another e-mail was fired in Charlie's general direction "plus, it's got to be freeware, or very affordable!". His reply arrived instantly: "Sorry, I don't have a clue". Poor Charlie, I don't blame him. He was in the middle of the Fiorina soap opera, as it was unfolding.

Thanks to the magic of Google, however, I managed to land into the web page of Passmark Software, which I later learned has offices in Australia. Quickly I found that this program had everything I was looking for: a set of synthetic benchmark tests to stress every possible subsystem, and at a very detailed level as well (MMX/SSE and 3DNOW, file compression tests, memory read/write/cache, etc). All in small package delivered as a one point three MB download!. So I decided this was a good opportunity to run a benchmark on the eMachines M6810, the notebook I purchased late last year after a long deliberation with myself, and finally praised some time later.

Time to Compare and Contrast
I dual-booted WinXP Home and installed the program, noticing the very simple and friendly user interface. Perhaps the most useful aspect of Passmark's Performance Test 5.0 (PT from now on) is the ability to show visually (using colour bar graphs) how well does your system scores on each category compared and contrasted with other brand name PC systems and models. Not only that, once you run PT on a given system, you can export the results and then load those on any other system for a quick visual "mine is faster than yours" comparison.


CPU scores compared, contrasted


You can download benchmark results from the Passmark web site "baselines" in passmarkspeak) from the company's web site, and then IMPORT those into the program. The "baseline" files use the ".pt" extension and are zipped. After extracting the .zip file contents to the program's folder (usually "c:program filesperformance test"") and selecting the file from Baseline->"Install Baseline from File...", the new benchmark results for the given PC brand and model will immediately appear on the list of available systems that you can compare your own system results against.

Uploading your results, or downloading others'.

In case you want to suffer by comparing your system to similar ones, it's all a matter of using the company's search function and entering the CPU and Mhz speed, to obtain a list of matching ".pt" performance results files. A checkbox on the web form allows you to restrict the search just to notebooks. Very, very handy. The list of tests performed by PT 5.0 is long, but here it is:

CPU Tests: Integer, floating point, MMX, SS3/3DNow, compression, encryption, image rotation, and string sorting.

2D GUI/Graphics tests: lines, rectangles, shapes, fonts and text, and "windows GUI", all measured individually

3D tests include individual results for 3 levels of DirectX 3d animation: simple (several "3d" balls without texture being zoomed in and out), medium (same as simple level, but with added textures), and complex (same as medium but running in full screen mode, at different resolutions).  Of course this is not the same as running a full 3d game, but re-read my introduction... this is not supposed to be a gamers' graphics benchmark program. Besides, that's why there's built-in benchmarking functionality in some games.


Looking for results files at Passmark's site

Memory benchmarks include: Allocate small block, read cached, read uncached, and write. On the disk front it's got: sequential read, sequential write, and "random seek + rw". Finally, "cd read speed" is measured as well (you're prompted to insert any data cd-rom or dvd-rom, in case there's none).  With all these results, PT compiles individual "Mark" scores for each subsystem, using all the results of each category as the source. For instance, with all CPU results a "CPU Mark" magic number is created, with all 2d graphics results a "2D graphics Mark" one, and the like.  So you end with additional totals titled "memory Mark" "disk Mark" etc.  In case the default tests are not enough for you, the program offers an "advanced tests" option where options like "Networking" and "Multitasking" are available, and where you can also customize many of the values used by the normal tests (like disk sequential r/w, etc).

Final PT "average score" or "Passmark rating" for the eMachines M6810.
Yes, even this program has a magic number, but its beauty lies in the ease for comparing individual subsystem results.

Saving results is easy: you can choose to export results of your system to a .CSV (comma delimited text file), save the results into the program's native ".pt" (baseline) format, or even save the results (including bar graphs) as a very long and narrow bitmap file, ready to be posted on the web. And speaking of  the web, the program has a killer option: export results as html. This creates a resulting web page like this one. Notice, however, that the last column with the M6810 results was highlighted manually, by changing the background colour with an html editor for easier readability The GIF exported version of the same data can be seen over there.

Saving the eMachines M6810 results as a new "baseline"

Saved "baseline" files include all the relevant information about the system tested including CPU, Mhz, Windows version, and most important, screen resolution and colour depth. I was able to see how much does the colour depth impact the graphics results. With the eMachines M6810 running at full wide-screen (1280x800 WXGA) resolution with 32-bit colour depth, the PT 5.0 passmark rating obtained was 282.3. When I reduced the colour depth to 16-bit ( colours are more than enough for me, thanks!), the PT passmark rating jumped to 298.2 due to the much higher 2d and 3d scores. Moral of the story: test in 32-bit mode to contrast with other systems, then try reducing your Windows colour depth configuration to 16bit for everyday use, to get an extra performance boost.

The verdict

I give Passmark Performance test 5.0 four and a half Fernandos in my personal one-to-five rating scale. Their tech support guy answered in less than a day when I had a question, and promised to forward my suggestions to the developers, even hinting that my idea of (highlighting the results column of the current model when saving 'baseline' comparison results as html) could very well be implemented on the next update. Some folks will undoubtedly prefer the several gigabytes king-size benchmarking monsters. For me, Passmark's Performance Test give me everything I want to know, and makes the saving and exporting process of the benchmark data a piece of cake, to boot. It's as if someone had brought the fun back into simple home benchmarking.

If your needs don't include measuring a bubble scenario of assorted Microsoft apps and just need to compare performance between your home or office systems, or perhaps see what's the impact of one component change in a desktop PC -say video card x swapped for video card y or 5,400 rpm drive replaced with a 10,000 rpm one-, I'd say you need to look no further than this. Clearly the import-export features (and save as bitmap, save as html) features show a level of attention to details unknown in other software companies.µ

L'INQs:
eMachines M6810 Performance Test 5.0 results (html export function)
eMachines M6810 results (GIF)