Microsoft slammed for badmouthing benchmark

Well duh ! Looks like it's going to stay like that for the foreseeable future as well.
This is so stupid it's not even funny. What a moron this White is.

"Nick White, a Vista launch team Product Manager, wrote in his bog Friday that Microsoft hasn't released any benchmark testing results for Vista SP1 because ' the code is still in development and... remains a moving target.'"

Then why release it and charge hundreds of dollars for it? I can't believe he said it so flatly-- that is just begging to be fodder in a class-action suit (at least in the EU where you guys have consumer-protection).

I suppose even ubuntu is too commercial for you, I guess we should go back to using command line applications because it doesn't use as many resources as a window.

We should scrap JSP/Java/NET and all use PHP. 

We should refuse to use 99% of products that don't have open source drivers, we should refuse to use any linux operating systems that ship with media players and codecs as they are bloatware and it's much more fun installing it all ourselves. 

Of course this is after we've read the source code line by line as its probably distributed by aliens who want to take over our bodies and we need to make sure it's safe.

*for those that haven't figured it out - this is sarcasm


quote from the blog: "...as a general rule, we avoid sharing benchmark tests of software that hasn't gone RTM (i.e., final code)."

Point #1: If Vista hasn't gone 'final code' then why was it released to the public? So what they are saying is that they sold an unfinished product?

Point #2: If is not 'final code' because it receives updates, patches/fixes and service packs then exactly WHAT program has MS ever offered or is being used TODAY that IS 'final code'?

Under this guise the only s/w they have that IS final code is those programs no longer receiving support or updates. Nice to see they finally finished Win98 lol

http://cia.microsoft.com

I think its about time we have benchmarks and other tools to determine wether microbloat is just slapping on the eye candy on that pig of an OS or actually just putting icing on the proverbial cake.

Why do we need these benchmarks? after all, us techies know such things for quite sometimes...but guess what? everyone else just doesn't get it. Maybe these tools will enable the public to finally make an informed decision without being a tech.

So Microsoft is out shooting the benchmark messengers again. What else is new?

That's why Vista is so slow compared to everything else.

Both sides do have a point. I'd agree that a scripted test suite is not the same thing as real world use.

On the other hand, my experience of Office 2007 under real world use, indeed sucessive office versions is that some tasks have quite definitely become slower.

One of my biggest gripes with lateer model versions of office is the help system which insists on forcing itself over content, or moving the content, so after I've closed help I have to scroll. The old way of having help in a completely separate window rather than the idiot side bar was far more efficient for me.

The ribbon in O2K7 is a mixed bax. In some operations it's quicker, in others it's decidely slower.

A benchmark I'd like to see is one that measures the amount of mouse movement and the number of mouse clicks required to carry out various tasks. That would give quite a good 'real world' test for the efficiency of various office versions, with the less mousing the better.

Microsoft are the kings of FUD - they've been involved in a huge fear campaign against Blu-ray Disc over the last 18 monts to try and confuse the market and allow digital downloads to be the successor to DVD.
Microsoft's Amir Majidimehr is the ring leader - has gone as far as convincing innocent HD newcomers on the www.avsforum.com back in the Summer of 2006 that Blu-ray Disc would never use anything higher than a 25GB disc and that only MPEG-2 would be used, so HD DVD with it's Microsoft driven VC-1 video codec, more efficient than MPEG-2, and it's 30GB disc would be the better choice. Had it not been for people like Amir, MIcrosoft's Corporate Vice President, Consumer Media Technology Group, HD DVD wouldn't have gotten off the ground and we'd all be able to confidently buy into the new HD format.
Kudos for calling them out.

......Done that. 

A few months ago I helped a manufacturing company rollout Office 2007, so this is first hand experience. Even an admitted Microsoft Apologist like myself found Office 2007 to be absurdly bloated, and even running on a 2ghz Core Duo felt like Office 2003 on a 450mhz P3.Not to mention Office 2007 conflicts with every other MS product and plugin causing your CPU to ride 100%. If the problem is twice as bad in Vista, then MS is in for serious trouble. Office 2003 and even 2000 hold up very well against the Open Source alternatives, but 2007 *is* and utter joke.

Basically this guy comes along and shows that currently Windows XP, with the latest SP3 installed, is faster than Windows Vista with its latest SP1 installed (well, the current beta of each) and takes Vista behind the woodshed. And by running a "script" benchmark (the same set of tasks is run in succession on each OS to be fair), this makes Vista look bad ==> i.e., why switch to Vista EVEN WITH SP1 when it still looks like a slow, bloated moose compared with XP.

Many of us know the answer to that: DON'T! RUN! My free copy of Vista sent from HP enjoyed a brief stay of eBay, not my computer.

Microsoft's (notice the lack of a retarded play on words on the name?) response is an expected, unimaginative dismissal of the report:
(1) SP1 is beta code, so even if what you say is true now (and it may not be), it doesn't mean anything down the road. 
******But you and I know that no miracle is going to happen between beta=>RC1, RC2, etc =>RTW.
SP1 will be a lackluster addition. 
At least Microsoft is sorta telling people this already (don't hold off for SP1 to buy Vista, nothing to see here...).

(2) By stating a scripting code is a "synthetic" test of sorts and not representative of "real user experience," he takes a stab at the validity of the test itself. 
*****AGAIN, However, we know that scripting codes are GREAT for Apples to Apples (no pun intended) OS testing because they allow the exact same string of events to occur on both OS's in the same order. Sort of like playing the exact same musical piece on two pianos to see which is better tuned, better sounding. A scripting test was the perfect choice.

But hey, the Microsoft guy has to say SOMETHING, right. I mean, he gets paid to instill public confidence in MS products. And many people probably accepted the superficial half-truths he burped out as fact and are now smiling and waiting for Vista's SP1 with anticipation.

"Oh, he;s right, it's just beta code, let's just see what happens as the code (COUGH) - matures."

Everything in the world is happy again. 


First the company tries to push through headlines by comparing Office 2007 on Vista to Office 2003 on XP. 

Ok fine, but a bit on the dirty side of getting headlines.

There are other issues that should be talked about with the latest tests as well, like if they are drawing the UI elements super fast, then it is benchmarking the animations and not the OS or the applications. Also they using a dated COM scripting model that doesn't run as native in Vista as it does in XP.

But with even all of that aside, go read their test report pages and posts from individuals. Most of the people that have the capacity to run their tests on equal hardware are showing that XP and Vista finish the test within 1sec of each other, and sometimes it is Vista that is 1sec faster.

And this is just not a few people testing, but several that are providing their results back to the company and then being ignored.

We have an internal test lab, and one of our techs set up exact tests, and he as well found that XP and Vista finished within 1-2secs of each other, and depending on if Vista was indexing or optimizing in the background the winner would be XP or Vista by the 1-2sec difference.

My tech has several questions for the testing company, but can't get any real answers. 

For example, are they running the test with Aero on or off? Vista actually runs faster with Aero on if you have a GPU higher than a 5600 Geforce from 2003. Even in games, running full-screen or in a Window, Vista is faster with Aero on. In fact for business applications, even when the Vista drivers were 'slow' in Jan/Feb of 2007, Aero on was faster than both Aero Classic/Basic and XP.
Links: http://www.firingsquad.com/hardware/windows_vista_aero_glass_performance/page3.asp

http://arstechnica.com/journals/microsoft.ars/2007/1/2/6453

What version of Vista are they using, and what other applications or anti-virus do they have running that could cause such a dramatic difference in their tests and what people like my tech found. 

There is also the fact that their test is completely off scale of any other online review or test of Vista.

Take gaming, as of June ATI and NVidia's drivers had matured to the point where XP and Vista were the same speed. And gaming is far harder hitting than anything else most people are using their OS for.
Link: http://www.firingsquad.com/hardware/amd_nvidia_windows_vista_driver_performance_update/page9.asp

I wonder what Microsoft would say if the results were reversed?

<Q> He also denigrated script-based benchmark testing because he claimed it " isn't representative of real-world user behavior and hence isn't an accurate guage of the actual end-user experience." </Q>

Ummm, no. See, for this...is the one thing that office has done very well since office 97 that OpenOffice can't hold a candle to: document automation. Unfortunately MS doesn't know just how powerful VBA is, I think but I digress.

Anyway, I disagree. Lots of folks here at our shop use word. No scratch that. They use our home grown apps to generate word docs. Lots of word docs. LOTS and LOTS of word docs, and I'm not talking basic form-filler letters either. Automation _IS_ how they "use" office 90% of the time. So there.

Windows Vista, for the loss.



Thanks for bringing this test to my attention. I'll be ordering my copy of XP shortly.