Teeth make smiles, and smiles make sales - Unidentified Harrods person in Alan Sugar's The Apprentice
WHEN THE NEW Macbooks came out a few weeks ago, Nvidia stated that the chips they provided to Apple did not contain the proverbial 'bad bumps'. Unfortunately for them, an investigation led by The Inquirer proves that not to be the case.
If you recall, Nvidia has been in the spotlight all summer for failing chips due to bad materials and thermal stress. The end result is that bumps, the tiny balls of solder that hold a chip to the green printed circuit board it sits on, crack, and the computer it is in dies. If you want the full technical analysis, read this article (and parts 2 and 3).
Nvidia took a $200 million charge over the problem in July, but the firm refuses to support its customers by saying which parts are defective, and what computers they were sold in. You can get some clue from message boards, with Dell, HP, and Apple being prominent victims.
In the end, what you have is a massive cover-up that keeps affected customers in the dark. Doing right by them would cost a lot of money, which says a lot about the reason for a cover up. Fixed parts with a new 'material set' - basically new bumps and underfill - were phased into production starting in mid-summer, and the old, defective bumps are being sold off slowly alongside the new.
The question of the season is whether or not the brand new Macbook was designed and sold with 'bad bumps'. Nvidia told us directly that the chips were not using the 'bad bumps', and we took their word for it even though internal Nvidia sources were telling us that this was not the case.
One thing to keep in mind however, is that these bumps are so small that they are virtually invisible to the naked eye. In this case, they are about 100 micrometers in diameter, near the diameter of a human hair. To complicate things, they are permanently sandwiched between the chip die and the green fibreglass carrier, the bumps literally solder the two together. They are then covered with an epoxy-like material called underfill.
Nvidia could have shipped chips with bumps made of peanut butter and said that they were gold. As long as the chips functioned, there was almost no way of knowing exactly what they were made of. It is a pretty safe bet for Nvidia to call the parts good publicly, even Apple might not bother to check up on them. Again.
To say definitively what the bumps are made of, you would need to buy a Macbook off the shelf, disassemble it, desolder the chips, saw them in half, encase them in lucite, and run them through a scanning electron microscope equipped with an X-ray microanalysis system like this.
That is exactly what we did.
Here is the G96 cut in half, ready for analysis
Yes, you read that right, a brand new 15-inch Macbook Pro was purchased in California as soon as they went on sale. This was an off-the-shelf part, not a review sample, not a gift, but a normal model that hundreds of thousands of you bought. It was then secreted to a small lab of mad scientists who do not wish to be named, fearing repercussions from Nvidia and Apple.
These well-meaning boffins took it apart, desoldered the parts, and cut the defenseless notebook into many pieces. With meticulous care, they then ran it through multi-million dollar tools that would tell them exactly what materials the bumps used. Exactly.
This is what most of the Macbook Pro motherboard looks like
The motherboard of the new 15-inch Macbook Pro looks like this (above), with the heat pipes removed. There are basically three chips on it, the Intel CPU on the bottom, the Nvidia MCP79 chipset, and the Nvidia G96 CPU. The MCP79 is marketed under the name 9400M, and the G96 is called a 9600M GT GPU, but we may refer to them as just the 9400 and 9600 in this article.
The bumps have two possibilities, new and old, good and bad respectively. According to Nvidia documentation, the 'bad bumps' consist of mostly lead, 95% lead (Pb) in fact, with the remainder being tin (5% Sn). That is why they are called high-lead bumps. The newer 'good bumps' are called eutectic, and what they do differently is explained in great detail in the technical links at the top of the article. As far as composition goes, they are about two-thirds tin (63% Sn) and one third lead (37% Pb).
An electron microscope image of the chipset bump with analysis
Here is a closer look at the bump.
An electron micrograph of a 9400 chipset bump
Take a closer look at a portion of the graph that contains the elements in question.
Material analysis of the 9400 bump
Even if you don't have a degree in material science, you can plainly see that there are two big clumps in the graph. The two-pronged one on the right is tin, and there is notably more of it than there is lead, the spike on the left. This means the bumps on the MCP79/9400 are made of eutectic material (63% Sn, 37% Pb), and they are 'good'. Nvidia's story checks out so far.
Take a look at the same data for the 9600.
9600 micrograph and analysis, coloured red for ease of reading
And again, a closeup of the bump.
A good close look at a 9600 bump
And once again a close-up of the graph. (Please note that the original was black and white, we filled it in with red for clarity)
Part of the 9600 analysis, the tail was cropped for readability
Even a communications major can tell that there is one big spike at lead (Pb) and a very small one at tin (Sn). This would fit the profile of high lead (95% Pb, 5% Sn), and is radically different from the 'good bumps' of the 9400. The 9600 is unquestionably using 'bad bumps', directly contradicting the statements from Nvidia.
If you want more evidence, look at the surfaces of the bumps in the pictures above. Eutectic solder has a melting point that is the same for all components. When it cools, you should get an even physical structure with a fairly consistent grain. That is what you see on the 9400. With non-eutectic solder, the component that cools first should clump, and you can see that on the surface of the 9600.
You will notice that we stated in the beginning of the article that Nvidia said the bumps in the chips were good, and you can see from the above data, this is definitely not the case. The computer that 'donated' it's guts for the above analysis was a 15-inch Macbook Pro, purchased off the shelf in California. There was nothing special about it, not a press sample, not even a pre-production version. This is what Nvidia said was good.
How did they say that? Below is the last mail in an email chain between the author and Mike Hara, Vice President of Investor Relations and Communications at Nvidia. Phone numbers and email addresses were removed, and only the formatting was slightly changed for readability. It was sent at 1:52PM CST on October 15, 2008, titled "RE: 9300/9400 materials sets". The emails appear here in reverse chronological order.
Michael Hara - NVIDIA Corporation - Vice President of Investor Relations - ( 408) xxx-xxxx - Fax: (408) xxx-xxxx
From: Charlie Demerjian [mailto:charlie@xxxxx]
Sent: Wednesday, October 15, 2008 11:44 AM
To: Michael W Hara
Subject: Re: 9300/9400 materials sets
I assume you mean 9400, not 6400.
Michael W Hara wrote:
> The 9300/6400 and 9600 discrete all use the new material set.
> Michael Hara - NVIDIA Corporation - Vice President of Investor
> - (408) xxx-xxxx - Fax: (408) xxx-xxxx
> -----Original Message-----
> From: Charlie Demerjian [mailto:charlie@xxxxx]
> Sent: Wednesday, October 15, 2008 9:27 AM
> To: Derek Perez; Michael W Hara
> Subject: 9300/9400 materials sets
> The obvious question of the day is, what materials set is used on
> the 9300/9400 and the discrete 9600 used in the macbooks?
As you can see, the question was asked the day after the new Macbooks came out, and it is quite clear in naming all three potential parts, 9300, 9400, and discrete 9600. There is no nuance, and the then brand new Macbooks are directly named. As we have proven above, the statement in that email is simply not true.
One problem that journalists run into every so often is that PR people don't always tell the truth. They usually do, but every once in a while, they don't. Sometimes this gets found out, and that inevitably leads to a very embarrassing story, sometimes it doesn't. The only down side to this for the PR person is their getting caught with their corporate pants down, and the inevitable hit to their reputation.
The problem this time is that Mike Hara is not PR, he is IR, Investor Relations. His main job is to deal with stockholders and analysts, and is consequently under a very different set of rules when speaking to such people. People buying and selling stock expect, and are required to get honest answers to the questions that they ask.
The legal boundaries for IR are the same for talking to analysts, stockholders, the milkman or the press.
So, what does this all mean? It suggests that there are 15-inch Macbook Pros being sold with 'bad bumps', the same materials that brought down so many HP, Dell and Apple parts, both laptop and desktop. For some odd reason, Nvidia really does not want you to know this.
The first and most obvious question is, does Apple know? Repeated calls to Apple PR were not returned prior to this story, and while that looks pretty damning, it isn't. Apple will not talk to journalists unless they are assured the response will be fawning, and we do not fit that mold. That said, given the history between Apple and Nvidia, it could go either way.
When we ran our finding past Nvidia prior to publication, Mike Hara replied: "You asked me specifically if the 9400 and 9600 used in the MacBooks were free of all bad bumps. I responded to you that the combination of material underfill and bump is different from the combination that was exhibiting the bump crack field failures earlier in the year."
We find this problematic. If the bumps were not a problem at all, why were they changed on the 9400? The 9400 is a much cooler-running chip than the 9600, so why change the part that is less likely to die? If the 9600 with 'good bumps' is being phased in, why bother with the qualification costs, time, and inventory hassles if it is not a problem?
The other problem comes down to heat. The new Macbooks run hot, very hot. The net is filled with reports of them overheating and hanging. This is most often seen when gaming, a task that stresses the GPUs hard, and results in a 'black screen of death'. These beasties run extremely hot.
On the surface, the explanation of the Macbooks not getting hot enough to crack the bumps doesn't stand up. If it is hot enough to sear the flesh off your thigh, it is likely more than able to reach an internal temperature of 60-80C, the point where the underfill softens. If the chips get hot enough to crash, it is unlikely they are running within prescribed thermal boundaries.
If you assume the engineering work was done, and done correctly to keep the 9400 and 9600 in the correct thermal range, why didn't Nvidia simply say so? Barring a total failure of their lot-tracking system, they had to have known the Macbooks shipped with 'bad bumps'.
What do you do from here? At the moment, the simple answer is: avoid the 15-inch Macbook Pro. While there is no assurance that the high-lead bumps will cause a failure, given their history, we cannot recommend that you take the chance.
Apple and Nvidia need to clearly mark which machines have the 'bad bumps' so consumers can decide for themselves. Given that Nvidia claims to be transitioning from high-lead to eutectic bumps, it is only a matter of time until the high-lead inventory is depleted, and the Macbooks are safe to buy.
Until that time, you would be well advised to avoid these potentially problematic notebooks. µ
In a statement just before publication, Nvidia's Mike Hara had the following comment on the situation. "The GeForce 9600 GPU in the MacBook Pro does not have bad bumps. The material set (combination of underfill and bump) that is being used is similar to the material set that has been shipped in 100s of millions of chipsets by the world's largest semiconductor company."
Sign up for INQbot – a weekly roundup of the best from the INQ