A DOCUMENT HAS COME TO LIGHT that details the lengths to which Nvidia has gone to cover up the problems it has been having with its graphics chips.
The most recent lawsuit against it by the National Union Fire Insurance Company (NUFI) claims the company has withheld information on the nature of its bad bumps. The very same information it has withheld from us or any other nosy hack or awkward analysts.
The story was broken by a certain Mike Magee at TG Daily on Friday, and it has a lot of juicy bits. The short story is that the list of defective chips shipped by Nvidia goes back to the NV4x generation, and the list of OEMs affected counts ten and basically includes every Nvidia customer.
NUFI complains bitterly that Nvidia has been covering up essential information it is entitled to receive as Nvidia's insurer by refusing to disclose even the most basic facts about the company's GPU chip failures.
We had the same complaint. Let's go back over what happened so you can see the depths of this debacle.
On July 2, 2008, Nvidia issued an 8-K report that essentially said: "We have a big problem." But then it got really vague. Its 8-K mentioned devices "in certain versions of our previous generation MCP and GPU products used in notebook systems." Nvidia also claimed that it didn't know why the problems were happening, saying, "We have not been able to determine a root cause for these failures." But was sure that it had fixed them. No, really, it said so quite explicitly. "All newly manufactured products and all products currently shipping in volume have a different and more robust material set," it stated.
We said at the time that this was cobblers. Nvidia was blaming everyone but itself. Nvidia was not saying which OEMs were affected, and privately it was telling financial analysts that there was only one OEM, HP, that had problems. We said then that it was a design problem and unlike Nvidia's crack team (pun fully intended) of failure finders, we told you precisely what it was, that is, high lead "bad" bumps cracking between the die and the substrate.
We also told you that there was more than one OEM was affected, starting with Dell, then HP, then Apple, and lastly Toshiba. If you read the Apple link, we mentioned it, but it looks like Toshiba never got the guts to do right by its end-users. The ten OEMs listed by NUFI in its complaint are Dell, Toshiba, Apple, HP, Quanta, Wistron, Compal, Asus, Samsung, and Fujitsu-Siemens, "and others." That list adds up to at least ten times the number of affected OEMs that Nvidia had been originally spinning.
The claim all along from the company has been that it still doesn't know what is wrong, and it has no way of figuring anything out. That didn't prevent Nvidia from coming up with the dollar amount it would cost to fix the problem, with great precision, but it couldn't figure the problem out.
So now, the defective chip list has expanded to the G86, G86A2, G84, C51, G72, G72M, G73, G72A3, MCP67 and NV42 officially, but those are only the ones that are named in the lawsuit. There could be others that haven't cropped up yet, or that had problems that Nvidia managed to suppress early on.
We were first to name the G84 and G86 directly, and then pointed out that others were bad too. Nvidia denied it, loudly. We said there were chipset problems, but Nvidia denied that. We said the 9x line was affected. Nvidia denied, but Apple however confirmed. Nvidia claimed it was only laptops that were affected, in an SEC filing no less. We pointed out that this wasn't the case. Nvidia still denies it.
Apple has also confirmed that, and the parts listed in the above links include, desktops. NUFI also names the exact chipset that we said was bad in our article as well. Nvidia would say we must have just gotten lucky and guessed right.
Everything we claimed and more has been confirmed, and more models have been added. We also claimed that desktop cards, and newer 55nm models were defective too, but those don't show up on the list. We guess that's just a matter of time.
Nvidia's chip design problem is a long-term one, and it takes months or years to start showing up. The technical explanation is in a three-part article, here, here and here. It may be long, but it is the only place you find an expanation. Meanwhile, Nvidia still claims it doesn't understand it.
We find this doubtful, mainly because of two things. First, Nvidia claimed to have fixed the problem in the 8-K filing mentioned above, and identified a specific fix. "All newly manufactured products and all products currently shipping in volume have a different and more robust material set." That sounds like it was pretty conclusive. The problem is that Nvidia's claim in that SEC filing just wasn't an accurate reflection of reality.
If you look at the three PCNs we were shown, written up here, here and here, they clearly show the first ship dates for the supposedly 'fixed' products as July 25, August 17 and August 17. Those dates are all after the SEC filing, and after the claim was made. Maybe there is a way to spin "first ship date", but it seems pretty clearly damning, and it does not seem to align with the statement, "All products currently shipping in volume." Maybe the SEC just didn't care, or was overworked, or something more sinister.
Back to the story at hand, the insurance lawsuit. Nvidia's prognostications have been comprehensively debunked, its claims have been shown to be untrue, and it still denies the problem. It refuses to come clean, say what chips are affected, what models of computers they were in, and who they were sold to.
When we first asked it, Nvidia claimed that it wanted to protect its customers, that is, the big OEMs, so it wouldn't say a thing. The people who buy Nvidia cards? Well, that was their own problem, as Nvidia was more worried about OEM feedback than end-user harm.
This tale really starts to spin down the rabbit hole when you consider that Nvidia has been using the same line on the insurance company. According to the complaint, Nvidia has been asking NUFI to pay up, but has not provided it with any information. You really have to read this to believe it, from the suit (Page 2, lines 12-19):
"Most importantly, NVIDIA has failed and even refused to provide material information about the Chip Claims to National Union, despite repeated and specific requests by National Union for that information. Instead, NVIDIA has provided substantial information about the GPUs themselves, held meetings to discuss the GPUs, and flooded National Union with technical data. That information, however, does not contain basic information about the Chip Claims that would allow National Union to evaluate whether any settlements are reasonable, whether coverage exists for those settlements, whether other parties may be at fault, or even what precise injuries the settlements are compensating the Chip Claimants."
Other than the part about flooding with technical data, this is exactly what Nvidia did to us humble hacks here at The INQUIRER. Lots of words, much chest thumping about its greatness and infallibility, but never answering the question.
The questions that Nvidia will not answer are not at all out of line. To quote Page 12, Paragraph 68:
"The information that NVIDIA has failed and/or refused to provide to National Union includes, but is not limited to, the following Material Information:
- Records showing the dates of manufacture of all affected notebook computers;
- Records showing the dates notebooks were shipped to end users;
- Records showing field failures to date, and the specific dates of those failures;
- Records showing the specific dates of repair of the affected notebooks;
- Records showing what component parts were replaced and when and why;
- Records showing any injury to component parts other than NVIDIA chips, including descriptions and dates of injuriers; and
- Documentation of settlement discussions, NVIDIA's estimation of claim exposure, and supporting documentation of any estimate."
Do these sound like things that are reasonable to ask before an insurance company pays out?
Reading the fine print in the suit, you can see there are several good reasons why Nvidia is playing dumb. The first is the civil lawsuits - if Nvidia admits anything to anyone, it will surely have a harder time defending against those civil lawsuits. You could say that what is going on is obvious to even a slow monkey, and we knew over a year before that some Nvidia GPU parts that were out were problematic.
Luckily for Nvidia, it is still in the dark, claiming that the science behind the cracking bumps is not understood. Somehow, every packaging expert talked to by The INQUIRER understood it, was able to explain it, and the explanations all matched up. Nvidia might want to hire one of these guys in the future to fill the glaring hole in its corporate semiconductor knowledge base.
Secondly, NUFI is claiming that Nvidia did not involve it in the negotiations for which it is being asked to pay Nvidia's settlements. Nvidia's insurance policy states that NUFI must be involved in any such negotiations, or at least that it must be given the opportunity to decline to do so.
It looks like Nvidia didn't provide NUFI with details, then sent it a bill with a big number on it. NUFI wisely declined to pay.
Things get even thornier on the next bit, on Page 25, Paragraph m(1). That says that property damage from "A defect, deficiency, inadequacy or dangerous condition in "your product" or "your work"" does not qualify for coverage under the policy. That sounds like defective chips might not be covered because they are defective, not because they broke for some unknown mysterious reason that Nvidia won't say.
In the end, the whole sad story backs up what we have been saying since the early days of this whole Nvidia "bad bumps" defective chips fiasco.
Nvidia is knowingly covering things up, denying information to the people who need it, and stonewalling everything. The list of affected parts has grown longer and longer, the list of affected OEMs by now basically comprises the list of all of Nvidia's OEMs, and now it looks like the company's hope of an insurance payout is under threat.
The only common thread is that Nvidia simply won't tell the truth no matter what. Every public statement it has made, from sworn SEC documents to off-the-record claims, has been disproven, but still it carries on with its version of events, almost as though its reality trumps all others by force of will.
While this story is fairly well fleshed out now, the ability of Nvidia to dig its own grave deeper seems to be never ending. The allegations in the NUFI lawsuit are so farcical at times that they stretch the ability to believe, unless you have been dealing with Nvidia before for a while. Then you'll recognise it as business as usual. µ
Note NUFI/AIG was contacted twice for comments on this story. We have not heard back from them either time, but will update or follow up should they respond.
Windows 10, 64-bit OS devices susceptible to rootkit attack
Malware suite likened to Stuxnet worm
Not the biggest fish out there
Redmond says figure is closer to the five million mark