Intel has CRAC at datacenter rack management

PROBABLY THE MOST UNINTENTIONALLY FUNNY bit at IDF was during the session called Anatomy of a Server: Power-Thermal by William Hammond. William kept talking about datacenter power use, and specifically air conditioning. Computer Room Air Conditioning, or CRAC was constantly referred to, and it sounded like he was repeatedly telling me about the importance of crack. To a mind as infantile as mine, it was hilarious.

Seriously though, the talk was about the uses of power in a modern server and datacenter, along with ways to control it. It was quite sobering to realise that the main problem with a modern datacenter is power density. While that is a problem, efficiency and cooling, aka CRAC, is also problematic.

Some numbers first. Of the power used by the entire datacenter, according to Intel, about half goes to things other than the server, mainly cooling and other incidentals. Of that 50%, 50% is used by power conversion, VRMs, fans, der blinkenlights, and other things not used to directly crunch numbers.

In fact, of the power coming into the server, <30% is used by the CPU. Less than one out of three watts goes to the CPU, and how much of that is actually has something useful done with it? After leakage, idle and other things that do not flip bits that you want flipped, Intel estimates about 7.5% of the power that goes into the datacenter actually does what you are intending it to. That leaves another 92.5% to be picked up through efficiency and design changes.

To put things more into perspective, a sample rack filled with 1U and 2U servers was profiled. The numbers were about 894W and 1082W in, and with 40 and 20 servers, you get 35760W (~35.8KW) and 21640W(~21.6KW) per rack. This is about 1192W/sq ft and 721W/sq ft.

Other than basic math, it goes to show one thing, if you fill a rack, it will obliterate the 90W/sq ft most datacenters are speced for, and even the high end of common, 150W/sq ft, is woefully inadequate. If you have a 1500W hair dryer, the lesser of the two would be like putting 14 of them running at full tilt in a small closet, then putting an air conditioner at the top to keep things under control. You can make datacenters in the 4-500W/sq ft range, but they are said to cost as much as a fab to build. Not much sanity in that plan either.

So, what do you do? The easy part is to reduce the number of servers per rack. If you hit the 90W number, you can pack 7 1U systems or 6 2Us in there leaving your precious rack less than one third filled at best. Try explaining that to the CEO on the next tour.

The other, and Intel rightly thinks better, way is to be more efficient and to monitor things more closely. Fair enough, and it just so happens it has the technology to do that.

First comes CPU power usage. If you look at a server, it mostly is used during the work day, IE 8am to 5pm or so, and almost sits there at idle the rest of the time. Modern servers don't consume much less power during the boring bits of the day than at the peak loads. Intel gave numbers for a typical .NET web server showing about 170W used during the slow times, rising to 190W during heavy loads. CPU usage went from about 10% to sustained 85%+ during the peak time periods. There is room for improvement.

Enter DBS, or Demand Based Switching, Intel's name for the non-mobile version of Speedstep, or a similar technology to Cool and Quiet found in the AMD K8 CPUs. It can ratchet down the CPU speed and consequently power use during idle times. The examples shown had a server running in the low 80s to mid 90% power use without DBS, but falling to below 50% with a max of less than 80% when you turned DBS on. If you take the earlier example of 180W or so per machine, you can save about 80-90w with DBS, or enough to slot another 1U server in a rack.

While this may not seem like much, a hypothetical datacenter with dual 3.4GHz Xeon machines sporting 550W PSes, consume a lot of power. 876 machines with DBS turned on will save about 28% compared to non-DBS machines. If you are at $.10/KWh, over a year, with the lessened power distribution and cooling costs, you will save about 3MWh, or $300K a year. This is almost enough to get a CFO's attention.

You can either pocket the cash, or set up a denser rack configuration. If you go for the denser rack, Intel has a way to do this better. The old way was to look at the number on the power supply and take a guess. If you saw 500W, you may say that the server, on average will consume no more than 400W, and use that to pack the rack. Not a bad way to do things, but as servers get more and more efficient, if you guess 400, and it really does use 500, you will find out how expensive a cooked server can be. In the industry parlance, this is known as a career limiting move.

You can also make a rough guess by adding up the components in each system, and using that number. Not bad, but if you add in four more disks later on, problems may arise. To solve this, Intel came out with PConfig, a power configurator for servers. PConfig is a little program that runs at the BIOS level and reports what the server really will use. You get a real number based on what is there, not a guess.

This, when sent to a central management console, is what you would call a 'good thing', and can save time, aggravation, and careers. No more walking around, no more guesses, it does the work for you.

The next bit to make things more efficient is PSMI or Power Supply Management Interface. This reports the load on the power supply in real time. It can tell you that all your drives on your database servers are acting up, so power draw is at peak. It can even report these to the same console that PConfig talks to.

So, you may be thinking that having a remote view of your servers cooking to death in real time doesn't do much good. If you have active cooling in your, heh heh, CRAC, you can set it to turn the fans up as soon as the additional power draw starts, and before it gets way to hot. If the fans go up as the load does, your servers are much more likely to survive. All this in an integrated console, most likely miles away, will make your life easier. You can even sleep more on the job.

In essence, if you have a big feedback and monitoring loop in your datacenter, you can do things you never could before. You can also set the automated controls to do it for you, and get back to what you are paid to do, surf and sleep.

Since Intel makes chips mostly, there is more that can be done to make chips themselves more efficient. We have talked about throttling the CPUs down, but this is just smarter use, not necessarily attacking the problem itself. Intel has ideas here also, but they are a little further out.

One of the big problems Intel has been facing lately is CPU power use. The 90nm process is much maligned in the press for power consumption. While no 90nm figures were given, they did compare a 130nm Itanium2 to a 180nm version of the same chip. As a percentage, the 180nm chip has 90% of it's power used as dynamic power use, 5% as I/O and 5% as leakage. The 130nm had only 74% as dynamic, with 5% as I/O, and the other 21% as leakage. 4X the leakage, and one of five watts doing nothing related to computing. Eeek. Rumor has it that the 90nm is worse here, with number as high as 40% for non-computational use.

The numbers were a little vague in the presentation, but they gave a bunch of hints as to future leakage control techniques. Body Bias can lead to a 2-10x leakage reduction, and sleep transistors are shown to have a 2-1000x reduction. That is quite a range, but you get the idea that there are techniques which can make things dramatically better. If sleep transistors only give 10x reduction, that 40% will be less than the leakage at 180nm. It was manageable then, so it should be once again. If you can get to 100x, things look quite rosy again.

Further down the line at the 45nm node comes High-K gate dielectrics. These can reduce leakage by 100x or so. Throw in tri-gate depleted substrate transistors for even less leakage, and you have a potentially quite manageable problem. Like everything in the future, you can't say for sure how well it will work, or even if it will pan out at all. If two of the techniques work as planned, 45nm leakage may not be a big deal.

Overall, the talk showed that Intel was quite serious about power consumption. They are taking steps at the transistor, chip, server and datacenter level to attack most areas of the power consumption problem. While each one only accounts for a small decrease in energy consumed, they can all add up. Evolution, not revolution is the name of the game here. µ