Supercomputing beyond academia

LAST WEEK WE BROUGHT YOU IDC analysis on the current HPC state, with some observations of our own. This time, let's see what are the chances for the expansion of HPC beyond academia, as well as some expectations for the near future of this - still more technically than financially - exciting field of computing.

According to IDC, High Performance Computing covers all computers - mostly servers - used for mostly computationally intensive tasks. So, it needs not be only a scientific or engineering tasks. A financial simulation of a stock market in myriad various conditions - something possibly quite useful these days - can be just as computationally demanding. Other "fun" stuff like algorithmic trading - if you believe there is an algorithm that can justify what was approved for Wall Street yesterday - even have to run on custom accelerators like FPGA.

Besides the financial institutions, other growth areas in HPC include many other fields that are fairly immune to the current crisis. For instance, oil and gas, with mostly seismic simulations in the resource search and exploration phase. This would - unfortunately - only increase as the competition and high prices force drilling in hitherto pristine places, requiring sophisticated modelling and simulations prior to the next drill to ensure that most likely there is oil there. It's not uncommon to have 2,000 node clusters in this field - Singapore and Malaysia alone have several, one of this size being half a mile from your reporter's house.

"Homeland security", where a nice shared memory cluster of a certain size could just as nicely pack all the desired data of all the citizens of a given country - or the whole planet - in an in-memory database, and enable lightning-quick searches or comparison operations on any of them at a keystroke, will obviously continue to be in demand by many powers that be. Some of the world's largest machines, publicly known and otherwise, will be in this sector.

Defense, both on research and 'operational' basis, was the kick-start customer for HPC years ago anyway. While some of their installations, especially in the US, do surface on the Top 500 supercomputer lists, many more are classified and never officially known to the public. Some other powers never mention any of their defense machines, of course...

General engineering for manufacture - whether car or aircraft engine simulations, or stress tests for a new type of concrete - were previously mostly done on individual workstations, with only the most complex stuff done on the central machine of some kind.

Now, however, both compute and visualise tasks - especially for the final assembly - may be done on a large visualisation cluster, where a gigantic, say, 50 or 100 Mpixel, multi-screen 'cave' or 'wall', often with some degree of immersion, is the display device.

Visualisation clusters differ from 'regular' ones in a sense that the nodes are basically souped-up rack-mount workstations, not plain servers, and that a single Open GL real-time visualisation is processed by all the machines towards a single ultra hi-res combined output device. Of course, the extreme demands on both the system and display side often mean these are the most profitable clusters due to the most "custom work" required.

Talking about simulation and visualisation, manufacturing isn't the only client there. A whole range of potential users - city governments simulating their daily traffic flow down to every car and pedestrian movement on all the roads, and impacts of adding or removing such facilities there; weather forecasting jobs which see meterological centres worldwide with some really nasty machines; or the current render-farm user base - film studios and such - relying this time on real-time movie scene and effects visualisations rather than slower frame-by-frame renders.

One common factor runs across most of these usage models, insulating them somewhat from the current financial crisis - heavy doses of R&D, with its 'strategic funding' implications. That applies not only to the government or academic use, but also to large corporations - especially those with a heavy engineering tilt.

After all, Boeing's 787 Dreamliner was completely simulated 'virtually' and, as you can guess, that's quite a lot of parts and putting together to do. Only the largest systems would suffice.

What about the near future - putting aside stuff like Cloud Computing and focusing on the real current demands? IDC estimates that, partly due to the factors above, the HPC market will continue to grow despite any crisis over the next few years.

While there are about a dozen "petaflops and above" super-duper-computer initiatives going on right now in US, EU and Asia - with more to come - the real growth is expected to be in the mid range. The divisional and departmental systems, in the range of few hundred grand, might provide the best value for money.

With even more cores per chip, such a 'simple' system may provide you up to 1,000 CPU cores in a single rack within a year from now, and I'm not talking about underclocked low-power blade versions - Nehalem Beckton 8-core and its AMD equivalents will provide that, with 4-socket 1U nodes or blades having 32 cores per node, and, well, 32 nodes getting packed into a rack. Not bad, eh? Well OK, the storage, networking and management infrastructure will probably have to go to another rack then...

IDC thinks that the highest growth in the near future will be seen in mechanical design, CAE - Computer Aided Engineering, weather, DCC (Digital Content Creation for multimedia incl movies) and of course "geo sciences", a polite name for relentless fossil fuel exploitation. All these should see 12-13 per cent average yearly growth.

After all, there still are many small and mid-size companies that use only desktop-level computation these days, but do require digital virtual prototyping or large scala data manipulation - obvious uses for HPC. However, both the HPC infrastructure, application and operation cost and ease of use will have to improve to attract more of such users to jump onto the supercomputing bandwagon.

And that's what our next HPC Watch round covers: the INQUIRER take on the hardware and software challenges when creating - and running - a cluster. µ