Scalability problems revealed in TPC-C tests

IN THE INFORMATION age, there is lots of information publicly available, and often valuable observations can be made just from analysis of this information.

This time I've taken official TPC-C tests information from www.tpc.org. TPC-C tests are a current industry standard for benchmarking of OLTP systems and databases - they are measured in tpmC - transactions per minute. For the first observation I've tried to compare the scalability of Power5 based systems with that of Itanium 2. To do it, I've taken from www.tpc.org the best available single-system (non-clustered) results for the different number of CPUs in the system. Results for Power5 1.9GHz and Itanium2 1.5Ghz are shown on the graph below.

Notes
- real (non-interpolated) data are marked with dots;
- currently there is Itanium2-1.6 available, but there are too few TPC-C results for it, and its scalability isn't expected to be dramatically different from 1.5Ghz one, so data on 1.5Ghz Itanium2 was used.

From this graph it seems that while Power5 scales perfectly up to 64 CPUs, doubling performance with doubling the number of CPUs installed, Itanium2 behaves much worse, especially when number of CPUs is high: when doubling CPUs from 32 to 64, it gains less than 50% improvement. This is also supported by the fact that 'clustered' 64-CPU configuration (16 4CPU boxes) delivers 15% better performance than non-clustered one (all 64 CPUs in one box) despite all communication expenses. This scalability issue probably can be attributed to the Itanium2 problems with memory bandwidth, as OLTP applications in general and TPC-C in particular are known to be extremely memory-bandwidth hungry.

Another interesting graph can be obtained from comparing results on similar hardware (Itanium2-1.5Ghz) using different OS/databases. If we compare Oracle10g-running-under-Unix-or-Linux with MSSQL-64bit-under-MSWindows (see graph below), we'll notice that on similar hardware Oracle seems to scale significantly better than MSSQL for 64CPUs. While Oracle gets 47% improvement with doubling CPUs, MSSQL gets only 36%.

Necessary disclaimers: by the very nature of such observations, they are mostly speculations rather than exact results: there are very many factors in consideration, and strictly speaking it is incorrect to come to a conclusion about system components comparing such different systems. For example, it could happen that the scalability degradation I've attributed to Itanium2, is in fact a great scalability advantage of DB2 over both Oracle and MSSQL. It just doesn't seem too likely). Saying all that, I hope there is still some value in such an analysis.

Sergey Ignatchenko