The Transaction Processing Performance Council (TPC) has replaced its flagship benchmark for testing transaction processing performance.
The new benchmark, called TPC-E, is designed to replace the TPC-C specification, which had been updated with new versions several times since its 1992 ratification. It is designed both to reduce the cost and time associated with testing, and to make it more relevant to today’s firms, according to Michael Majdalany, whose acts as executive director for the Council.
“At the time that TPC-C was approved, we were in the early nineties and technology was at a certain stage,” says Majdalany. “Databases were fairly rudimentary and processor power was at a certain level, as were all the components of the system.”
15 years on, those things have evolved, but disk seek times haven’t. Consequently, vendors needed more disks to provide enough data to test the CPU. Testers have been running tests with 7000 disk drives. “That’s ridiculous,” says Majdalany.
Real-world data
TCP-E cuts the disk space needed to test the CPU by around 90%, and also introduces a foundation collection of code that makes it easier for vendors to produce benchmarking software for their own environments.
The test also features the use of real-world data (from the 2000 US census) rather than the TPC-C’s randomly generated input. This makes it more applicable to commercial environments. To that end, the new benchmark includes an updated transaction processing scenario. TCP-C used a warehousing scenario, whereas the new version tries to mirror some of the transactions that a brokerage house might experience.
“Order entry and bills of materials – twenty years ago that was a modern thing to do. That was the first big application for client/server,” says Long Zhu, manager of the database and reporting group for insurance firm First Canadian Title. He is also interested in the increased complexity of the back-end database parameters used by TCP-E. There are more tables, and more foreign and primary keys.
Long overdue
One company that will re-embrace public TPC benchmarks is Sun. The company ceased publishing its TPC-C results in 2001, but has been working closely with the Council on the new specification. John Fowler Jr, Sun’s primary representative to the TPC, helped develop TPC-E. He says that the company will begin publishing results again under the new specification.
Ashish Nadkarni, principal consultant at storage services and consulting company Glasshouse Technologies, speaks to clients regularly on infrastructure issues. “It was long overdue, and a welcome change to see them make their benchmarks more current. The OLTP and ecommerce environment have evolved over the years and the benchmarks did not,” he says.
However, he is concerned that customers are not using the TPC benchmarks in purchasing decisions. “I haven’t seen a single customer who has quoted TCP benchmarks to compare subsystems,” adds Nadkarni.
Zhu, who works mainly with Microsoft software, says that he finds the TCP specifications useful, especially when finding out how much performance a database version upgrade will give him. But he doesn’t directly use the benchmarks when making purchasing decisions, he admits.
Nadkarni hopes that vendors will make the tests relevant to customers using different database configurations. “I’d like to know how that sample implementation varies across databases at a functional level,” he says. “What are some of the database quirks? That may not be the premise but the customer would want to know.”
The TPC-C will not immediately be retired, said Majdalany. The Council also publishes other tests, including the two year-old TCP-App, which tests application servers and emerged from the TPC-W ecommerce testing benchmark. TCP-DS, which is due in about a year, will replace the TCP-H benchmark, which is aimed at decision support systems.