The Transaction Processing Performance Council (TPC) has replaced its flagship benchmark for testing transaction processing performance.
The new benchmark, called <bTPC-E, is designed to replace the TPC-C specification, which had been updated with new versions several times since its 1992 ratification. It is designed both to reduce the cost and time associated with testing, and to make it more relevant to today’s firms, according to Michael Majdalany, whose acts as executive director for the Council.
“At the time that TPC-C was approved, we were in the early nineties and technology was at a certain stage,” said Majdalany. “Databases were fairly rudimentary and processor power was at a certain level, as were all the components of the system.”
Fifteen years on, those things have evolved, but disk seek times haven’t. Consequently, vendors needed more disks to provide enough data to test the CPU. Testers have been running tests with 7000 disk drives. “That’s ridiculous,” said Majdalany.
TCP-E cuts the disk space needed to test the CPU by around 90 per cent, and also introduces a foundation collection of code that makes it easier for vendors to produce benchmarking software for their own environments.
The test also features the use of real-world data (from the 2000 U.S. census) rather than the TPC-C’s randomly generated input. This makes it more applicable to commercial environments. To that end, the new benchmark includes an updated transaction processing scenario. TCP-C used a warehousing scenario, whereas the new version tries to mirror some of the transactions that a brokerage house might experience.
“Order entry and bills of materials – twenty years ago that was a modern thing to do. That was the first big application for client/server,” says Long Zhu, manager of the database and reporting group for insurance firm First Canadian Title. He is also interested in the increased complexity of the back-end database parameters used by TCP-E. There are more tables, and more foreign and primary keys.
One company that will re-embrace public TPC benchmarks is Sun Microsystems. The company ceased publishing its TPC-C results in 2001, but has been working closely with the Council on the new specification. John Fowler Jr, Sun’s primary representative to the TPC, helped develop TPC-E. He said that the company will begin publishing results again under the new specification.
Ashish Nadkarni, principal consultant at storage services and consulting company Glasshouse Technologies, speaks to clients regularly on infrastructure issues. “It was long overdue, and a welcome change to see them make their benchmarks more current. The OLTP and ecommerce environment have evolved over the years and the benchmarks did not,” he said.
However, he is concerned that customers are not using the TPC benchmarks in purchasing decisions. “I haven’t seen a single customer who has quoted TCP benchmarks to compare subsystems,” added Nadkarni.
Zhu, who works mainly with Microsoft software, said he finds the TCP specifications useful, especially when finding out how much performance a database version upgrade will give him. But he doesn’t directly use the benchmarks when making purchasing decisions, he admitted.
Nadkarni hoped that vendors will make the tests relevant to customers using different database configurations. “I’d like to know how that sample implementation varies across databases at a functional level,” he said. “What are some of the database quirks? That may not be the premise but the customer would want to know.”
Fowler Jr. countered that Sun will concentrate on testing with its own storage subsystems, which it can more easily optimize to work with its own equipment. It may not publish results from certain database configurations if they don’t reflect well on the hardware, he admitted. “They are specialized tests that we use for development, for regression testing and analysis, and also for marketing,” he saud. “We will still tune it and tweak it and present our systems in the most favourable light to us.”
Comment: [email protected]
