ORLANDO – Michael McIntire stood in front of a standing-room-only crowd, brandishing a long wooden cane with a Fritz style horn-shaped handle. He was smiling.
“This is our ‘engineer alignment’ stick,” he said. “We need to use this a lot.” McIntire was joking, of course, but if there was one point eBay’s chief architect made repeatedly over the course of his remarks at Teradata’s Partners 2006 users’ conference Tuesday, it was that database administrators and other IT staff have to be kept in check. As one of the world’s largest e-commerce companies, eBay’s growth is creating problems of scalability that make resources scarce.
According to McIntire, the key to making eBay’s IT architecture work is thinking of the online auction giant as a one billion-customer company. Though it’s trading volumes are considerable, that’s not the case today. It has 193 million registered users, lists 89 million items at any given time and offers more than 500,000 categories. Its gross merchandise volume reaches about US$139 million a day, or US$1,608 per second. On an average day, a piece of jewelry sells every six minutes on eBay in India, a bottle of wine every two minutes in France and a car every minute in the United States.
Given that most IT infrastructure has a 10-year lifecycle at best, McIntire said it was useful to think of a billion customer as the “end state” of how many users will one day place a demand on its systems.
“It’s a reasonable design goal,” he said. “It’s not a case of when we get there, it’s before we get there . . . we have to account for that growth.”
The data warehouse that gathers eBay’s sales and other information consists of a 64-node NCR 5400 primary production server and a 60-node NCR 5380 secondary production server, along with a 16-node co-existent system it gained through the acquisition of PayPal. Like a lot of organizations, one of the problems it faced is that querying parts of its data warehouse meant scanning large tables of information, even though its database administrators knew that only a small number of rows were relevant to the search. There isn’t a lot of point in having software scan through tables with 10 years of sales data when a user is only trying to get year-over-year sales trends from last year.
To solve this problem, McIntire said eBay has been making extensive use of Partitioned Primary Indexes (PPI), a tool Teradata included in version 2R5 of its database. PPI allows tables to be structured around frequently used queries. “We love this tool,” he said. “We could not have added to our platform without it.”
One of the toughest parts of making eBay’s architecture work, McIntire said, is figuring out what queries to process first and what can be moved down in the queue. Besides making use of Teradata Dynamic Query Manager and Priority Scheduler, the company has created strict business rules about what gets pushed out when. It has also adopted a common load-balancing trick: identifying “free time” when the system isn’t running at peak demand.
“We’ve balanced it against our international usage,” he said. “About 50 per cent of eBay users are in the U.S. The other 50 per cent are outside that, so it is possible to shift it around.”
In an earlier session, IDC analyst Henry Morris said businesses need to design their data warehouses and BI systems around “exceptions” – out of the ordinary events or transactions that could affect the performance of the organization.
“Exceptions are repeatable,” he said. “Think of a situation where an order comes in that exceeds a client’s credit limit. You have to have someone make a decision about whether you’ll allow that . . . this is the area that’s left to be automated.”
McIntire said eBay has worked hard to create the right business rules around exceptions, but the challenge is determining what kind of problems should be flagged, who should be notified and what kind of decisions should be made.
“I don’t think we’ve spent enough time as an industry thinking about how we notify people,” he said. “We can’t just tell them that the system isn’t working. We have to tell them what to do, what’s the escalation path.”
To help balance the workload on the data warehouse and other systems, eBay offers DBAs and other staff a dense view of active sessions taking place, with deep links into the SQL information and how much is in the queue. It also offers a “users-centric view” to business executives who may wonder why they aren’t able to retrieve information as quickly as they want.
“The phones stopped ringing,” after users got more insight into the system, McIntire said. “It’s data democracy and self-policing. They know what’s important to them, we don’t.”
Scalability isn’t just about hardware and software, McIntire admitted. Managing people so that they can be their most productive is an ongoing battle. Although the IT projects to be supported keep increasing over time, the number of people do not, even at eBay.
“For me to say, ‘I just need to grow (my staff) as fast as the data’ – that’s not going to fly,” he said. “It’s like painting wings on rocks. You can do it, but they’re still not going to fly.”
Comment: [email protected]
