CC: Can we begin by having Doug and Albert of Indigo and MFX describe what your current hosting situation is?
Caldwell: We outsourced our computing environment to Q9 Networks in the April to June time frame of last year. We have both our online environment and our corporate back end services
running at the Q9 facility.
Bird: We outsource to Fusepoint Managed Services as well, and we have two other data centres in the U.S. We are a business solutions provider, primarily focusing on financial and insurance verticals. Our core expertise is infrastructure and managed services.(MFX’s solution with Fusepoint includes managed infrastructure services (14 servers housed in Fusepoint’s data centre), data backup and a wide area network. They are collapsing four networks into one and Fusepoint’s facility will be the central point for all of Canada.)
CC: Explain the process you went through prior to moving to a request for proposals. What business units were involved in the legwork leading up to it, and did you have a detailed costing of what your expenses were prior to moving to outsourcing?
Caldwell: We had come through a merger of four different companies so one of the goals was consolidation of the data centre facilities. The other was that we are a retail organization with about $800 million in annual revenues.
We had looked at the cost of running a data centre and it can get pretty prohibitive when you look at being able to provide the high level of availability. The online environment demands the 7/24 enterprise class data centre environment and to build it yourself is just prohibitive. So we looked at a couple of things: high availability, secure environment and consolidation of the infrastructure itself.
In planning for the RFP itself, the first step we took was to ask: What do we need? The answer is an inventory of equipment. You need to provide the bidding vendors with an inventory of what you have, which is everything from space to power to networking requirements as well as what your expectations are from a service-level perspective.
The facility itself is something that became very important to us. The real test was the power outage we had last summer. From a power standpoint you need to make sure you have redundant power coming in to the data centre itself and from multiple power grids. You need to make sure you have battery backup and diesel backup. Once you have diesel backup the question is how long the generators run for? How do you replenish fuel in those diesel generators? Where is the physical facility located and how accessible is it to those types of things? Everything from normal operation to crisis mode is really what you need to look at in terms of your requirements.
Looking at space and power helps those responding to the RFP determine whether it is a house or a mouse. The other thing that’s important is to determine your growth requirements. When you’re looking at going into a facility you need to be able to share with the vendor whether you’re going to grow five, 10, 50, 100 per cent.
The other area we looked at was billing requirements. I need to break it down across different parts of the organization, so whether it be the online business, retail, finance or whatever, can you help me break down the elements of the cost?
Finally, what is the reporting you want to have out of the facility? The kind of reporting they provide is important in terms of being able to prove they are delivering against the service level you have.
CC: Did you have a good idea of what your expenses were before you went into the arrangement?
Caldwell: We looked at the physical rent of the facilities we had so we knew how much it was costing in terms of maintenance on UPS (uninterrupted power supply) and network connections coming into the facility. In terms of just the pure rent of the physical floor space itself we were able to have a good baseline of what it was costing. The one thing we had to make sure we took into account was that we were providing an inferior service to date to our business and paying a premium or it.
In terms of the business units that were involved, the project very much came from the audit committee and the executive team. The original requirements were that we needed to be able to sustain a crisis-type environment, be it fire or whatever, and our current facility was not meeting the standards of what the senior management of the organization needed to provide. We had to step up to that.
The second business unit we worked with was from a legal standpoint, in terms of our internal and external counsel, to understand what would be the components of getting them involved early on from scheduling of their resources to the parameters of the framework needed.
Bird: Some of the decision to move to Fusepoint was based on us looking for an alternate hosting vendor. Key aspects for us were 9/11, certainly in the U.S., and how secure the data centre facilities were. We had a power outage similar to what happened here and we had one that happened that was not related to any other event, but related to a transformer power outage in our building, so we lost the ability to trade on the stock exchange. We also wanted to look at the overall hosting needs as a business for MFX as well as disaster recovery and business continuity. So when we looked at the combination of those things we really focused on finding something that was not in the downtown corridor Fusepoint matched that parameter (Fusepoint’s data centre is located in Mississauga, west of the Toronto core) and we could grow our infrastructure. MFX has seen tremendous growth and we wanted to have a place where we could grow our business and host it in a stable and redundant environment while we were going out and soliciting business from customers. We needed to know we were matching their customer service level agreements and we needed a data centre that could have 24/7 365 redundancy that was robust, stable and secure.
One of our key parameters was security. Some of the goals we had in mind were to reduce some of the infrastructure spend we had. Building out redundant infrastructure is (expensive). We have two data centres in the U.S. and we were looking to leverage along the price point to create a northeast hub. Where we were hosting at the time was downtown and it was not serving our needs.
When we looked at the disaster recovery model we were trying to achieve, it was about getting outside the city limits, and looking at a provider we could partner with and grow. Fusepoint seemed to have the people and the process, in terms of service level agreements and operating environments. If we were just looking for a hot pipe to the Internet and some backup power we could find that in several places. What we didn’t want to do is have the same staff having to manage the hosting facility as well as managing our key clients. So we handed that over to our partners at Fusepoint and we focused on our core competencies — bringing our products to market and servicing our current clientele.
CC: Was cost a big issue?
Bird: Yes, it was.
CC: Had you been doing comparisons as to what it had been costing you?
Bird: Yes. And there are obviously economies of scale — the more infrastructure you’re hosting, the better price point you’re reaching.
CC: What are the main things companies need to do in terms of homework to be ready to make a decision to go to hosting?
Offley: Clearly for any business it’s an important decision. You have to go through the due diligence process. In terms of looking at a provider, I think the market has evolved over the last couple of years. In the early days people just looked at the facilities. From my point of view that’s almost table stakes in terms of being a provider and I think as a company, you have to look at it in terms of whether or not the provider can grow with you. The physical space perspective and the people and process are important too. You have to spend time with the people — do they have standard operating systems? The other area is security. Do they have the security to deal with worms and viruses that we’ve seen come across many Canadian businesses? From my point of view, as this industry evolves, people are not just looking at facilities but whether or not the SLA (service level agreement) matches and how the SLA is also built around having that critical application available.
Arafat: First and foremost the company should decide what exactly they want to outsource and what they want to keep in-house. I think this is a critical decision and one that’s going to affect the decision of who the business actually goes to. That can range from ‘I want to hand the keys of my IT department to the service provider’ on the one extreme to ‘I just want the infrastructure component’ on the other extreme. There’s a whole host of things in the middle that people can look at and they should have a clear picture of where the line of demarcation is between the responsibility within the
organization and the responsibility of the service provider. Once that’s decided you want to go off and find the most suitable service provider and there are many aspects to this, as Robert pointed out, other than just who has the biggest generators. The key is infrastructure due diligence. The company is key and the personnel and processes and so on. Security is also key, as is financial stability. Is the service provider going to be around next year? If you look at the experience in the U.S., for example, a lot of companies have been insourcing because they got burned one too many times with Exodus and then Sprint exiting the business, so that is absolutely key. Location is also important. There are people who absolutely want to be in the downtown due to proximity to their own infrastructure. Indigo is a good example — they are two or three blocks from us and they need to be able to support their prime infrastructure. Or you may have a disaster recovery situation where a customer may want to be in a different part of town and more importantly a different province, so we have customers who are hosted in our Calgary centre.
CC: Before the time comes to write the contract, Denis, do you get involved with advising clients on what they need to do to get ready?
Chamberland: If everything goes well, the lawyers will be involved early in the process rather than later. (Laughter)
I’ve heard a lot of good buzz words here such as “”due diligence”” and “”developing the business case”” and so on. But in business relationships that don’t go well, in my experience it’s been when contracts are developed in three weeks when it should take three months or more to make them happen. When you look at the primary areas, there are the pure legalities, your very important provisions; you have the minor provisions, and the others that have no quantitative impact on the parties. In many cases I’ve seen contracts where parties have wanted to sign very quickly, in a week — no negotiation no discussion, and some of the very important provisions are not in there. That’s pure legalities, but more importantly from your perspective, in outsourcing, as we all know relationship management has become very important. Ten years ago people got the best SLA they could, if they even did one. Then they would recognize a few years later that the relationship and that the value proposition had been sub-optimized and they wondered why.
It was because they were treating everything like a commodity product. In the past few years everyone has recognized that when you have a contract that’s a good thing, but you still have to have the dedicated resources available to manage the relationship so you’re just not throwing everything out there and abandoning to it if you’re not paying attention to it. The RFP can be used to develop the relationship right up front and start managing it, not when the operations begin, at the contract execution, but on the front end. It allows the parties to get to know each other a little bit better, not just the personal side of things, which is not unimportant, but to find out if you are talking at cross purposes. Very often, especially if the customer has never outsourced anything before, they don’t understand the terminology the vendor is using. When CIBC did their outsourcing deal in 2001 it was exactly that situation. I think they did a lot of due diligence and it was a very thoughtful process but it still moved very quickly.
I was involved in that from one angle but it took a year after that for the parties to realize in the first few months after the contract had been signed that when you use a certain word it meant something entirely different. There were huge cultural gaps. It wasn’t unsurmountable and they are currently doing very well but it took a long time to get there. These are all things that relate to relationship management and things that if you have RFPs you can really dig out that information and make it work. It’s a relationship building experience.
Offley: It’s an opportunity to date before you get married. With MFX we went through a long process and they were really testing us along the way, so then when it came to contract we felt we really know each other.
CC: How long was that process?
Bird: About six months and there were several teleconferences and visits.
Offley: One of the areas that MFX really focused on was the security side and we are finding that more and more.
CC: And for Indigo, was it about the same amount of time, Doug?
Caldwell: From the time we did the RFP to the time we started moving stuff in to the data centre, it was about five or six months.
CC: Does that time period seem adequate, Denis?
Chamberland: It depends entirely on the arrangement the parties are considering.
CC: A U.S. survey looked at 85 long-term outsourcing deals, and showed of the ones with very detailed contracts, most of the expectations were met on the client side 75 per cent of the time. With looser contracts the survey showed most expectations were met zero per cent of the time. So, Denis, what makes for a good contract? What are the components of a solid outsourcing or managed services contract?
CHAMBERLAND: It’s all the things we just talked about. If the parties get to know each other they have a better opportunity to address all their relevant issues and to also delegate the appropriate resources. Big or small you have to have people assigned to it. The other thing is if the process is moving a little bit more slowly the appropriate people are going to become involved at the right time. One thing that often happens in outsourcing arrangements, where there is a recognition that you need to manage the relationship, very often people are brought in a week or two before contract execution and told, “”This is what the contract looks like today and you’re going to be managing it.”” Inevitably after the contract execution, issues do arise. But it’s how the relationship is dealt with and how much flexibility there is in the contract. More than likely, the vendor being the expert in the services they provide, they will have the appropriate people onside and if the customer brings in fresh people who have never been involved in negotiating the contract it can imbalance the relationship from the beginning. Also, more than likely these people will not have received any coaching on what it means to manage a relationship. If they are very technical people they have great skill sets but you need to have different skill sets to manage a relationship. Something I see all the time is when there are glitches or things appear as though they might become glitches and they think if they raise it (at the beginning) it might cloud the relationship so let’s put it aside and the vendor will pick it up and things will get worked out.
CC: For you, Doug, was having a team managing the relationship important?
Caldwell: When I looked at the overall contract itself, one of the most important things was the schedule of the contract. The harsh reality of it is you need to be prepared to walk and that’s a negotiating point. So if you already have commitments you’re up against you’ve basically lost your negotiating ability from that standpoint. Getting a contract started very early on is really important and making sure you’ve got, as Osama said, the demarcation points of who does what. We talked about service levels and they are really very important and it’s how do you measure it and the reality is what are the damages associated with it? You have to have teeth to those service levels.
There are also issues around insurance and liability. You are now in a facility that you don’t own and there are other customers in the facility itself so you have to respect the fact when you are in that facility and if you do damage to somebody else’s equipment that’s your responsibility. And its also your equipment in somebody else’s facility so you have to be able to ensure that, but also make sure that your partner, like Q9, has sufficient insurance as well.
There are different levels of insurance and each one’s got a price associated and in that case other members of the company need to get involved with the discussion. In our case it was the CFO who was involved. Definitely the responsibilities: what we can and cannot do in terms moving equipment in and out of the facility, who actually does the wiring, what sort of network pipes do you need to have in place using part of the facilities infrastructure itself.
Certainly there are also the terms and termination of the contract itself. It is not very pleasant at all to have to move a data centre, especially when it’s an online environment. It’s like changing an engine in an airplane at 30,000 feet. You’re trying to keep the engine running and now you’re moving things. You don’t want to do it every couple of years so the term and contract is important.
CC: Al, from your perspective what are the important components?
Bird: As Denis said you really need to understand where the demarcation line is. Part of the reason our engagement took six months to get off the ground was because there was significant negotiation. The current hosting facility we had was somewhat loose and while it wasn’t zero per cent satisfaction, it was pretty close. So our legal team and Fusepoint’s, while we were negotiating with them, there was a lot of discussion about what manner they would manage and being in effect the managed service provider to our customers. Negotiations stalled a little bit when we were discussing the price points for the services we were looking for. The legal teams kept at it and we actually came to an agreement that was mutually beneficial to both companies. One of the key elements for us was I didn’t want a data centre that would tell us yes we can do that, yes we can do that just because they wanted the business. There were points in time where they came back and said, “”No, we can’t or we will not do that.”” We really wanted somebody who was going to stand up for themselves because what it meant to us is the service level agreements are solid, they are locked in stone, their operating procedures, their change management policies were really nailed in stone as we are for our customers. I think the RFP process is significant in terms of understanding the customer, understanding what the data centre has to offer and understanding the point in time when it flips from their responsibility to yours is absolutely paramount.
CC: What was the most important lesson you learned from your previous arrangement before going to Fusepoint?
Bird: We learned about loose contractual terms and I think the ability for the data centre to handle our growth. We were growing faster than they were capable of keeping up with it. It’s about their ability to support our growth and sustain their operating environment as our operating environment expands. For example we had extended periods of outages and for an e-business solution provider that relies heavily on the data centre presence in a global economy — people in Asia or the U.K. are trying to access our environment at 3 or 4 o’clock in the morning U.S. or Canadian east coast time and it’s not available — so those are the calls that get people up at night. So I think when you’re looking for a data centre, they should be seeking to understand you; they’re not saying OK, give us your infrastructure, put it in here. They’re asking, What is your expectation of growth? How much environment space do you need, what are your service level agreements for your customers?
Caldwell: When you talk about understanding the business model, in retail there’s a system-freeze period of time that for us is mid-October to mid-January. Other businesses, if you’re in tax preparation it’s Feb, March, April. It’s important that those things are clearly understood so there are no major changes happening in the data centre.
CC: Experts advise companies should spend five to 10 per cent of the value of the outsourcing contract in managing it on a day-to-day basis. Can you talk about managing the ongoing relationship with your provider?
Caldwell: First thing, when I look at something as important as the hosting of our environment it’s establishing a senior relationship. It’s very important to establish that relationship and it’s certainly paid off on multiple occasions. The other part of it is sharing your plans as early as possible. We do strategic plans on an annual basis and being as open and honest with your partner in terms of sharing those business plans. And I share the entire plan, I don’t care if it’s relevant or not. The more they know about us, I think, the better aligned we are in terms of working together.
There are also regular meetings the teams have. If it’s around a specific project or a change in the environment, there is a monthly status meeting in terms of reviewing the service level.
The thing I will always go back to is last summer when we had the power outage. I got a call at home at 10 o’clock Friday morning from Q9 saying “”just wanted to make sure you guys are alright.”” They could see from within the data centre that everything was running fine, but they wanted to know if they could help at all and that’s the willingness to go beyond just contracted services to work together. I was impressed with their willingness to do whatever they could to help out the business.
The other thing mentioned by Al is the honesty of what you can or cannot do or what you weren’t willing to do. I respect that honesty to say that we can do this or we can’t do that.
CC: So would you say the five to 10 per cent is accurate?
Caldwell: I think it’s more in the five per cent range. But is that five per cent of a 40 hour week or five per cent of a 120 hour week?
Bird: In the early on stages when you have data centre infrastructure staff moving services and hardware in and out it’s probably in the higher end of 10 per cent. The value proposition is when you get further out and operating costs and contractual arrangements are flattening out. I think we have come more in line now more to the five per cent range. It gets back to what Doug just mentioned: Early on there’s going to be more cost expenditure because you’re going to have to spend time and resources on it. Once again there has to be an understanding of where you want to be in terms of cost moving forward, on a monthly basis. That model starts to flatten out.
It’s like anything else, there’s a ramp up when you move in and then it flattens out. Then the environment is stable and settled and you are able to scale back and realize the benefit. We’re at about five to six per cent now.
CC: And if it’s the same sort of arrangement (As Indigo/Q9) Do you have regular meetings or just meet on an as-needed basis?
Bird: We’re getting ready to move more of our footprint into Fusepoint. I have had two or three meetings with them reviewing service level agreements. I think it’s always that due diligence when the two sides are coming together to make sure there is the right negotiations with the right understanding.
CC: Do you have a dedicated account manager?
Bird: We have a dedicated financial/insurance account manager.
CC: Denis, in your experience what are the scenarios you’ve seen in terms of the resources allocated to managing the relationship?
Chamberland: The numbers are all over the place, but the five to 10 per cent is a figure that comes out of Vantage Partners. One question I had for you Albert and Doug is have you ever thought of attaching metrics to the governance or relationship management issue? What I’ve noticed in the past is that you need to have a proper governance structure in place. Things can get off to a good start, but very quickly what you realize is if the meetings aren’t scheduled people say we don’t need to meet because things are going well and very soon you have a governance structure that’s not in place any more. You need to have an internal audit done of how often the meetings are held and minutes should be taken so you can go back and consult them. It adds a lot of rigor to the process — it adds to that five per cent figure or whatever it might be. You have to have consequences, that’s the bottom line.
Arafat: We’ve talked a lot about the contact which is a very important aspect, but one thing we haven’t touched on is customer references which can’t be understated. It’s one way you can avoid the trap from the very beginning very easily. Look at five or even ten — most service providers are willing to give you as many references as you want, if they are worth their salt.
Offley: And you don’t just want to pick the canned ones, you almost want to offer a selection they can choose from.
Arafat: Exactly, a lot of customers publicize or you may know from other intelligence or there is a customer in a similar business or similar size that could be asked. Say, “”I want to speak to so and so because I know they are your customer and I want to know what their experience is.”” Look at things such as whether a service provider has under promised and over delivered. Have they delivered on everything they said they would? What were the gotchas that the previous customer had. I think this is a very important pre-cursor, because legally you can do quite a bit, but nothing beats the human element and reference element.
Chamberland: Let me ask you though, would you make available not only a pool of references from current clients and former clients and make available also a reference for a relationship that didn’t go well? I tell my clients when asking for a customer that yes, you want to hear the happy stories, but also get a couple from the unhappy. Every company has some unhappy customers
and the real issue, when the customer calls and hears the negative stories, there’s a rationale behind it, there’s some explanation, but if you hear there was a disconnect between what the parties thought they were getting it’s definitely legitimate.
Arafat: In our industry I think this is easier to do because it is a very close-knit community. Everybody knows everybody. If there was a unhappy story, you’re probably going to find the person at the other service provider. And that’s one of the key questions: Where were you before you were with Q9 or Fusepoint or whoever and why did you leave?
Offley: There are always going to be issues in a relationship and we’re all kidding ourselves if at some point we don’t think there’s going to be some issue in the business that comes up and what we’re finding is customers want to know, more and more, how you dealt with that. It may just be that you didn’t set up those early demarcations or lines of communication. We try and give balanced references where perhaps we didn’t structure things right but we feel it’s important to try and get some balanced references as much as the canned great reference.
CC: What do you think about what Denis said about assigning metrics to the number of times you meet and what your expectations are around those meetings?
Bird: We do have formalized documented meeting minutes and we discuss them. We also have an issues list when we are tracking items of interest or up-and-coming things that have to happen within a sub-project. I think it’s absolutely paramount. You have to keep fresh in both party’s minds what the current issues are on the table. As far as data centre service-level providers, everybody is struggling for five nines, and that’s impressive, but you also have to have the people in place and that’s very important.
Caldwell: You can put metrics in, but again there is motivation for both parties to meet. As a customer I’m interested in making sure I communicate my plans and from the service provider there’s always that business development opportunity for new business, so if anything, there’s motivation to get together.
Offley: One of the things we’ve made investment in is automating. We provide a dashboard on their project and they can log on and see how far they are in their implementation.
We are also trialing software where, after there had been an account review meeting, we send out an e-mail to the client and say ‘Do you think we met the objectives of this meeting?’ We use that as an audit tool.
The other thing that is important to us is that this is a service business; it’s not a facilities business. It’s more about the people and process than it is about the facilities.