Vendor References: What are they worth?

I’ve recently been working with quite a few vendors as we are in the process of doing a decent size (for us) infrastructure upgrade. We finally have it narrowed down to two solutions, but we can’t decide which one is the better fit. A large part of the indecision comes from the fact that we don’t know how these things work in real-world conditions vs. what the marketing material claims.

Any vendor worth the time of day should be able to give you references. These should be customers who have bought the same or similar products as the ones you are evaluating, and who can provide insight into their experiences. But you also have to keep in mind that you are only going to hear from the ‘good customers’.

So, how much faith can you put in a vendor reference? That’s up to you to decide, but asking questions and having a conversation with them should give you a feel for their experiences.

Here are some key questions that you can ask vendor references:

• How long have you been using the product?

You want to make sure you are talking to someone who is out of the honeymoon phase; us techies can definitely get excited when new hardware comes in – we want to dig in and start playing with it immediately. This can sometimes cloud our judgment when it comes to negative aspects – we don’t necessarily want to admit that the new toy doesn’t do something well.

If you are talking to someone who has had this thing in production for at least a few months, they have hopefully had a good chance to see how it performs, both the good and bad side of it.

• What was implementation like?

It might be the greatest product in the world that can do everything you want, but it has to be setup first. How was that process? Was it a breeze like most sales folks claim, or did you have to spend 4x as long as planned to get it up and running. Were there any gotchas along the way?

• Have you had to use support, and if so, what was the experience like?

A lot of vendors make the good revenue on support contracts – it is basically insurance. The fees can be ridiculously high, but let’s face it: if you find yourself calling support it is likely because you need it. If they have used support, how was it? I’m sure we’ve all had a large variation with vendor support. Sometimes you get the rock solid techs who close your case in 15 minutes and your problem is solved permanently, while other times you spend an hour on the phone just trying to get a ticket open.

I want to know ahead of time that if I am calling these guys at 10 PM at night that it will be a good experience.

• Is there anything about your environment that you think would be unique to why this product works so well for you?

Sometimes a solution might work wonders for an organization, but it might be because of a problem specific to them.

Suppose you are evaluating a backup solution, and you are given a reference. You call them up and they sing praises about this product, how it can do what they need it to and it is the best in class that they have tried. Dig a little deeper … what did they ‘need’ it to do? Well it might turn out that they have a bunch of Word documents that need to be backed up to a proprietary tape library, and only this vendor supports that library.

In the case above the product is a great fit for them because they need something that only that vendor can offer. If you don’t have the same need, the product might be junk compared to everyone else. Know the use case ….

• Is there anything you would have done differently?

I think we have all reflected on this after wrapping up a project: what could we have done and what should we have done differently? Sometimes things go off without a hitch (yay!), but all too often we hit bumps along the way. Take this opportunity to try and learn from someone else’s experience.

One other key point: don’t take too much of your references’ time. He/She is likely just as busy as you, and they are doing you a favor by taking the time to chat. Be sure to be polite, don’t press them for answers (especially if it is a question for the sales guys), and thank them for their time.