Google/Dell Hardware Recall Nightmare...

I'm usually not one to blog about things closely related to work, but I've got to get this one off my chest...

Being the web systems architect for one of North-America's best universities, enterprise search is one of the biggest challenges my team and I are being faced with. So a couple of month ago, I made the call to try and replace the heavily customized ht://Dig based architecture we've had in place for many years with one of Google's Search Appliances: a GB-1001, to start with.

Being, like so many others, a big fan of most of Google's offerings, and being often confronted with the ongoing Google Mania when it comes to search (Why don't you do it like Google? Google does it this way. etc), I thought this was pretty much a no-brainer. I, of course, did have my concerns about the solution not being open source, even though it itself relies on it so, but I figured something made and backed by the oh so great Google could only be a safe bet. Right?

So we ordered one, received it, and as you can expect, we were pretty excited to start hacking at it to implement the engine in our web search strategy.

And that's when the madness started...

Dear Google Customer,

This is a notice to specific appliance customers regarding a configuration error that is present on a very small set of recently manufactured GB-1001 Google Search Appliances (GSA).

Since you are receiving this notice, the GSA with the serial number ([serial removed]) that was recently shipped to you has been identified as one of the affected appliances. These affected appliances may either display disk errors at boot time resulting in a non-functional appliance or boot without errors and appear to be functioning properly but then be at risk for having these errors at a later time.

Google will ship within days a replacement GSA to you along with a return label for the affected appliance. Please be sure to replace the affected appliance as soon as possible to eliminate your risk of incurring any production issues.

I'll spare you the rest of the email...

Not really what I'd expect after shelling out around US $20k, but this did not entirely come as a surprise to me, since that issue showed up in some of the research I had conducted before we received the appliance. And you know what? Stuff happens, and no one is ever shielded from a manufacturing defect.

So I chose to give them the benefit of the doubt, and just waited for our new appliance to arrive. I even signed up for a Google enterprise search seminar passing by Montreal, mostly to network with other GSA owners. Funnily enough, there was no mention of this problem as they were pitching their products to large IT companies. I didn't either. Why would I? We just happened to be among the few unlucky customers, among hordes of gleeful ones, that's all. Remember? Just a very small set of the machines were affected, after all.

So we got a new one between 1 and 2 weeks after being told it was on its way. The shipping time frame really had nothing to do with Google or Dell (hardware provider for the GSAs), but was simply because the box was held up at customs, like the first one had been. You'd think express air shipping would take this kind of things in consideration, but I guess not.

Because the return label that customer service had mentioned was actually nowhere to be found in the new box (really), we contacted Google Enterprise to enquire on how to proceed from there, to get the defective server back to them.

And lo and behold, this is what we got back from the customer relations department:

The appliance should have shipped with a return label. However, I will contact someone within our organization to send one to you.

However, we have just discovered that the replacement GSA that was shipped with the following serial number ([serial removed]) is still affected by the same issue and will need to be replaced. We will ship another GSA to you along with a return label for the affected appliance. Please be sure to replace the affected appliance as soon as possible to eliminate your risk of incurring any production issues. We have included a prepaid return label for you to use in returning the affected appliance. Please use the return label and the original shipping container to send the faulty appliance back to Google. Your replacement appliance has been shipped to the following address: [address removed]

If you have already put the appliance into production we recommend that you backup the configurations for import into the replacement appliance. We also recommend waiting to deploy your appliance to production until any affected appliances have been replaced. We deeply regret any inconveniences this may cause.

Best regards,

Google Enterprise Support

Fantastic, isn't it?!? Now that is enterprise service! Right? Right...

But, I thought only a very small set of the GSAs were affected?!?

Is that what we are to expect throughout our two year contract with them? Because if it's gonna be this way, I'm not sure I'm willing to bring any of it online anymore. Especially when my name is going to be attached to it in the eyes of the entire university... And that, when I was already eying the GSA clusters, considering the vast number and types of documents we have the potential to index.

Good thing none of it was in production yet.

Maybe writing our custom search solution wasn't so much trouble after all. One thing is for sure, it has served us well during the past 5 to 6 years or so, and despite certain scalability and character set issues, it sure has been a lot more reliable than the GSA's been so far.

I'm just stunned by it all, really.

PS: Why does it feel so strange to post this on Blogger, while using Analytics and AdSense?

Update: 2006-12-04:

I have now been assured by Google Canada that the new appliance has been verified not to have the same issue.

I'll wait until it's online before I celebrate this time though.

Update: 2006-12-05:

I have to give credit where credit is due, and say that Google Canada has been nothing but helpful since we got in touch on Monday. It's giving me back my confidence in the level of service I can expect.

Dare I say I'm getting excited to see the GSA arriving again? Yeah, okay. :)

Update: 2006-12-07:

The new GSA is now on the floor of my office and will be going to the server room tomorrow.