Google's secret of success? Dealing with failure | Tech News on ZDNet

Google's secret of success? Dealing with failure | Tech News on ZDNet: "On Wednesday, Urs Hoelzle, a vice president of engineering and of operations at the search giant, shed some light on how Google's data centers operate. Many people consider the company's operations expertise more valuable than the actual search algorithms that launched the enterprise.
Hoelzle spoke at EclipseCon, a conference for application programmers that's going on till Thursday here.
The way Google has been able to build out its computing infrastructure for millions, rather than tens of millions, of dollars is by buying relatively cheap machines. Looking at hardware costs, company engineers saw that purchasing a few high-end servers, with eight or more powerful processors, costs significantly more than dozens of simpler 'commodity' servers.
The trick is to make these racks of hardware operate in tandem and to ensure that the failure of one machine does not derail an operation, such as returning a search query or serving up an ad.
Consider a home PC, Hoelzle said. Optimistically, a consumer PC might crash once in three years from a software glitch or hardware problem.
'At Google scale...if you have thousands of PCs, you can expect one (failure) a day,' he said. 'So you better deal with that in an automated way, or you will have service outages.' "