Redundancy Allowed Google to Function with Cheap and Failure-Prone Hard Drives

(p. 42) . . . as the web kept growing, Google added more machines–by the end of 1999, there were eighty machines involved in the crawl (out of a total of almost three thousand Google computers at that time)–and the likelihood that something would break increased dramatically. Especially since Google made a point of buying what its engineers referred to as “el cheapo” equipment. Instead of commercial units that carefully processed and checked information, Google would buy discounted consumer models without built-in processes to protect the integrity of data.
As a stopgap measure, the engineers had implemented a scheme where the indexing data was stored on different hard drives. If a machine went bad, everyone’s pager would start buzzing, even if it was the middle of the night, and they’d barrel into the office immediately to stop the crawl, copy the data, and change the configuration files. “This happened every few days, and it basically stopped everything and was very painful,” says Sanjay Ghemawat, one of the DEC research wizards who had joined Google.
. . .
(p. 43) The experience led to an ambitious revamp of the way the entire Google infrastructure dealt with files. “I always had wanted to build a file system, and it was pretty clear that this was something we were going to have to do,” says Ghemawat, who led the team. Though there had previously been systems that handled information distributed over multiple files, Google’s could handle bigger data loads and was more nimble at running full speed in the face of disk crashes– which it had to be because, with Google’s philosophy of buying supercheap components, failure was the norm. “The main idea was that we wanted the file system to automate dealing with failures, and to do that, the file system would keep multiple copies and it would make new copies when some copy failed,” says Ghemawat.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipses added.)

To Page and Brin Search Speed “Was Like Motherhood, and Scale Was Apple Pie”

(p. 37) The average search at that time, Hölzle recalls, took three and a half seconds. Considering that speed was one of the core values of Page and Brin– it was like motherhood, and scale was apple pie– this was a source of distress for the founders. “Basically during the middle of the day we were maxed out,” says Hölzle. “Nothing was happening for some users, because it would just never get a page basically back. It was all about scalability, performance improvements.” Part of the problem was that Page and Brin had written the system in what Hölzle calls “university code,” a nice way of saying amateurish.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.

Google Started in Garage

(p. 34) On September 4, 1998, Page and Brin filed for incorporation and finally moved off campus. Sergey’s girlfriend at the time was friendly with a manager at Intel named Susan Wojcicki, who had just purchased a house on Santa Margarita Street in Menlo Park with her husband for $615,000. To help meet the mortgage, the couple charged Google $1,700 a month to rent the garage and several rooms in the house.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.

“We Just Begged and Borrowed” for Equipment

(p. 32) Google was handling as many as 10,000 queries a day. At times it was consuming half of Stanford’s Internet capacity. Its appetite for equipment and bandwidth was voracious. “We just begged and borrowed,” says Page. “There were tons of computers around, and we managed to get some.” Page’s dorm room was essentially Google’s operations center, with a motley assortment of computers from various manufacturers stuffed into a homemade version of a server rack– a storage cabinet made of Legos. Larry and Sergey would hang around the loading dock to see who on campus was getting computers– companies like Intel and Sun gave lots of free machines to Stanford to curry favor with employees of the future– (p. 33) and then the pair would ask the recipients if they could share some of the bounty.
That still wasn’t enough. To store the millions of pages they had crawled, the pair had to buy their own high-capacity disk drives. Page, who had a talent for squeezing the most out of a buck, found a place that sold refurbished disks at prices so low– a tenth of the original cost– that something was clearly wrong with them. “I did the research and figured out that they were okay as long as you replaced the [disk] operating system,” he says. “We got 120 drives, about nine gigs each. So it was about a terabyte of space.” It was an approach that Google would later adopt in building infrastructure at low cost.
Larry and Sergey would be sitting by the monitor, watching the queries– at peak times, there would be a new one every second– and it would be clear that they’d need even more equipment. What next? they’d ask themselves. Maybe this is real.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)

George Mitchell, Father of Fracking, Took 20 Years to Make It Work

MitchellGeorgeFatherOfFracking2013-08-04.jpg

“George P. Mitchell with a statue of himself at The Woodlands in 2007.” Source of caption and photo: online version of the WSJ obituary quoted and cited below.

(p. B3) George P. Mitchell turned hydraulic fracturing from an experimental technique into an energy-industry mainstay, making it possible to pump oil and gas from once untappable rocks and unleashing an energy boom across the U.S.

Known as the father of fracking, Mr. Mitchell died Friday [July 26, 2013] at age 94 at his home in Galveston, Texas.
. . .
“George Mitchell, more than anyone else, is responsible for the most important energy innovation of the 21st century,” said Daniel Yergin, vice chairman of consulting firm IHS and a Pulitzer Prize winning author on energy.
. . .
His first efforts at fracking, in the late 1970s, were expensive, and at times investors and his board of directors questioned the spending. But by the late 1990s the company had figured out the right mix of techniques and materials to produce shale gas economically, and began to do so on a major scale.
Devon Energy Corp. bought Mr. Mitchell’s firm in 2002 for $3.1 billion, combined the hydraulic fracturing techniques with horizontal drilling, and helped launch the current surge in oil and gas production.

For the full obituary, see:
TOM FOWLER. “REMEMBRANCES; George P. Mitchell 1919-2013; ‘Father of Fracking’ Helped Unleash U.S. Energy Boom.” The Wall Street Journal (Sat., July 27, 2013): B3.
(Note: ellipses, and bracketed date, added.)
(Note: the online version of the obituary has the date July 26, 2013, and has the title “REMEMBRANCES; ‘Father of Fracking’ Dies at 94; George P. Mitchell Helped Unleash U.S. Energy Boom.”)

Excite Rejected Google Because It Was too Good

(p. 28) Maybe the closest Page and Brin came to a deal was with Excite, a search-based company that had begun– just like Yahoo– with a bunch of sharp Stanford kids whose company was called Architext before the venture capitalists (VCs) got their hands on it and degeekified the name. Terry Winograd, Sergey’s adviser, accompanied them to a meeting with Vinod Khosla, the venture capitalist who had funded Excite.
. . .
(p. 29) Khosla made a tentative counteroffer of $ 750,000 total. But the deal never happened. Hassan recalls a key meeting that might have sunk it. Though Excite had been started by a group of Stanford geeks very much like Larry and Sergey, its venture capital funders had demanded they hire “adult supervision,” the condescending term used when brainy geeks are pushed aside as top executives and replaced by someone more experienced and mature, someone who could wear a suit without looking as though he were attending his Bar Mitzvah. The new CEO was George Bell, a former Times Mirror magazine executive. Years later, Hassan would still laugh when he described the meeting between the BackRub team and Bell. When the team got to Bell’s office, it fired up BackRub in one window and Excite in the other for a bake-off.
The first query they tested was “Internet.” According to Hassan, Excite’s first results were Chinese web pages where the English word “Internet” stood out among a jumble of Chinese characters. Then the team typed “Internet” into BackRub. The first two results delivered pages that told you how to use browsers. It was exactly the kind of helpful result that would most likely satisfy someone who made the query.
Bell was visibly upset. The Stanford product was too good. If Excite were to host a search engine that instantly gave people information they sought, he explained, the users would leave the site instantly. Since his ad revenue came from people staying on the site–” stickiness” was the most desired metric in websites at the time– using BackRub’s technology would be (p. 30) counterproductive. “He told us he wanted Excite’s search engine to be 80 percent as good as the other search engines,” says Hassan. And we were like, “Wow, these guys don’t know what they’re talking about.”
Hassan says that he urged Larry and Sergey right then, in early 1997, to leave Stanford and start a company. “Everybody else was doing it,” he says. “I saw Hotmail and Netscape doing really well. Money was flowing into the Valley. So I said to them, ‘The search engine is the idea. We should do this.’ They didn’t think so. Larry and Sergey were both very adamant that they could build this search engine at Stanford.”
“We weren’t … in an entrepreneurial frame of mind back then,” Sergey later said.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis between paragraphs added; ellipsis in last sentence, in original.)

It’s Hard to Be Consistent

TheFirstBillionIsTheHardestBK2013-08-08.jpg

Source of book image: online version of the WSJ review quoted and cited below.

(p. A13) Both Adam Smith and Horatio Alger would find something to like in the rise of T. Boone Pickens. “Boy geologist” Boone quit a promising job at Phillips Petroleum in the mid-1950s and built, over the following decades, Mesa Petroleum, a top North American independent oil and gas producer. Mesa found lots of oil and gas, provided jobs for hundreds of workers, and earned wealth for thousands of investors. During the same years, Mr. Pickens’s attempts to take over Cities Service, Gulf Oil, Phillips and Unocal made the whole oil industry shape up: His bids required the managers of each company to look hard at its practices and improve its shareholder returns.

Such accomplishments are the core of Mr. Pickens’s 1987 autobiography, “Boone,” which was updated 13 years later and retitled “The Luckiest Guy in the World.” In those books, Mr. Pickens’s political philosophy rang loud and clear. “I believe,” he stated, “the greatest opportunity lies in a free marketplace.” He warned: “There are powerful forces afoot trying to restrict that freedom in the interests of the vested and already wealthy. I am talking about a relatively small collection of corporate executives who would use the engine of American commerce for their own narrow ends.”
. . .
Now Mr. Pickens has new dreams — and he is lobbying Washington to make them come alive.
In particular, Mr. Pickens wants the federal government — through a mix of tax incentives, mandates and subsidies — to override the market and redirect the uses of natural gas.
. . .
“The First Billion” argues for this plan, along with recounting Mr. Pickens’s business ups and downs. The book is often entertaining, featuring the usual “Boone-isms”: e.g., “Show me a good loser, and I’ll show you a loser.” But readers unfamiliar with Mr. Pickens’s earlier memoirs may not realize that the new one represents a kind of bait-and-switch. Mr. Pickens’s standing to pronounce on energy matters was earned as a free-market producer. He is now using that standing to defy the market itself.

For the full review, see:
ROBERT BRADLEY JR. “BUSINESS BOOKSHELF; When Effort Is Energetic.” The Wall Street Journal (Weds., September 10, 2008): A13.
(Note: ellipses added.)

The book under review is:
Pickens, T. Boone. The First Billion Is the Hardest: Reflections on a Life of Comebacks and America’s Energy Future. New York: Crown Business, 2008.

“No Innovation Happens with 10 People in a Room”

EnglishPaulKayakCofounder2013-08-04.jpg

“Paul English, the co-founder of Kayak, said the company valued testing new ideas, not talking about them.” Source of caption and photo: online version of the NYT article quoted and cited below.

(p. B2) Q. You were a co-founder of Kayak nine years ago. What’s unusual about the culture?

A. We’re a little bit reckless in our decision-making — not with the business, but the point is that we try things. We give even junior people scary amounts of power to come up with ideas and implement them. We had an intern last summer who, on his very first day at Kayak, came up with an idea, wrote the code and released it. It may or may not have been successful, but it almost doesn’t matter, because it showed that we value speed, and we value testing ideas, not talking about them.
. . .
Q. What else?
A. We’re known for having very small meetings, usually three people. There’s a little clicker for counting people that hangs on the main conference room door. The reason it’s there is to send a message to people that I care about this issue. If there’s a bunch of people in the room, I’ll stick my head in and say, “It takes 10 of you to decide this? There aren’t three of you smart enough to do this?”
I just hate design by consensus. No innovation happens with 10 people in a room. It’s very easy to be a critic and say why something won’t work. I don’t want that because new ideas are like these little precious things that can die very easily. Two or three people will nurture it, and make it stronger, give it a chance to see life.

For the full interview, see:
ADAM BRYANT, interviewer. “CORNER OFFICE; Paul English; Ten People in a Meeting Is About Seven Too Many.” The New York Times (Fri., July 26, 2013): B2.
(Note: ellipsis added; bold and italics in original.)
(Note: the online version of the interview has the date July 25, 2013, and has the title “CORNER OFFICE; Paul English of Kayak, on Nurturing New Ideas.”)

“A Jigger of Asperger’s in the Mix”

(p. 11) Page was not a social animal– people who talked to him often wondered if there were a jigger of Asperger’s in the mix– and could unnerve people by simply not talking. But when he did speak, more often than not (p. 12) he would come out with ideas that bordered on the fantastic.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.

In the Plex Helps Us Understand Entrepreneurs Page and Brin

InThePlexBK2013-04-06.jpg

Source of book image: http://mastersofmedia.hum.uva.nl/wp-content/uploads/2011/09/intheplex.jpg

In the Plex goes from detail to detail of the values, actions and quirks of a large cast of characters who have been involved in the Google story. I did not find the book as consistently gripping as Isaacson’s Steve Jobs biography.
But some of the details help suggest new hypotheses, or test old ones, on important issues of entrepreneurship and technological progress. Some parts are revealing of the goals and methods of Page and Brin.
During the next weeks I will quote some of the more interesting passages.

Book discussed:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.