Google’s Redundant, Fault-Tolerant System Worked with Cheap, Low-Quality, Failure-Prone Equipment

(p. 183) Google was a tough client for Exodus; no company had ever jammed so many servers into so small an area. The typical practice was to put between five and ten servers on a rack; Google managed to get eighty servers on each of its racks. The racks were so closely arranged that it was difficult for a human being to squeeze into the aisle between them. To get an extra rack in, Google had to get Exodus to temporarily remove the side wall of the cage. “The data centers had never worried about how much power and AC went into each cage, because it was never close to being maxed out,” says Reese. “Well, we completely maxed out. It was on an order of magnitude of a small suburban neighborhood,” Reese says. Exodus had to scramble to install heavier circuitry. Its air-conditioning was also overwhelmed, and the colo bought a portable AC truck. They drove the eighteen-wheeler up to the colo, punched three holes in the wall, and pumped cold air into Google’s cage through PVC pipes.
. . .
The key to Google’s efficiency was buying low-quality equipment dirt cheap and applying brainpower to work around the inevitably high failure rate. It was an outgrowth of Google’s earliest days, when Page and Brin had built a server housed by Lego blocks. “Larry and Sergey proposed that we design and build our own servers as cheaply as we can– massive numbers of servers connected to a high-speed network,” says Reese. The conventional wisdom was that an equipment failure should be regarded as, well, a failure. Generally the server failure rate was between 4 and 10 percent. To keep the failures at the lower end of the range, technology companies paid for high-end equipment from Sun Microsystems or EMC. “Our idea was completely opposite,” says Reese. “We’re going to build hundreds and thousands of cheap servers knowing from the get-go that a certain percentage, maybe 10 percent, are going to fail,” says Reese. Google’s first CIO, Douglas Merrill, once noted that the disk drives Google purchased were “poorer quality than you would put into your kid’s computer at home.”
(p. 184) But Google designed around the flaws. “We built capabilities into the software, the hardware, and the network–network– the way we hook them up, the load balancing, and so on– to build in redundancy, to make the system fault-tolerant,” says Reese. The Google File System, written by Jeff Dean and Sanjay Ghemawat, was invaluable in this process: it was designed to manage failure by “sharding” data, distributing it to multiple servers. If Google search called for certain information at one server and didn’t get a reply after a couple of milliseconds, there were two other Google servers that could fulfill the request.

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: ellipsis added.)

Hubel and Wiesel Are an Example that ‘Luck Favors the Prepared Mind’

WieselTorstenAndHubelDavidNobelPrizeWinners2013-10-06.jpg

“Dr. David Hubel, right, celebrating with his longtime collaborator, Dr. Torsten Wiesel, after they won the Nobel Prize in 1981.” Source of caption and photo: online version of the NYT obituary quoted and cited below.

(p. A25) Dr. Hubel and Dr. Wiesel liked to recall that their initial discovery about how vision works resulted from luck. Working in a tiny basement laboratory at Johns Hopkins, the pair struggled for days to coax brain cells in cats to respond to images of dark and light spots. Becoming increasingly frustrated, they waved their arms, jumped around, and, in a moment of levity, displayed images of glamorous women from magazines.

Then, as they shifted a slide in the opthalmoscope, a cell in the cat’s visual cortex suddenly started to fire. The edge of the slide had cast a straight, dark line on the animal’s retina. “It was what the cell wanted, and it wanted it, moreover, in just one narrow range of orientations,” Dr. Hubel said in his Nobel lecture.
They studied the cell for nine hours, and then, Dr. Wiesel recalled, ran down the hall screaming with joy.

For the full obituary, see:
DENISE GELLENE. “David Hubel, Nobel-Winning Scientist, Dies at 87.” The New York Times (Weds., September 25, 2013): A4.
(Note: the online version of the obituary has the date September 24, 2013.)


Samuelson Disputed Nephew Summers’ Praise for Milton Friedman

(p. A4) [Uncle Paul Samuelson and nephew Larry Summers] clashed over the fate of struggling mortgage giants Fannie Mae and Freddie Mac, which were bolstered by a government backstop in July 2008 and later taken over completely by the U.S. Treasury.
Mr. Samuelson found “strange and harmful” his nephew’s skepticism about the government backstop for the firms. Mr. Summers, a longtime critic of the two firms, wrote back that shareholders and management of Fannie and Freddie didn’t deserve taxpayer support.
Friction had emerged earlier in 2006, when Mr. Summers praised the late Mr. Friedman in a New York Times column. Friedman was “the most influential economist” of the second half of the 20th century, Mr. Summers said.
“For your eyes only,” Mr. Samuelson wrote to his nephew of Mr. Friedman, “I had to grade him low as a macro economist” and “stubbornly old fashioned.”

For the full story, see:
JON HILSENRATH. “A Close Bond and a Shared Love for ‘Dismal Science’; Correspondence Between Famously Brash Summers and His Uncle, a Nobel Economist, Reveals Flashes of Humility and Tenderness.” The Wall Street Journal (Sat., September 14, 2013): A4.
(Note: bracketed words added.)
(Note: the online version of the story was updated on September 15, 2013 and has the title “Letters Show Little-Known Side of Summers; Correspondence With Uncle, a Nobel Economist, Reveals Flashes of Humility and Tenderness.”)

Covey Was Amazon’s Entrepreneurial CFO

CoveyJoyAndSonTyler2013-09-25.jpg

Joy Covey and son Tyler. Source of photo: was posted on Joy Covey’s Google+ page: https://lh6.googleusercontent.com/-3CNdHv-7W3A/Thx9kuMHEPI/AAAAAAAABKs/lX9H2JlJ_lg/w763-h762-no/J+%2526+T+snowbird.jpg

(p. D8) As Amazon’s first chief financial officer, Ms. Covey helped take the company public and was an independent-minded advocate for Amazon’s plans to ignore Wall Street and invest for the future. That notion, radical in its day, was the foundation for Amazon’s growth into a $61 billion retailing and entertainment behemoth.

In its early days, Amazon prided itself on its unconventional hires, telling staffing agencies to “send us your freaks.” Ms. Covey did not have a traditional background. She dropped out of high school at 15 and worked as a grocery clerk. She attended Cal State Fresno and later Harvard Law School, where, she said, she did not fit in.
“We’d go to lunch and people would talk about their favorite 17th-century poets, and I’d be thinking, ‘Could I even name five poets? From any century?’ ”
But after joining Amazon in late 1996, when its annual revenue was less than $20 million, she thrived. She sold Wall Street the debt that the company needed to expand. The company went public on May 14, 1997, with an initial offering price of $18. Shares this week were selling for more than $312. Her own wealth is estimated at more than $200 million.

For the full obituary, see:
DAVID STREITFELD. “Joy Covey, 50, Top Executive in Amazon.com’s Early Days.” The New York Times (Sat., September 21, 2013): D8.
(Note: the online version of the obituary has the date September 19, 2013, and has the title “Joy Covey, Top Executive in Amazon.com’s Early Days, Dies at 50.”)

Gates Did Not See that Gmail’s 2-Gig Storage Would Beat Hotmail

(p. 179) About six months after Gmail came out, Bill Gates visited me at Newsweek‘s New York headquarters to talk about spam. (His message was that within a year it would no longer be a problem. Not exactly a Nostradamus moment.) We met in my editor’s office. The question came up whether free email accounts should be supported by advertising. Gates felt that users were more negative than positive on the issue, but if people wanted it, Microsoft would offer it.
“Have you played with Gmail?” I asked him.
“Oh sure, I play with everything,” he replied. “I play with A-Mail, B-Mail, C-Mail, I play with all of them.”
My editor and I explained that the IT department at Newsweek gave us barely enough storage to hold a few days’ mail, and we both forwarded everything to Gmail so we wouldn’t have to spend our time deciding what to delete. Only a few months after starting this, both of us had consumed more than half of Gmail’s 2-gigabyte free storage space. (Google had already doubled the storage from one gig to two.)
Gates looked stunned, as if this offended him. “How could you need more than a gig?” he asked. “What’ve you got in there? Movies? PowerPoint presentations?”
No, just lots of mail.
He began firing questions. “How many messages are there?” he demanded. “Seriously, I’m trying to understand whether it’s the number of messages or the size of messages.” After doing the math in his head, he came to the conclusion that Google was doing something wrong.
The episode is telling. Gates’s implicit criticism of Gmail was that it was wasteful in its means of storing each email. Despite his currency with cutting-edge technologies, his mentality was anchored in the old paradigm of storage being a commodity that must be conserved. He had written his first programs under a brutal imperative for brevity. And Microsoft’s web-based email service reflected that parsimony.
The young people at Google had no such mental barriers. From the moment their company started, they were thinking in terms of huge numbers. Remember, they named their company after a 100-digit number! Moore’s Law was as much a fact as air for them, so they understood that the expense of the seemingly astounding 2 gigabytes they gave away in 2004 would be negligible only months later. It would take some months for Gates’s minions to catch up and for Microsoft’s Hotmail to dramatically increase storage. (Yahoo Mail also followed suit.)
That was part of my justification for doing Gmail,” says Paul Buchheit of its ability to make use of Google’s capacious servers for its storage. “When people said that it should be canceled, I told them it’s really the foundation for a lot of other products. It just seemed obvious that the way things were going, all information was going to be online.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)

“Burning Bush” Depicts Communists’ Diabolical Harassment of Jan Palach’s Family

PauhofovaTatianaInBurningBushMovie2013-10-06.jpg “BURNING BUSH; Tatiana Pauhofova in Agnieszka Holland’s story of Prague under Communism.” Source of caption and photo: online version of the NYT review quoted and cited below.

(p. C6) The Polish director Agnieszka Holland’s magnificent docudrama, “Burning Bush,” is a three-part mini-series made for HBO Europe that remembers the Soviet crackdown in Czechoslovakia following the Prague Spring. It begins with the death in 1969 of Jan Palach, a Czech student who set himself on fire as a political protest, and follows the diabolical attempts of the Soviet occupiers to blacken his name by portraying him as a fraud and right-wing tool. The film’s depiction of the Communist regime’s relentless harassment of his family and its sowing of paranoia within the student resistance recalls the 2007 film “The Lives of Others,” about the Stasi’s operations in East Berlin. In the sophisticated worldview of “Burning Bush,” oppression may win in the short term, but the spark that ignites freedom movements, once lighted, can’t be extinguished.

For the full review, see:
STEPHEN HOLDEN. “CRITIC’S NOTEBOOK; Still Meaty, Film Festival Lightens Up.” The New York Times (Mon., September 30, 2013): C1 & C6.
(Note: the online version of the review has the date September 29, 2013.)

Prius Drivers Endanger Pedestrians and Cut in Front of Other Drivers

(p. B2) Jokes about BMW drivers being, on average, somewhat less than courteous are fairly common. They often run along the lines of, “Despite its good brakes, a BMW will usually stop with a jerk.” Sometimes the language is more colorful.
. . .
Paul K. Piff, a researcher at the Institute of Personality and Social Research at the University of California, Berkeley, has conducted a study linking bad driving habits with wealth.
. . .
In California, where the study was conducted, state law requires motorists to stop at crosswalks when pedestrians are present, allowing them to cross the road. Mr. Piff said his team selected a specific crosswalk to observe, then had a pedestrian appear on the edge of the curb as a car approached. As the pedestrian stepped into the road, a researcher marked down the driver’s reaction to the pedestrian. This was done with 152 drivers.
The team also watched a four-way-stop intersection over a week, noting how likely drivers were to cut in front of others when it was not their turn to go. In their observation of 274 cars, the researchers found that the more expensive ones were more likely to jump their turns in the four-way rotation, Mr. Piff said.
. . .
In the San Francisco Bay Area, where the hybrid gas-and-electric-powered Toyota Prius is considered a status symbol among the environmentally conscious, the researchers classified it as a premium model.
“In our higher-status vehicle category, Prius drivers had a higher tendency to commit infractions than most,” Mr. Piff said.

For the full story, see:
BENJAMIN PRESTON. “The Rich Drive Differently, a Study Suggests.” The New York Times (Tues., August 13, 2013): B2.
(Note: ellipses added.)
(Note: the online version of the story has the date August 12, 2013.)

The study discussed above is:
Piff, Paul K., Daniel M. Stancato, Stéphane Côté, Rodolfo Mendoza-Denton, and Dacher Keltner. “Higher Social Class Predicts Increased Unethical Behavior.” Proceedings of the National Academy of Sciences (PNAS) 109, no. 11 (March 13, 2012): 4086-91.

Brazilian Entrepreneur Inspired by “The Men Who Built America”

HangLucianoArrivesAtFlagshipHavanStoreInBrusque2013-09-29.jpgThe co-founder of the Havan chain, Luciano Hang, arrives at the chain’s flagship store, which is in Brusque, Brazil. Source of photo: online version of the NYT article quoted and cited below.

(p. 6) “My philosophy is pro-capitalism, so of course the best symbols for this come from the United States,” said Mr. Hang, who flies around Brazil on a Learjet to visit the nearly 60 stores in his chain, called Havan. “I tell people that we’re about freedom: the freedom to stay open when we choose, the freedom to work for us and the freedom to shop,” he added. “I know this can be controversial, but I think those who disagree with my approach are few and far between.”
. . .
The son of textile factory workers, descended from German and Italian immigrants, Mr. Hang said he admired European culture but preferred the United States. He said he was inspired by a show on the History Channel, “The Men Who Built America,” about industrial titans like John D. Rockefeller and Cornelius Vanderbilt.
“I couldn’t sleep after I saw that program,” he said.
His business model is partly based on Walmart, whose small-town origins he admires, as well as its method of turning economies of scale into low prices.

For the full story, see:
SIMON ROMERO. “Reshaping Brazil’s Retail Scene, Inspired by Vegas and Vanderbilt.” The New York Times, First Section (Sun., September 15, 2013): 6.
(Note: ellipsis added.)
(Note: the online version of the story has the date September 14, 2013.)

Larry Page’s Very Tough Love: “I’d Rather Be Doused with Gasoline and Set on Fire than Use Your Product”

(p. 171) Caribou took forever to develop. Part of the problem was that Larry and Sergey were so invested in the project. They adopted it as their primary email system and would often drop by to give criticisms and suggestions. Buchheit would often take a working prototype to the weekly Google product strategy meeting, where product managers submit their products to a human wind tunnel of executive criticism. Products have been known to die at GPSs; there are stories of teams entering the conference room, exhausted and hopeful after long hours of getting a demo just right, and Page saying, “You’re wasting our time” and ordering the project dismantled. Larry and Sergey liked Caribou too much to kill it but dished out very tough love. At one point Page told the group, “I’d rather be doused with gasoline and set on fire than use your product.”

Source:
Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. New York: Simon & Schuster, 2011.
(Note: italics in original.)

“Professors Have Lost the Courage of Their Own Passions, Depriving Their Students of the Fire of Inspiration”

WhyTeachBK2013-10-04.jpg

Source of book image: online version of the NYT review quoted and cited below.

(p. C4) Mr. Edmundson loves to teach, but he hates the conditions under which much teaching takes place today, even at an elite university like Virginia.
. . .
He knows the studies showing that students spend less time than ever on their classwork, and he writes of an implicit pact between undergraduates and professors in which teachers give high grades and thin assignments, and students reward them with positive evaluations. After all, given all the other amenities available through the university, the idea that “the courses you take should be the primary objective of going to college is tacitly considered absurd.”
. . .
Mr. Edmundson worries that too many professors have lost the courage of their own passions, depriving their students of the fire of inspiration.

For the full review, see:
MICHAEL S. ROTH. “BOOKS OF THE TIMES; How Four Years Can (and Should) Transform You.” The New York Times (Weds., August 21, 2013): C4.
(Note: ellipses added.)
(Note: the online version of the review has the date August 20, 2013.)

The book under review is:
Edmundson, Mark. Why Teach?: In Defense of a Real Education. New York: Bloomsbury USA, 2013.