Here is a lesson every Web site manager may be taking away from Hurricane Sandy: It is probably not a good idea to put the backup power generators where it floods.
As computer centers in Lower Manhattan and New Jersey shut down or went to emergency operations after power failures and water damage Monday night, companies scrambled to move the engines of modern communication to other parts of the country. Others rushed to find fuel for backup power generation. In some cases, things just stopped.
“Suddenly, nobody could get online,” said Arianna Huffington, president and chief executive of The Huffington Post, which went offline about 7 p.m. Monday when the computer servers of Datagram, which distribute its work on the Internet, stopped working because of rising water in Lower Manhattan.
About six hours later, Huffington Post was online, but it crashed again several hours later. It was running again at 8 a.m. Tuesday.
As more of life moves online, damage to critical Internet systems affect more of the economy, and disasters like Hurricane Sandy reveal vulnerabilities from the sometimes ad hoc organization of computer networks. In places like Manhattan, advanced technology comes up against aging infrastructure and space constraints, forcing servers and generators to use whatever space is available.
Power is the primary worry, since an abrupt network shutdown can destroy data, but problems can also stem from something as simple as not keeping a crisis plan updated.
“If you have an e-commerce system taking an order from the Web, it may touch 17 servers, all in different locations,” said James Staten, an analyst with Forrester Research. One server might contain customer information, he said, while others work with logistics, product availability or billing. “If you don’t list them all as mission critical, you’re in trouble when disaster occurs.”
Big nationwide providers of Internet service, like Google and Amazon Web Services, were for the most part unaffected by the storm. Their cavernous facilities, holding more than 100,000 computer servers each, were located out of the storm’s path, and had extensive backup power generation. Amazon’s facilities in Virginia, which had been affected in a storm last summer, had no problems.
The largest telecommunications company affected by the storm appeared to be Verizon, which lost a considerable amount of old-fashioned wired phone service to the flooding. Bill Kula, a Verizon spokesman, said the storm surge from the hurricane flooded its central offices in Lower Manhattan, Queens and Long Island, causing power failures. Cellphone service at both AT&T and Verizon Wireless appeared to be less affected.
In the days leading to the hurricane, the carriers staged fleets of emergency response vehicles — trucks that act as temporary cell towers — in strategic locations along the storm’s edge. They also took safety measures, like installing backup batteries on cell sites and moving important equipment to less vulnerable areas. They advised customers to use text messaging instead of placing phone calls to use fewer network resources.
Large companies avoided problems by moving data and people out of town before the hurricane hit, using specialized nationwide service providers.
“They fan out across the U.S.,” said Nick Magliato, chief operations officer of SunGuard. “They move data from New York to Philadelphia, Philadelphia to Phoenix.” His company also provides offices for workers across the country, with computers that replicate their office systems and phones that have their personal numbers.
Peer 1 Hosting, like many other service providers, updated customers via its blog Tuesday. Robert Miggins, senior vice president for business development, said the main options for customers were to shut down in an orderly way, preserving data, risk staying online, or consider moving data to Peer 1’s computers in other parts of the country. “We’re telling them to back up everything,” he said, “sometimes they are dealing with us with nothing but a smartphone.”
Besides Huffington Post, Internet media sites including Gawker and BuzzFeed were taken out by the flooding at the Internet services company Datagram. The sites later recovered, sometimes with reduced service, by transferring work to other providers in different locations. Services remained intermittent, however.
Datagram executives could not be reached for comment. According to the company’s Web site, flooding in the basement of its facilities at 33 Whitehall Street, in Lower Manhattan, damaged the building’s electrical system. The site said Datagram’s computers were not affected.
Many companies affected by the storm and its aftermath built out distributed virtual networks of people, as well. Ben Smith, the editor of BuzzFeed, said that engineers worked throughout the night to shift the site to Amazon Web Services. Mr. Smith said that one man, Eugene Ventimiglia of Emerson, N.J., continued working from home as a tree crashed through his house.
BuzzFeed also rerouted readers to blogs on the Tumblr publishing service, in order to maintain social distribution of its content while its own site was down. Despite the power failures, Mr. Smith said, BuzzFeed still received about two million unique visitors on Monday.
Providers of critical Internet services to business in the area will be set back until full power is restored. A large building on Eighth Avenue in Manhattan, owned by Google which houses several other service providers as well, was relying on a 90,000-gallon tank of diesel fuel to run emergency operations, according to one of the building’s tenants.
“We’re good for about 72 hours at a full load of customers,” said Ron Sterbenz, vice president for marketing at Telx, which maintains data services for financial companies and communications providers from the facility. “We don’t know when the power will be back up.” Google would not comment on its building’s operations.
Another downtown building, at 75 Broad Street, had one generator in the basement, which was damaged by water. There is another generator, but it is on a higher floor.
“We’re on a very limited amount of fuel,” said Mr. Miggins of Peer 1, which maintains computers there. “We’ve got a truck full of diesel pulled up to the building, and now we’re trying to figure out how to get fuel up to the 19th floor.”