Super Bowl blackout caused by equipment to prevent outage

NEW ORLEANS >> An electrical device that had been installed expressly to prevent a power outage caused the Super Bowl blackout, the stadium’s power company said today as it took the blame for the outage that brought the game to a halt for more than a half-hour.

Following the announcement, the manufacturer of the device, known as a relay, released a statement implying that the problem was not with the part but with how it was used.

Officials of Entergy New Orleans, a subsidiary of New Orleans-based Entergy Corp., said the device, called a relay, had been installed in switching gear to protect the Superdome from a cable failure between the company’s incoming power line and lines that run into the stadium.

The switching gear is housed in a building known as “the vault” near the stadium that receives a line directly from a nearby Entergy power substation. Once the line reaches the vault, it splits into two cables that go into the Superdome.

Company officials said the device performed with no problems during January’s Sugar Bowl and other earlier events, but has been removed and will be replaced. All systems at the Superdome are now working and the dome will host a major Mardi Gras event Saturday night, said Doug Thornton, an executive with SMG, the company that manages the stadium for the state.

The power failure at Sunday’s big game cut lights to about half of the stadium for 34 minutes, halting play between the Baltimore Ravens and San Francisco 49ers. The FBI had ruled out cyberterrorism as a cause.

Entergy’s announcement came shortly before officials appeared before a committee of the City Council, which is the regulatory body for the company, to answer questions about the outage.

Not long after the power company’s announcement, the manufacturer, Chicago-based S&C Electric Company, released a statement saying that the blackout occurred because the so-called trip setting on the device, “as set by the system operators,” was too low to allow the device to handle the electric load that was coming in.

The statement did not name the “system operators,” but the equipment was owned and installed by Entergy New Orleans.

“Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power,” said Michael J.S. Edmonds, vice president of strategic solutions for the company.

In a follow-up statement, Entergy said that “tests conducted by S&C and Entergy on the two relays installed at the Superdome shows that one relay functioned as expected and the other relay did not.” Entergy spokesman Mike Burns, who released the statement, did not have an immediate answer when asked if both relays had the same trip setting.

During the morning committee hearing, City Council member Susan Guidry asked Entergy executives whether they were “fairly certain” that the relay device was faulty.

“That is correct,” said Dennis Dawsey, an Entergy vice president.

However, when asked if the outage was caused by the design or a defect in a part of the equipment, Entergy New Orleans CEO Charles Rice said that had not been determined.

“The equipment did not function properly,” Rice said. “At this particular time, based upon our analysis, we cannot say definitively that there was a defect in design. What we do know is that the equipment for some unknown reason at this particular time, did not react the way that it should have.”

Asked if Entergy and SMG still plan to hire a third-party investigator to get to the bottom of the cause, Rice said that possibility remains open.

“We’ll work closely with SMG and if there is a need for a third-party investigation, we will do that,” Rice said, adding that Entergy was also working with the manufacturer.

Shabab Mehraeen, an assistant professor of electrical engineering at Louisiana State University, said the relay device is a common electrical fixture in businesses and massive facilities such as the Superdome.

“They are designed to keep a problem they sense from becoming something bigger, like a fire or catastrophic event,” said Mehraeen, who holds a doctorate from the Missouri University of Science and Technology in Rolla, Mo.

The devices vary in size, and while Mehraeen noted he was not familiar with the specifics of the relay at the Superdome, he added, “I wouldn’t be surprised if it was bigger than a truck.”

Mehraeen said the reasons the devices fail are the subject of much academic research into the interaction of relays with the complex electrical systems they regulate.

“It’s not unusual for them to have problems,” he said. “They can be unpredictable despite national testing standards recommended by manufacturers.”

Entergy and SMG had both upgraded lines and equipment in the months leading up to the Super Bowl. Rice said the new switching gear, with the faulty relay, was installed as part of a $4.2 million upgrade by Entergy, including the installation of a new power line dedicated solely to the stadium.

In a separate project, SMG replaced lines coming into the stadium after managers expressed concerns the Superdome might be vulnerable to a power failure like the one that struck Candlestick Park during a 49ers Monday Night Football game in 2011. That outage was blamed at least partly on a transformer explosion.

Thornton stressed that the dome was drawing only about two-thirds of its power capacity Super Bowl night, and said typical NFL games in late August or September can draw a little more.

City officials had worried that the Super Bowl outage might harm New Orleans’ chances of getting another NFL championship game.

But NFL Commissioner Roger Goodell downplayed that possibility after the outage, saying the NFL planned to keep New Orleans in its Super Bowl plans. Mayor Mitch Landrieu said the city intends to bid for the Super Bowl in 2018.