Honolulu Mayor Kirk Caldwell said a network outage at the city’s satellite city halls and driver’s licensing centers has been fixed. Some locations will be open this Saturday for those who missed appointments because of the outage.

Our Satellite City Halls and Driver's Licensing centers have recovered from this morning's network outage. The following locations will be open this Saturday for those who missed their appointments as a result of this outage. We apologize for any inconvenience this has caused. pic.twitter.com/P48q1WzpMC

— Kirk Caldwell (@MayorKirkHNL) June 26, 2018

The city’s computer network went down about 9:15 a.m. today, shutting down most operations at the city’s 19 satellite city halls and driver licensing services, rendering the city’s web capabilities useless and disrupting the work of employees who use the network.

Customer Services Director Sheri Kajiwara said that as of 11:30 a.m., most functions at the city’s 18 satellite city halls and driver licensing centers were still down. Road tests and photos were still being taken, but nothing that requires the use of the city’s computer system could be done until further notice, she said.

“We’ve decided to send people on early lunch, so in case the center comes up, we’re ready to go,” Kajiwara said. The offices may open beyond their scheduled 4:30 p.m. closings to help process people if that can be done, she said. “Right now, I can’t say when our services will be up.”

Summers are the busiest time for driver licensing activity because students are out of school, Kajiwara said.

The city has a contract to provide licensing and registration network services with the three other counties so operations on all other islands are affected too.

The city lost all network equipment, telephone, storage, servers and mainframe systems, city officials said. While power to the main backup system was restored before 10 a.m., some key services take hours to return, city Information Technology Director Mark Wong said.

The city’s 911 emergency system and emergency dispatch network were not affected.

“Within the city, we cannot email or get to our file storage, so for all practical purposes, we are without computing services,” Wong said.

Asked why it takes so long for the network to get back up online, Wong said “These are some of the largest systems in the state. They’re very complicated and they’re not simple systems. So if you look at a network computing system that has to serve the entire state … it’s a very complicated interconnection of systems. And for security reasons, every agency is segregated from the data and the systems from the other agencies. So it’s not like it’s just one big system, it’s hundreds of different systems with different technologies and different agencies. Some of them will come back automatically and some of them will need a lot of work.”

As of this afternoon, telephone, servers and mainframe are all back up, city officials said.

“I would say that of all the systems we have to bring up, maybe 95 to 96 percent of them are already up, but because there’s one critical system that helps you get to where you need to go … because that is not working, the systems may be running, you can’t reach them for all practical purposes,” Wong said.

That one system is apparently the fiber-connected storage, which affects most computer services.

Vendors were working on upgrading a fire suppressant system in the basement of the Frank F. Fasi Municipal Building when they “pulled or loosened a wire that sent a signal to our power systems to shut down for safety reasons,” Wong said.

“I’m told that they cut up a bundle of wires and something got loose,” he said.

All the power from the “uninterruptible” power supply went down and that triggered a series of events that caused the system to shut down.

“We have secondary backup systems in place so those took over and provided battery power for a short period until those eventually ran out at which point everything went down,” Wong said. “We lost all network, telephone, storage, servers, the main frame, basically everything that was technology related to the city — except for our 911 services and dispatch — those went down.”

Because the system believed there was a fire, power was automatically cut for safety reasons, the city said. “Once the main power backup system tripped, redundant power backups took over. In the event of a power outage from the utility, the building generators would have come online before redundant backup up units were drained. Since the building still had power, systems began to fail as the batteries ran out. “

Kauai County issued a statement at 11 a.m. saying “due to a statewide computer outage, the County of Kauai’s Driver License Division is unable to process driver licenses and state identification cards until further notice. All other programs, however, are operational.”