The city’s major computer network was offline much of Monday, shutting down most transactions at the 19 already hectic satellite city halls and driver’s licensing centers on one of the busiest days of the week.
Hundreds of Oahu residents seeking to renew their driver’s licenses, obtain state identification cards or conduct other business were told to wait indefinitely or to reschedule appointments.
Summers are historically the busiest time for driver’s licensing activity because students are out of school, city Customer Services Director Sheri Kajiwara said.
To make up for the lost hours, the driver’s licensing offices in Kapolei, Koolau, Kapalama and Wahiawa will be open from 8 a.m. to noon Saturday, as will the satellite city halls in Kapolei, Waianae, Ala Moana, Kapalama, Wahiawa and Hawaii Kai.
Additionally, those who obtained tickets for service Monday will be allowed to walk into a line reserved only for them through today “and possibly through Wednesday,” Kajiwara said.
The computer outage also affected registration and licensing services on all other islands because the city has a contract to provide network services for those functions statewide.
Meanwhile, people trying to access the city’s honolulu.gov site for online services or public information hit a snag, and roughly 5,000 city employees went without access to the network or even email.
The city lost all network equipment, websites, telephone, storage, servers and mainframe systems during the incident, city officials said.
“So for all practical purposes, the city was without computing — except for emergency services, which are on a separate network,” city Information Technology Director Mark Wong. Both the emergency 911 and dispatch services are on different systems and were unaffected, he said.
While power to the main backup system was restored before 10 a.m., others had to come back online gradually through the day. By about 2 p.m. most functions at satellite city halls and driver’s licensing centers were restored, and by 4 p.m. most of the network had returned, city officials said.
The network went down at about 9:15 a.m. when vendors working on upgrading a fire suppressant system in the data center in the basement of the Frank F. Fasi Municipal Building “pulled or loosened a wire that sent a signal to our power systems to shut down for safety reasons,” Wong said. “I’m told that they cut up a bundle of wires and something got loose.”
Because the network believed there was a fire, power was automatically cut for safety reasons.
“Once the main power backup system tripped, redundant power backups took over,” Wong said. “In the event of a power outage from the utility, the building generators would have come online before redundant backup up units were drained. Since the building still had power, systems began to fail as the batteries ran out.”
The contractor tried to power the system back up, “but it was about a minute too late to keep up some of our critical systems,” Wong said.
Asked why it takes so long for the network to get back up online, Wong said “it’s a very complicated interconnection of systems. … It’s hundreds of different systems with different technologies and different agencies.”
Road tests and photos could be done, but not much else before connection to the network was restored.
Monday’s incident occurred a year ago to the week after sudden power outages hit the Fasi building on consecutive days, causing the city to lose its main “uninterruptible” power supply as well as other equipment.
“As a result of that, we did an emergency procurement to bring in a lot of additional UPS equipment, which is what allowed us to recover from this (incident Monday) faster,” Wong said. “We had two layers of UPS.”
The city is also procuring a contractor to completely rebuild the data center in the Fasi basement with more modernized equipment, including the power system, he said.
Looking forward, the city might look at using a number of smaller system data storage units that could recover more quickly, Wong said.
