A technical glitch appears to be the key reason why nearly 8 in 10 of Oahu’s roughly 166,000 water customers received bills that were overestimated or underestimated during the first nine months of 2013.

But while the problem was resolved in a relatively short time, exactly what caused the hang-up is still unknown, said Henderson Nuuhiwa, Honolulu Board of Water Supply chief information officer.

The estimated bills issue caused a public uproar as water customers deluged the board staff with complaints about why they were suddenly hit with "back-billed" charges, some of which totaled several thousand dollars.

At a presentation to board members on Monday, Nuuhiwa said a thorough root cause analysis done by his staff identified four primary causes for the spike in estimated bills. All apparently could be traced back to a glitch that occurred when the agency tried to upload information into newly installed customer service software, which spits out monthly water bills, from the data collected through its older meter-reading software.

Bills based on estimated usage are generally issued when the billing system fails to see actual meter readings.

The agency’s information technology staff determined in April that batches of actual meter readings were not recognized by the new water-billing software, forcing it to come up with thousands of estimated bills based on previous usage and the usage of neighbors, Nuuhiwa said.

The staff traced the problem to an unknown number of compound meters, typically used by commercial customers with multiple users, Nuuhiwa said.

Meter readers were able to obtain actual readings from those meters, but the customer service software installed in January failed to pick up those readings because notes that accompanied the readings were inputted out of sequence, which flagged and deleted the readings, he said.

"Unfortunately, that’s what caused the bulk of the problems," Nuuhiwa told board members. "It also was something that was not easily anticipated, or projected, because you don’t know when this out-of-sequence scenario is going to occur."

The problem snowballed because readings were being gathered and submitted as a batch once a day into the water-billing software. So once a non-reading was found, flagged and deleted, it deleted all the previous readings recorded that day, Nuuhiwa said. That meant potentially thousands of recorded readings were not being inputted into the billing system, instead triggering estimated bills, he said.

There are only 946 compound meter accounts, but it would have taken only one to be flagged to potentially delete an entire cycle of readings, Nuuhiwa said.

The island’s 166,000 customer accounts are split into 20 geographic cycles. One day each month, meter readers collect data from the customers in a cycle.

The meter-reading software is made by Sensus, which describes itself on its company website as "the undisputed leading meter provider in the water industry." The new customer care and billing software is manufactured by Oracle. Previously, BWS used an in-house billing program.

The situation was exacerbated because of a separate policy change that the board implemented at the beginning of the year.

When readings cannot be taken directly from meters electronically, a second group of meter readers is supposed to return to customers’ meters and obtain a reading manually. But just when the board saw a surge in non-readings beginning in February, the agency switched from bimonthly billing to monthly billing. Fewer inspectors were left to deal with the surge in half the time before the next billing cycle.

Nuuhiwa said the problem was detected on April 15, and fixes were implemented four days later. Among the fixes was requiring each customer’s meter readings to be submitted directly into the billing software rather than once a day as a batch, he said.

"Every record as it’s read is presented to the billing system for processing," Nuu hiwa told the Star-Advertiser.

The agency also increased its meter-reading staff, Nuuhiwa said.

Although agency officials were able to identify the root cause and make fixes, they still do not know exactly what caused data to be read out of sequence or even if the error was human or software-generated, he said. That’s because they couldn’t see the errors.

As a result, "we really don’t know why it would come out of sequence in some situations and in others it doesn’t," Nuuhiwa said. "We don’t control that process, so we don’t see what’s actually going on when the information from hand-held meters are docked and the records are brought in."

Other reasons leading to an increase in estimated bills were also identified including no-reads caused by broken meters or meters without batteries. A battery replacement project began in February while meter repair, replacement and relocation to areas where they can be read more easily are ongoing.