A federal appeals court has upheld the Honolulu Police Department’s refusal to grant a firearm permit to a man who was convicted of petty misdemeanor harassment.

One of the three judges on the 9th U.S. Circuit Court of Appeals panel that ruled Friday in the case against Kirk Fisher, however, criticized Hawaii law for imposing a lifetime ban on a person’s constitutional right to bear arms for committing a misdemeanor.

Fisher’s lawyer, Donald Wilkerson, said he is encouraged by the criticism and that he and Fisher will continue to pursue the case. “We’re not going to give up on this,” he said.

Following his arrest in November 1997 for harassing his wife and daughter, Fisher surrendered whatever firearms he owned to HPD, as ordered by a state Family Court judge. He pleaded guilty to two counts of harassment the following month and was sentenced to six months of probation.

In November 1998, after Fisher had completed his sentence, HPD returned the firearms to Fisher, as ordered by the court. When Fisher applied for a permit for a new firearm in 2009, however, HPD refused to issue him one and told him that he needs to get rid of his other firearms.

HPD told Fisher that state law disqualifies him from owning or possessing a firearm because of his conviction for a crime of violence. The crime to which Fisher pleaded guilty defines harassment as striking, shoving, kicking, touching in an offensive manner or subjecting another person to offensive physical contact. Hawaii law also incorporates federal law, which disqualifies ownership or possession by anyone convicted of misdemeanor domestic violence.

Fisher filed a federal civil rights lawsuit in 2011 against HPD and the city. U.S. District Senior Judge Alan C. Kay found in favor of the city in 2014. He also said the state law that disqualifies Fisher from owning or possessing a firearm does not violate the U.S. Constitution’s Second Amendment right to bear arms.

Fisher appealed.

All three appeals court judges, including Circuit Judge Alex Kozinski, rejected Fisher’s appeal. But in a concurring opinion, Kozinski wrote that while federal law affords people four different ways of getting back their right to keep and bear arms, Hawaii law affords them only one: a pardon.

“The time has come to treat the Second Amendment as a real constitutional right. It’s here to stay,” Kozinski wrote.

Kozinski distinguished himself most recently when he filed a separate dissent to a 9th Circuit panel’s ruling that upheld a block of President Donald Trump’s first travel ban.