SEOUL, South Korea — North Korea could fire missiles at South Korea next year, analysts predicted Monday, as the isolated North’s hostility toward the outside world deepens while it undergoes a hereditary transfer of power.

Tensions are high on the Korean peninsula following a series of provocations from the North this year. More recently, however, as South Korea responded angrily with threats of its own, Pyongyang has shown some restraint.

Expect the pendulum to swing back in the other direction in 2011, the Institute for National Security Strategy warned in a report published last week and posted to its website Monday.

The country could conduct a third nuclear bomb test and wage more attacks on front-line islands — like Yeonpyeong, which was bombarded in shelling that killed four South Koreans last month — the report said. North Korea may even fire missiles and more artillery at the those islands, chief researcher Lee In-ho told The Associated Press after the report was posted.

The Yeonpyeong attack came eight months after the deadly sinking of a South Korean warship blamed on a Pyongyang in which 46 sailors were killed.

North Korea accuses the South of triggering the assault on Yeonpyeong by carrying out military drills from the island, and denies involvement in the sinking of the Cheonan.

However, such hostility is often used to drum up solidarity in the military at times of transition, and this year North Korea made the clearest sign yet that leader Kim Jong Il is grooming his third and youngest son — Kim Jong Un — as his successor. The 20-something was promoted to a four-star general and appointed to key ruling party posts in September to mark his formal political debut.

And the provocations are expected to become only more serious next year as North Korea pushes to cement the son’s leadership and achieve its goal of building a "powerful, prosperous nation" in 2012, the 100th anniversary of North Korea founder Kim Il Sung’s birth, said the report by the institute, which is affiliated with South Korea’s main spy agency. The report was jointly written by about 20 institute researchers, but they say it does not represent their organization’s official view.

Kim Jong Il, who himself inherited leadership upon his father’s death in 1994, reportedly suffered a stroke in August 2008, intensifying speculation about what’s next for the isolated yet nuclear-armed nation he rules with an iron fist.

Next year, the report said, new provocations could also include shelling South Korean guard posts along their land border to attempts on the lives of North Korean defectors in the South.

"A state of limbo in South-North Korea relations is inevitable," the report said.

North Korea may also carry out a third atomic test to bolster its nuclear capability and ratchet up pressure on the U.S. to acquiesce to its calls for direct talks to gain aid and other concessions, researcher Lee said.

North Korea does not have diplomatic relations with the U.S., which supported the South during the 1950-53 Korean War and keeps more than 28,000 troops in South Korea to protect the ally against any aggression. The peninsula remains in a state of war because the conflict ended in a truce, not a peace treaty.

"North Korea aims to stoke tension by committing more provocations to force the U.S. to come to the negotiating table," Lee said. "If that doesn’t work, the North may carry out a third nuclear test."

There are signs North Korea is preparing an atomic test at a northeastern detonation site, the report said, but didn’t elaborate. Pyongyang also carried out underground nuclear tests in 2006 and in 2009, drawing widespread international condemnation and sanctions.

On Monday, South Korean President Lee Myung-bak promised relentless retaliation if provoked again, saying he was not afraid of war with the communist North.

"Fear of war is never helpful in preventing war," Lee said in a regular radio address. "The armed forces must respond relentlessly when they come under attack."

Initially criticized as weak in the wake of the Yeonpyeong attack, Lee has gone on the offensive in the past few weeks, repeatedly lancing angry threats of retaliation if the North strikes again, staging a series of military drills, and ordering more troops to front-line islands.

North Korea, for its part, has also kept up the rhetoric with the country’s defense chief threatening recently to launch a "sacred" nuclear war against the South.

State TV broadcast a lengthy documentary Monday chronicling leader Kim’s recent public activities, including his attendance at the ruling Workers’ Party convention in late September and a massive military parade in October.

Several shots zoomed in on his son, dressed in a blue civilian suit with his hair slicked back, raising his red Workers’ Party membership card during the party meeting.