At least three explosions were heard at the start of the attack on the building housing the state accounts office, the first from a car bomb detonated at the entrance, said Attahullah Khogyani, a spokesman for the provincial governor.
At least nine people, including a child, had been killed and 36 wounded, public health officials said.
The attack, in a busy area of the city with many other official buildings nearby, is the latest in a series that have killed and wounded hundreds in Afghanistan this year and put heavy pressure on the Western-backed government of President Ashraf Ghani.
Most have been in the capital Kabul, but in January gunmen attacked an office of aid group Save the Children in Jalalabad, killing at least five people and wounding 25 in an attack claimed by Islamic State that followed much the same pattern as Sunday's incident.
As the battle continued, smoke could be seen rising from the accounts office building. Witnesses said the explosions had caused carnage among passers-by.
"I saw two rickshaw drivers on the ground with their arms blown off," said Khan Mohammad, a local resident who saw the initial blasts and the start of the gunbattle.
Gunmen rushed the building following the first blast at the entrance, a member of the provincial council told Reuters. More explosions were then heard coming from inside the building, he added.
He said at least four attackers, armed with rocket propelled grenades and machine guns, still appeared to be fighting police.