The gunbattle began after police and soldiers cordoned off the southern town of Chadoora following a tip that at least one militant was hiding in a house, said Inspector-General Syed Javaid Mujtaba Gillani.
As the fighting raged, hundreds of residents chanting anti-India slogans marched near the area in an attempt to help the trapped rebel escape.
Witnesses said intense clashes between rock-throwing protesters and government forces erupted just a few hundred meters (yards) from the besieged house, with police and paramilitary soldiers firing shotgun pellets and tear gas to stop the protest.
However, clashes spread to multiple places in the area.
Later, government forces fired into the crowd, killing three civilians, including a teenager, and injuring at least 20 others, police said, adding that eight police and soldiers were also injured in the clashes.
An army officer said one militant was killed in the fighting and one weapon recovered from the site. The officer spoke on condition of anonymity in keeping with military regulations.
Local residents said troops blasted the house with explosives.
In Srinagar, the main city in Indian-controlled Kashmir, police stopped an ambulance and took custody of the body of a 23-year-old student killed in the protests. The man had been brought to a Srinagar hospital where doctors declared him dead.
Police fired tear gas and swung batons after the dead man's relatives pleaded that they be allowed to take the body to Chadoora area.
A police officer, speaking on a customary condition of anonymity, said the action was taken to avoid more protests in Chadoora and the body would be handed over to the relatives for burial later Tuesday.
In recent years, Kashmiris, mainly youths, have protected anti-India rebels by engaging troops in street clashes during military operations against the militants.
Rivals India and Pakistan each administer part of Kashmir, but both claim the Himalayan territory in its entirety. Most people in the Indian-controlled portion favor independence or a merger with Pakistan.
Separatist leaders who challenge India's sovereignty over Kashmir called for a shutdown and protests on Wednesday against the killing of the three civilians.
Three key leaders, Syed Ali Geelani, Mirwaiz Umar Farooq and Yasin Malik, said in a statement that the killings were "brazen state terrorism."
Last month, the Indian army chief warned that "tough action" would be taken against stone throwers during counterinsurgency operations. But the anti-India protests and clashes have continued.
Rebels have been fighting against Indian rule in Kashmir since 1989. More than 68,000 people have been killed in the armed uprising and ensuing Indian military crackdown.