Despite the surrenders, hundreds are said to be still inside
Pakistani troops are tightening their grip around the mosque, which is at the forefront of a vigilante campaign to enforce a form of Islamic Sharia law.
The troops set off blasts outside the mosque, as negotiations continue with hundreds of students still inside.
At least 16 people have died in recent clashes at the complex.
The students have kidnapped police officers and people they accuse of involvement in "immoral" acts such as prostitution.
The BBC's Barbara Plett in Islamabad says the radicals do not have much support in Islamabad and people are quite glad to see the government taking them on.
But the authorities' action is likely to upset people in the more conservative North-West Frontier province, where most of the students come from.
'They want martyrdom'
The government said more than 1,200 students had surrendered so far, and that their details were being recorded at a camp set up near the mosque.
Following the expiry of an 0900 local time (0500 GMT) deadline on Thursday, at least four more female students and one male student were seen giving themselves up.
Maryam Qayyeum, 15, one of those who surrendered, told the Associated Press news agency that those who were staying back in the mosque "only wanted martyrdom".
"They are happy. They don't want to go home," she said.
Another student, Johar Ali, 20, said he had not seen any suicide bombers among the students.
Some children are thought to be in the mosque, but the mosque's deputy head reportedly refused to allow their release despite requests from parents.
Deputy head Abdul Rashid Ghazi said the radicals would not surrender unless troops pulled back.
Our correspondent says the threat of force is likely to remain but that talks are expected to continue.
Pakistani security forces set off a series of eight explosions outside the mosque before dawn. No casualties have been reported but there was some damage to the mosque.
"All people in the mosque should surrender or they will be responsible for losses," said the police warning broadcast over loudspeakers, according to witnesses quoted by Reuters news agency.
The mosque's head cleric was captured by police as he tried to leave the building disguised as a woman.
Maulana Abdul Aziz was caught wearing the burqa, an all-enveloping sheet-like dress adopted by some Muslim women.
A security official told the AFP news agency the cleric was noticed because his height and his large stomach set him apart from the women he was with.
Pakistani President, Gen Pervez Musharraf, has long been criticised for failing to clamp down on the mosque's activities.
The latest confrontation reportedly started when security forces tried to place barriers around the mosque.
On Tuesday, women students of one of two seminaries attached to the mosque protested in the street, while their male counterparts traded gunfire with security forces.At least 16 people died in the violence and scores were hurt.