Their actions began with gunfire inside a key military base, some 15km (nine miles) from the capital, Bamako, on Tuesday morning.
In the centre of Bamako, young men looted a government-owned building and set it on fire.
The mutiny has been condemned by regional group Ecowas and former colonial power France.
The unrest coincides with calls for more protests to demand that the president resign.
Mr Keita won a second term in elections in 2018, but there is widespread anger over corruption, the mismanagement of the economy and the worsening security situation with jihadist and communal violence on the increase.
It is not yet clear how many soldiers have taken part in the mutiny. Some reports say it was fuelled by a pay dispute.
The Economic Community of West African States (Ecowas) urged the mutineers to return to barracks.
“This mutiny comes at a time when, for several months now, Ecowas has been taking initiatives and conducting mediation efforts with all the Malian parties,” its statement said.
France’s Foreign Minister Jean Yves Le Drian said France condemned “in the strongest terms this serious event” and he too urged soldiers to return to barracks.
France has troops in the West African state to fight militant Islamists.
Footage from the AFP news agency shows a building owned by the justice ministry in Bamako ablaze.
Earlier on Tuesday, the president’s son, Karim Keïta, dismissed as false reports that he was arrested by troops.
In recent months huge crowds let by populist imam Mahmoud Dicko have been calling on President Keïta to step down.
Much smaller crowds reportedly gathered in the capital on Tuesday in support of the soldiers.