He told some 280 mayors that most of their offices were "dysfunctional" due to corruption and mismanagement.
There has been a recent spate of violent protests, with many people angry that they are still without housing, water and electricity.
Mr Zuma condemned the "wanton destruction" during the protests.
Many people blame corruption among local ANC officials for the slow pace of service delivery.
Mr Zuma, a delegation of senior government ministers and provincial leaders met the country's mayors in a hall in Cape Town's notorious Khayelitsha Township.
He said more needed to be done to get the government's house in order.
The president said municipalities were at the forefront of providing basic services to communities.
'We owe billions'
He added that many local offices were "bankrupt" due to financial mismanagement and outstanding payments from national government departments.
Some municipalities lacked the basic administrative systems needed to collect the revenue to pay for basic services, he added."Fifty-three billion rand ($727m; £444m) is owed to municipalities by some government department for services rendered… Why are we not paying?... Maybe in the process of interaction we could help one another to state why we are not paying," he said.
Mr Zuma also condemned recent violent protests, saying they would not be tolerated.
"Burning down libraries, torching houses of people, and looting spaza shops [makeshift kiosks] do not build a strong nation. It does not solve our legitimate problems," he said.
Mr Zuma was widely touted as "the people's president" - a man who understands the plight of the poor - during his election campaign.
The BBC's Pumza Fihlani in Johannesburg says that to retain this title among the poor, who form the majority of the ruling ANC's voters, he will need to show what decisive action the government will take to improve their conditions.
The ANC has been in power for 15 years but some people feel that their lives have not improved as rapidly as promised, if at all, our reporter says.
Malaysia oh Malaysia..apa bezanya dengan Afrika..masing-masing ada masalah rasuah..yang bawah tertindas, yang atas duduk selesa..sedihnya negara muslim yang menjadi juara..sama-sama fikir..
"Aku ke Paris, aku lihat Islam tanpa Muslim, tetapi apabila ku ke Cairo (Mesir), aku melihat Muslim tanpa Islam" - Muhammad Abduh
"Aku pergi Malaysia, aku lihat banyak masjid, tapi tak de orang sembahyang, yang ada sejadah yang bersembahyang..." - |single_breath|
Rangking Malaysia (Dunia)
*kedudukan rangking berdasarkan jumlah score atau Corruption Perception Index (CPI). Semakin tinggi rangking, semakin tinggi jumlah score yang d peroleh, menunjukkan semakin rendah amalan Rasuah yang berlaku dalam sesebuah negara.