The power struggle over Zuma's departure put the president at loggerheads with Cyril Ramaphosa, his expected successor, who is the new head of the African National Congress.
The party's powerful 107-member national executive committee (NEC) met for 13 hours at a hotel outside Pretoria, and decided in the early hours of Tuesday to "recall" Zuma from his post.
Ramaphosa and Ace Magashule, the party's secretary-general, had personally delivered a request for Zuma to resign to the president's official residence in Pretoria at about midnight.
But Zuma "was very arrogant. He told them he was not going anywhere as he did nothing wrong," an unnamed ANC committee member told the Mail and Guardian newspaper.
"He told them if the ANC issued a statement on its decision to recall him, he will retaliate."
Another source told the Times Live website that Ramaphosa had pleaded with Zuma to leave office, but Zuma had responded "in no uncertain terms" that he would not resign.
The ANC - which was once led by anti-apartheid struggle leader Nelson Mandela - reportedly dismissed a request by Zuma for a few more months in office and instead gave him 48 hours to step down.
ANC officials were not reachable Tuesday morning, but the party called a press conference for 2:00pm at its headquarters in Johannesburg.
The ANC can "recall" the head of state, essentially forcing him to resign, but the process is a party-level instruction and he is under no constitutional obligation to obey.
If he refuses, Zuma would then likely be ousted via a parliamentary vote of no-confidence within days.
Ramaphosa, the de-facto president-in-waiting, has been in deadlocked negotiations with Zuma, who dismissed an earlier request from party leaders to resign more than a week ago.
The stalemate plunged South Africa - Africa's most developed economy - into uncertainty over who is running the country, with a series of public events canceled last week including the annual State of the Nation address to parliament.
"We know you want this matter to be finalized," Ramaphosa, 65, told a party rally in Cape Town on Sunday to loud cheers.
"We know you want closure... Because our people want this matter to be finalized, the NEC will be doing precisely that."
South African opposition parties have called for early elections as the ANC's leadership battle grinds on.