BHP's aim was to secure Anglo's prized copper assets in Latin America and increase access to a metal central to the global shift towards clean energy and electric vehicles. But the structure of BHP's deal, which required Anglo to unbundle its South African platinum and iron ore businesses, was a major reason for its collapse.
Anglo had granted BHP a one-week extension until 1600 GMT on Wednesday to its original May 22 deadline to submit a binding offer, after rejecting a third takeover proposal. Under UK rules, BHP cannot return for at least six months, unless there is another offer for the London-listed miner.
"While we believed that our proposal for Anglo American was a compelling opportunity to effectively grow the pie of value for both sets of shareholders, we were unable to reach agreement with Anglo American on our specific views in respect of South African regulatory risk and cost," BHP said.
"We remain of the view that our proposal was the most effective structure to deliver value for Anglo American shareholders, and we are confident that, working together with Anglo American, we could have obtained all required regulatory approvals, including in South Africa," it added.
Anglo said on Wednesday after BHP's statement it was fully focused on delivering plans it has set out to increase value to shareholders.
Anglo has outlined a plan to divest less profitable assets and focus on expanding copper output.
Anglo's shares closed 3% lower at 24.80 pounds.
"While we believe an acquisition of Anglo would be a longer term positive for BHP, the fact that BHP is being disciplined in its approach is a near-term positive for its shares," Jefferies analyst Christopher LaFemina said.
"That said, we are surprised BHP did not have a plan to overcome the clear structural issues relating to this deal."
London-listed Anglo had agreed to talk with BHP to try to iron out concerns over the deal's structure.
In an earlier statement, BHP said it needed more time to engage with Anglo, while outlining commitments to minimise regulatory risk in South Africa and saying it would offer a break fee if the deal failed to gain regulatory approvals.
Those commitments included job security for employees in South Africa. BHP also said it would shoulder the costs of increased South African employee ownership that is expected to be required in any demerger.
But Anglo said those commitments were not enough.
"BHP continues to restate its belief that the risks of its complex structure are not material, yet has repeatedly and consistently stated both publicly and during the engagements that it is unwilling to amend its proposed structure to assume these risks," Anglo said in its statement.
Anglo was founded in Johannesburg in 1917 and employs more than 40,000 South Africans, so any withdrawal would be a further economic blow to the country whose miners have been cutting jobs and investment as platinum especially falls out of favour.
South Africans are voting in an election on Wednesday, with polls suggesting the African National Congress could lose its majority after 30 years in power, in part due to anger about high unemployment and a stagnant economy.
JP Morgan analysts have estimated a takeover of Anglo by BHP could lead to outflows of $4.3 billion from South Africa and weaken the rand.
A source close to Anglo's thinking said its investors shared its reservations about BHP's proposal.
"The majority of the Anglo shareholders fully understand the concerns that are being expressed and I don't believe that they feel that the risks in the structure and the price are fully taken into consideration by BHP," the source said.
BHP's latest proposal had valued Anglo at 29.34 pounds per share or 38.6 billion pounds ($49 billion).
"I'm not surprised it was rejected really by Anglo ... because there wasn't really a lot in the statement from BHP ... it didn't seem that compelling," said George Cheveley, portfolio manager at Ninety-One, which holds a stake of about 2% in Anglo.
Anglo is attractive to its competitors for its prized copper assets in Chile and Peru, a metal used in everything from electric vehicles and power grids to construction, whose demand is expected to rise as the world moves to cleaner energy and wider use of artificial intelligence.