He faced fierce social media criticism after climbing on the creature during last week’s episode of ITV programme My Travels And Other Animals.
In a statement, Born Free confirmed Clunes’ “deeply unfortunate” departure for riding a “captive, wild” elephant.
It said Clunes’ actions reinforced a practice it is “resolutely against”.
“Born Free has always been opposed to the exploitation of captive wild animals for entertainment and human interactions.
“There is clear evidence that training, keeping and riding captive elephants causes distress and suffering,” the charity added.
Mr Clunes’ representatives have been contacted by the BBC for comment but are yet to respond.
During the episode the actor expressed open concern about riding the mammal, but said using the animals for tourist purposes was “a kinder life than hauling heavy logs”.
After clumsily climbing atop the elephant, he is shown apologising, saying: “I didn’t want to hurt her.”
The scenes prompted strong reaction, including from Born Free’s president Will Travers, who told The Daily Mirror it was “deeply unfortunate”.
The Men Behaving Badly star was trending on social media throughout Tuesday morning as a result.