Jaco Peyper will not referee a World Cup semi-final after mocking sent-off France lock Sebastien Vahaamahina in a picture with Wales fans.
World Rugby says the South African recognises posing for the picture "was inappropriate and he has apologised".
In the picture Peyper's elbow is raised into a fan's chin hours after he sent off Vahaamahina for elbowing Aaron Wainwright in Wales' quarter-final win.
That prompted World Rugby to investigate the post-match incident.
Vahaamahina announced his retirement a day after his dismissal, saying that decision was pre-planned.
Welshman Nigel Owens will referee England v New Zealand on Saturday with France's Jerome Garces in charge of Wales v South Africa on Sunday.
Frenchmen Romain Poite and Pascal Gauzere will assist Owens, with South Africa's Marius Jonker as television match official (TMO).
Wayne Barnes of England and New Zealand's Ben O'Keeffe will be Garces' assistants with Ben Skeen of New Zealand the TMO.
French Rugby Federation vice-president Simon, a former France prop, said on Twitter the Peyper picture was "shocking".
Peyper was officiating in his 50th Test match as Wales progressed to their semi-final against South Africa in Yokohama on Sunday, 3 November.
Wales coach Warren Gatland said he was aware of the picture of Peyper with Wales fans.
"He's had his 50th cap, he's posing with a few fans and he's making a bit of a joke about the incident in the game," Gatland said.
"How people interpret that is up to them. Obviously, the way things are, people like to make mountains out of molehills."