"The events before and after the match with Botev were not acceptable for me, for my assistant Alex Dyer, or for our players."
Pardew confirmed that former Kilmarnock manager Dyer, who is black, is also leaving CSKA.
Pardew had agreed to take charge for the remainder of their current campaign, which ended on 22 May, and for the 2022-23 season.
But the ex-Newcastle and West Ham boss, who has managed seven English teams, chose to leave after numerous conversations with club officials over the past 10 days.
"The reason no-one gave an interview after the match was that we were all very outraged by the situation that had escalated," the statement added.
"Our players decided to play only out of loyalty and to protect the club. The small group of organised racist fans who tried to sabotage this match is not what I want to lead and represent the team.
"This is not the right path for the benefit of CSKA, because such a club deserves much more."