Being ignorant towards racism can be seen as racism, especially when those being ridiculed are minorities. What you are currently doing is trying to justify Coach Doyle for being a jerk. You don't fire the highest paid S&E coach for being a jerk. There have been white players that have literally come out and said they've seen the differences in treatment between white and minority players.
Lots of time players only have a limited point of reference. For example Matt Hankins said that Doyle ripped him up and down for spitting. Hankins said this was because he was black, he had never seen him do this to a white player. Tyler Kluver tweeted at him that years earlier he was ripped by Doyle and kicked out of practice for spitting, and then spent the next year in Doyle’s doghouse. Hankins wasn’t around when they happened to Kluver so he assumed it was because he was black, but Doyle had done the same to a white player years before he did it to Hankins.
While I have no doubt Doyle did many of the things these players said, I don’t think every one of them was racially motivated. There were enough incidents with Doyle though that were obviously racially motivated so he had to go. He made too many disparaging references and frankly Doyle probably held the Iowa program back in many ways over the last 15 years because of his racial insensitivity and bias.