A lot of people would rule out a minority or a woman without even looking at the issues. I think most people would be okay and not let this affect them, but even a small percentage that do reject them immediately would put them in a hole.
There's also people who will vote for a woman or a minority candidate because they feel "it's about time." Lots of people like seeing history being made. I'm more interested in having the best possible President.