One of the great things about being in college in the early 2000s was how much the internet helped you hear weird bands, and websites like Pitchfork would help folks outside of their geographical area hear those bands.
Some of my favorites from that era:
Starlight Mints, "Valerie Flames"...