Last year, Louisville took in nearly $88,000,000 in athletic revenue. I think that would put them about third in the Big 12 (according to the EADA numbers). And Louisville got that revenue despite being in a conference with a horrible (by today's standards) TV deal. For reference, Kentucky had revenue of nearly $86,000,000, and that being a member of the SEC.
For a school of 12,000 undergrads, either Louisville has one enormously rich alumni base, or they have a much larger national fan base and following than you give them credit for.