I used to be like that, only listening to anime related music.
But after branching out into other albums from artists who sang various anime themes, then listening to other artist who were "similar" to said previous artists, I eventually stopped listening to anime music all together.
I'm proud of my J-music heritage though, without anime I never would have heard Ayu, Utada, Koda Kumi, Gackt, BoA, X-Japan or heaps of other artists. Though these days I've been selling off my anime DVDs to pay for my Ayu addiction...