I'm a girl, so I can't really answer the question, lol
But~ I can add that many people's ideas of "girly" and "manly" are pretty...strange. I mean, where I live, if a guy were to be seen listening to anything other than rock or rap, he's labeled as gay. It's so sad, really, because I try to see everything as the same. I love the color blue, even though it's a "boy color" and I don't care if guys wear pink, even though it's a "girl color." I think those kind of "categories" are ridiculous and stupid 

 So, that being said, what's the difference with pop? I mean, sure, some of Ayu's songs are aimed towards women, but a guy can still listen to them; what's the difference? Many people don't even look up the translations 

 And even if they did, again: what's the difference? And, like AyumiAi said, many of Ayu's things are aimed towards men, too 

 Most girls who listen to Ayu watch things like the my name's WOMEN PV because they are envious of her, but most guys watch because...you know 

 I know I don't watch some of Ayu's PVs because I feel strange doing so, but that shouldn't be. I want to sit down and watch Ayu's concerts in public, but because of the categories and stereotypes people have created, I would be looked at strangely for watching tours like AT08, which has Ayu in revealing clothes. I hate these kind of stereotypes...
One of my best friends is gay, and he loves rock. So I don't understand all these stereotypes and categories that people have made. 
I commend any guy who listens to Ayu freely~ Good for you~ <3