Kinda depends on how one defines popularity. If one thinks popularity is all about sales, then yes, they could say that after Duty, Ayu's popularity started decreasing.
But if one defines popularity as presence, influence, longevity, etc., then it's a different story.
__________________
"Remember, don't let others dictate your music taste. If you like whatever you're listening to, keep listening to it."