In my opinion, the popup box that presents itself when trying to play a song/album/playlist when something is already playing is intrusive and should be removed.
Instead, anything you select should start playing immediately without any dialogue, automatically clearing the queue. To add things to the queue, there could be a simple dedicated button next to Play, Shuffle, Favorite, etc. We already see something like this if you right click on an individual song in the form of "Play Next".
Adding things to a queue should be treated as a secondary action that the users have to seek out in order to perform (i.e. clicking the separate button). It shouldn't be the basis of whether clicked content is allowed to play immediately or not.