You are loading the auditory canals unevenly. If this becomes a habit, the ear, which will be more loaded, will lose sensitivity to a certain frequency spectrum relative to another. In other words, you will simply hear worse with this ear. It may also be difficult to localize sounds in space, but this will require a very long time listening to music in this way, from 10 years.
Sometimes it's even useful, especially when crossing the road on a busy street - it's better to listen alone :)
_ _ _ _ _ _ _ _ _ _ _ ___