After an hour in headphones, your ears need a day of complete silence. I will not specify the volume, since it is almost always set to maximum. So, otherwise, the hearing threshold increases. That is, as you put it, hearing deteriorates.
But if you set the headphone volume a little higher than the average in the city, then nothing bad will happen.
It all depends on how loudly you listen to music in them and how long, but in general, not much more harmful than any other way of listening. If you listen at a comfortable volume, no longer than 4 hours, you will most likely be fine.