Personally, I find it worse when my ears sweat. The frames sitting against them make them feel raw and sore.
Though, it could be from the scars left by my first pair. The metal frames eroded the skin on the top of and behind my ears. Can't really see the scars, but I can certainly feel them.