Regarding your second point here, about the eye adapting to the different amount of light displayed within the EVF, yes, this is something to consider, but such cases, which don't come up often for me, underline how great it is to have the tiny, translucent histogram displayed within in the EVF.Do you always use the same shadows and highlights compensation values in your final images? If not, EVF cannot be WYSIWYG to you.
And if the brightness level of the scene is much different from the brightness level of the EVF, your eye may be misadapted to the EVF image brightness when it's most needed.
Both of this is even worse with the smartphones, which don't have EVFs to shield the display from the ambient light.
Here's how I'm using the EVF for precise exposure. My Picture Style is Standard. I use spot metering. I can either move the spot slightly up and down or side to side on, say, a face, subtly changing the exposure. When I get what looks just right with highlights and shadows, I hit AE Lock with Hold (see user manual pp 555-556). Then I can shoot the composition freely without exposure changing on the subject. I shoot only RAW. When I use the term WhatYouSeeIsWhatYouGet (WYSIWIG), I'm referring to both the image on the back of the camera and what I'm seeing with default Lightroom settings on my desktop monitor.
It did take some experimenting to get the EVF and back-display brightness levels "calibrated," so to speak.