I don't want to get stills from video but I also don't know why I should accept poorer frames for video than I do for stills.
even with a 50inch 4k display you will hardly see a difference from normal viewing distances.
the human eye is not able to resolve that good.
Erm, this is so completely wrong. It entirely depends on your visual acuity. The whole notion of the "average" person having 20/20 vision implies that a certain percentage of people also have better vision (and some much better), and that a similar percentage of people have worse vision (and some much worse). For the people who have worse vision, corrective optics these days EASILY correct vision beyond the 20/20 mark. Personally, I am slightly near sighted, and with my contacts or glasses, I have 20/10 vision, like so many other people with corrective lenses. (At my eye doctors office, they generally purposefully try to find the absolute best correction possible, aiming for the highest visual acuity possible. They see thousands of people a year, so you have to figure that between people with excellent vision, people who are slightly far sighted, and everyone walking around with corrective lenses...the "average" visual acuity is actually higher than 20/20.)
I currently have a trusty old Samsung 46" 1080p TV, and I can just barely see pixels when sitting from the TV at a "comfortable" distance. The distance is ideal for my room setup, with the TV at the recommended distance from my couch. There is no question that bumping the resolution up to 4k would do wonders for quality. It is just simply not enough to have pixels just on the border of 20/10 visual acuity (which is what's recommended)...you need to have the pixels be much smaller in order for them to NEVER intrude on your experience.
Same thing goes for using higher resolution computer screens. Even sitting an appropriate, comfortable distance from my 30" 2560x1600 screen, which until 4k displays started arriving had one of the smallest desktop pixel pitches, exhibits this slight pixellation effect. I can't exactly see individual pixels, but they are again just on the border of my visual acuity...so they bug me. A 4k 32" display would almost reduce the pixel pitch in half, and do wonders for microcontrast and allow me to see
fine detail in my photos as fine detail, rather than fine pixels that contain detail I should be able to see. Furthermore, if you print, you'll know that it is extremely difficult to soft proof a print on a screen that has at least 1/3rd the pixel density of the print. You can never really tell how the detail will turn out in a print. Personally, I'd be ecstatic with a 28" screen that had a 300ppi pixel density. I'd be able to properly soft-proof the majority of my larger prints at a directly comparable resolution.
There is a LOT going for 4k screens, both TVs as well as workstation screens. The human eye absolutely can resolve that well unless you have particularly poor vision or just have average vision and don't use corrective lenses. Same as with sensor pixel densities, however...the actual output resolution of any system is effectively approximated by the RMS of the individual components. Increase the resolution of a screen, and the ultimate resolution of what your mind's eye sees will still improve, even if you have only 20/20 vision.