I didn't claim we were consistent. English spelling (or American spelling) is a train wreck. Imagine a language where a spelling bee would be pointless. Whatever that language is, it's not descended from anything spoken by the Angles and the Saxons.
"Glamour" is ironically appropriate since it describes hoity toity-ness anyway. I place it on the level of "ye olde shoppe." But, as I said American spelling is a train wreck too.
And at least four/for makes a distinction between two different words. What word would "rumour" be confused with if it were spelled "rumor"?
I guess we have to keep the u in hour too for the same reason.
(Kidding aside, that "ou" actually makes a difference in sound.)