It was rude.True ... but their misunderstanding does not diminish your value and success.
And for the "keep the client"- argument:
I refuse having clients like this.
Being nice and handsome most of the people misunderstand as weakness.
Regarding refusing clients: if photography is our full time job (or in any business for that matter) one cannot make emotional decisions based on one incident, one has to adapt to situations and find ways of retaining customers/clients, coz good times and bad times don't always stay forever ... occasionally one does need even the 'not-so nice' clients to tide through tough times ... unless of course we are so stinking rich that we don't need their business.