Many things going on.
Depending on scene, noise is more or less apparent. Large areas of OOF 'negative space' show noise more prominently (subjectively, because there's nothing else there to pay attention to, you eye starts looking for patterns).
Shutter speed does affect noise, but meaningfully only with longer exposures (you'll notice more noise at the same ISO in a 0.5 s exposure vs a 1/200 s exposure, but not between 1/1000 and 1/640.
ISO matters - 'tweener' ISOs have more or less ISO noise than the full stop settings, depending on camera model. So, ISO 500 is noisier than ISO 800, while ISO 640 is slightly less noisy. That's beacuse with a full stop base ISO, the up tweeners are pushed (1/3 stop up, e.g. 125, 250, 500, 1000, 2000, etc.) while the down tweeners are pulled (1/3 stop down, e.g. 160, 320, 640, 1250, 2500).