Each photosite is sensitive to all of the "colors", which aren't really colors at all until they're perceived by our brains, it's just that photosites filtered with one of the three filters are more sensitive to some wavelengths than others. But they all share overlapping sensitivities. The photosites under all three colors of the filters in a Bayer mask are sensitive to all three "colors". There's no sensor at any one location for any single color.
It works just like our retinal cones. Our brains construct the perception of color, which really does not exist in light itself, from the differences between the S, M, & L cones. The peak sensitivity of our M & L cones are very close to the same wavelength, and most of each type's sensitivity overlaps the others. Our brain uses that minor difference to create color.