Part of that might be that some people have a structural difference in the retina so that they have four colours of cone cells instead of three (I don't remember whether this is a strictly sex-linked trait, or merely more common in women than in men).
Re: colour taxonomy