For spectrum inversion thought experiments to be philosophically interesting, the variations in color experience that they wonder about have to be behaviorally indetectable. Here's a quick summary of some of the asymmetries of color space that would make variation detectable:
1. The most saturated yellows are lighter than the most saturated reds and blues.
2. There are more distinguishable hues between blue and red than there are between yellow and green.
3. Dark yellow is a qualitatively different color than yellow (it appears brown), whereas dark blue is still blue.
4. Desaturated bluish-red is a qualitatively different color than saturated bluish-red (it appears pink), whereas desaturated greenish-yellow is qualitatively similar to saturated greenish-yellow.
(All of these asymmetries are taken from Alex Byrne's excellent Stanford Encyclopedia article on the topic. I'm merely summarizing them for your convenience.)