This question is actually quite technically deep. Component video and VGA are in different colour space. While it is quite well known that VGA uses RGB colour, video signal standards usually separate an image into Luminance and Chrominance components. Component video is also known as YCbCr, where Y is the Luminance or brightness information, Cb is the blue difference and Cr is the red difference. Green is inferred from the Y channel.

Now, in computer graphics, the textures and frame buffer are also stored in RGB format. When the final picture is sent to the TV it has to be encoded into YCbCr or Y-C (s-video) or YUV (composite). At the TV, the final display, no matter if it's a CRT or LCD or Plasma, is also in RGB format, so another conversion step is involved. Common sense dictates that this sort of repeated conversion results in some data loss.

In addition to this, the chrominance information is often sampled less than the luminance. This is known as 4:2:2 sampling, where for every 4 horizontal "pixels" of Luminance, there are only 2 Cr pixels and 2 Cb pixels. The SMPTE (Society of Motion Picture and Television Engineers) developed this short cut because the human eye is less sensitive to colour differences than brightness aka luminance. When colour television was developed over 30 years ago the available transmission and recording equipment were not capable of supporting the enormous bandwidth required for full resolution colour, so that's why this system was developed.

VGA was developed more recently, and by a different industry. Therefore, it doesn't have these limitations. Furthermore it helps that VGA is much closer to the "native" image format used in the console frame buffer. The only thing better than this would be an all digital transmission format such as DVI or HDMI.

At the end of the day though, this is mostly irrelevant. At normal TV viewing distances of ~10 feet, you will hardly notice such fine detail. Even at close range "PC" distances at ~3 feet, for example with an LCD monitor, the difference is subtle and you have to be really looking for it to notice the difference.