I've seen this happen on my Sega Dreamcast before. In my case, I was using a 3rd party S-video cable. The cable worked fine for the first year or so, then it started showing symptoms that you described. In fact, the banding issue that you showed on the sreenshot looked exactly like my problem. Replacing the cable with the stock composite video cable corrected it. Have you tested with a different cable before?