Hello,
...Your all system specs are more than adequate.
I see you have a Windows XP MCE PC. You've got video capture / video input on your system.
What program are you using to view the PS2 video?
It's not your PS2 (especially if it works fine when hooked up to a TV). It's most certainly something to do with your PC.
Your graphics card is fine. It's probably the program you're using to view the PS3 video input or your video capture device.
Are there any video input ports on the back of your PC? Not confusing the input with your graphics card video out, check for a yellow composite / RCA input, and plug your PS3 to it if you find one.