The screen saver goes off, or does the computer/monitor shut off? Sounds like your power settings barely outreach your screensaver's time limit. Check the "Power Settings" in the Control Panel.
But if it goes on, then back to the desktop without touching anything, I'm not sure what's going on.
you might have an active process that is bringing the screensaver out of operation. alt-ctrl-del it, and look for a process that is sapping mem. If that isn't it, hmmm... I'll look into it.