I can see two probable causes:
A. As Law said, you probably have some spy-ware on your machine, most likely the real nasty kind that latches onto IE (notice any new toolbars lately?)
B. A memory leak in IE is causing the problem. I'm not sure but I think I remember hearing a while back about a patch that was supposed to fix some sort of memory leak in IE. Regardless of if that's the problem or not, its always good to update IE anyway.