In Windows XP, Vista and 7, go to the Start menu.
Go to All Programs.
Scroll down to the 'Startup' folder.
Right-click any programs in there that you don't want starting up, and delete them.
There's a more advanced (and thus more dangerous if you make a mistake) way of doing it:
If the programs you're looking for aren't there, you can go to Run, type in 'msconfig'. Go to the Startup tab. In there, you can unclick any programs you don't want to start up.