Nothing. Keep it.
Think of it this way. You want to mitigate using that hard drive space for storage of your program files correct? Well Vista is only going to use that storage if you run out of RAM. But if you turn it off Vista will have to wait for more RAM to free up. So... if you have plenty of RAM the swap file will rarely be used, BUT if it is used, it is much faster than the alternative you are wanting - which is the processor waiting on more RAM to free up.
MaKe sense? So while i understand you don't want to use that slow hard drive space for extra storage during processing, your only other option would be to turn off page swaping. If you did that, your processor is going to act SLOWER than if you had the processor putting files within your hard drive.
And jesus, seriously do you even use 6 gig of ram? Monitory your usage of it before doing something drastic like turning off your swap file.