Well you should have a thorough understanding of CPU overclocking before you tackle RAM overclocking.
RAM overclocking requires adjusting the timings, loosening them up and at the same time raising the Frequency and even the Dimm voltage is generally required.
There's a certain point you reach where loosening the timings will give you no gain even when you add more MHz behind the RAM. You gotta find the 'sweet spot' if you will of the tightest timings and highest frequency for the best transfer.
RAM generally doesn't get too hot, so if you don't have heatspreaders I wouldn't worry about rushing out to by one. RAM won't get hot unless you're pumping some major voltage through it and really high clock speeds.