Would something like this be a noticable upgrade?
It should. It is a 2.083 GHz chip, versus the 1.something already in it.
Oh, and OEM means CPU only, so, yes, you'd have to buy thermal paste.
You have an AGP slot. Even though it is only a 4x, it is still better than a PCI slot. PCI-E 16x is the current standard, but your motherboard doesn't have it. So be sure that whatever card you look at/buy is an AGP card.
Here's NewEgg's list of 4x/8x AGP cards: