Start with memory and increase it by a small amount (1 or 2 MHz) then run an intensive 3d app such as 3dmark 2003 for a few minutes. Check for artifacts (distortion in display). If none, then repeat process until artifacts appear. Once there, go back a few MHz until you have them no more and that's your memory overclocked.
Repeat same steps with core.
I personally prefer radclocker or ATITOOL for overclocking but then, that's your choice....
some poeple also prefer the Omega drivers (do google search) with their cards, again, a question of choice...
My gaming machine:
P4 2.8 GHz
2 X 512 MHz PC3200 in dual DDR mode
Intel D865PERL MOBO
integrated 6.1 MAX audio
Logitech duo MX mouse and KB
ATI RADEON 9800 PRO