Actually, if you are running a 64-bit OS, it can handle 8GB of RAM (4 x 2GB). If you are only running a 32-bit OS, it is limited to 4GB.
Since you have an Nvidia 590 SLI MCP chipset, you are limited to only SLI. According o the specifications for that model, you have either a 750W or 1KW PSU, so you are good to go for power.
A pair of GTX 260's would be a great combo, making your CPU your limiting factor.
The Dell specs are pretty vague as far as the best CPU you could drop in your rig. It only says "Intel Pentium 4, Intel Pentium D, Intel Core 2 Duo". I checked the BIOS update for your model, but it was just as helpful. >.< A little Googling found that the E6400 seems to be the top dog, but I would hazard to guess that's what you are running (2.13 GHz).
Here's your spec page, by the way: