OK, at least a couple things to check out:
Did you lock AGP/PCI? If not, do so at 66/33.
How is your PSU, is it undervolting?
Have you tried any changes to vcore or vdimm?
I guess I just won't be able to overclock or something.
No, you just need to read more and do more testing. I would at least isolate the problem between the RAM or the CPU. Lower the multi to x11 and try again. Keep the RAM at 1:1. If it works, it is not the RAM. You can keep going by continuing to lower the multi but raising HTT and keeping the CPU under 2.4Ghz and see what your RAM can take.
Now, do the same thing with the CPU. Put the multi back to x12. Change the FSB:RAM divider to 4:3 or something, and raise HTT. Check how high you can go and with what vcore while continuing to keep the RAM under 200Mhz. This way you can find your max CPU speed.
Keep note of what vcore and vdimm is needed at speeds, and then put together a combination that comes as close to possible of maxing out each such as 11x240 or whatever it might be.