yes you can mix ati and intel... you can mix and match whatever brands you like. In your price range, like i've suggested above, the phenom and a radeon card will be your best bet. You can go with the 5770 i suggested, or a 4870 or 4890 like others have suggested.. they're all close in price and performance.
As for memory, since ddr2 and ddr3 are pretty much the same price, why not go with ddr3? More future-proof. So i've selected socket AM3 which supports DDR3 memory.