6800GS is a more powerful card. There's the basic answer.
It takes quite a bit of power for it to smooth out those jaggies on objects. Back when the 9800 was on top of it's game, graphics weren't like they are now where theres just a LOT more things on the screen.
On a newer game there will be a lot more objects and lines that needed to be smoothed out and since your 9800 is kind of getting obsolete these days it's more than it can handle.
The 6800GS was made during these graphic enhancements in games so naturally it should be able to handle it better otherwise it wouldn't be too much better than the 9800