Artifacts are areas of blank colour that, generally, shouldn't be there. If you OC your card's core or memory too much (as I have been doing tonight and I STILL can't hit 80000 on aquamark3!), you'll see things on the screen that shouldn't be there. You'll know when you see them, they usually manifest themselves as triangles of solid colour that link points between, say, branches on trees, or they may fill an area that should be shaded, and so on.
Believe me, you'll know it when you see it!