Alright I think I've found the problem.
I was getting like a 3 second lag and sometimes game crashes in CSS before. I've managed to make it less annying and it doesnt seem to crash.
What I think is happening is that the graphics cards hyper memory is stealing all the remaining memory in my computer. So when a game needs to load a new sound effect of visual effect it has to first take memory off of the video card, then put it in for what it needs. This is what I think causes the lag because after I've been playing source for a good 15 mins, it goes away and you can play normally.
It's a bit annoying but I guess I can just live with it. Too bad I cant find a way to force the graphics driver to leave 100 megs of memory untouched so I dont get the lag, but whatever.