I used to have Audigy 2, it didn't take me long at all, but that was when I had my gaming computer. (Intel P4 3.6GHz, Mushkin 2x1024mb ram, Nvidia GeForce 256 DDR)
Do you use Norton AntiVirus? If so, thats the problem. Norton doesn't like it so there are conflicts sometimes and the mixer doesn't start. I switched to Trend Micro and haven't had a problem since.
joshd: Its normal for it to take a few seconds.. its basically a hidden program. It isn't the same as the windows volume bar... its like your hiding a program in the taskbar, so it takes a minute to load just like a normal program.