It could possibly be due to the excess of maps. I used to play Halo: CE, and it used to take forever to start up becaues of all the maps.
Here's what you can try to do:
Take your maps and move them to another folder.
Test the loading times, and see if there's any difference.
If there is, leave the maps out unless loading 3 minutes is fine by you.
If there isn't, well, then it's something else.
It's probably your ping. Your distance from the server is farther than others. When you have a high ping, it's usually the case. Otherwise, it could just be that your comp is slow. I'm not too sure with this one because I don't get those problems...