Your OC settings were simply inadequate. Overclocking takes experience and knowledge. You have to learn how much voltage is needed at a particular increase or decrease. To much and to little wont work very well. Also you need a cooling system, so you can OC safely. You will come to a point where you need to configure your RAM latency.
Since you ask that type of question, its clear your not ready yo. You need to learn what your doing. Also, when your computer shuts down like that from windows. Theirs a huge chance you will end up corrupting windows. This means windows cannot operate normally, because you have somehow damged it from OC stress. So all it takes is a simple OC failure, then windows will not boot up. Depending how much computer knowledge you have, will determine if you can fix this problem. Most of the time you must reboot windows completely. Also, do not OC directly from windows, these types of OC failures will lead to HUGE PROBLEMS.
Overclock through your BIOS!