Doesn't seem to be a connection problem. Connection problems cache (buffer) the whole stream and pauses it, not just the video or the audio separately.
Sometimes the connection is just fine but other problems exist like the stream itself of very low quality that the encode originally dropped frames, you are viewing a very low bitrate stream, your computers is not powerful enough, your operating system is has issues, drivers, etc.
However, if you wanna make sure the connection is the problem, test your device (computer) on another connection.
For me it seemed to be a hardware problem. My OptiPlex 960 with 8GB RAM was regularly stuttering or outright freezing. At least two or three times while just listening to Darkness Prevails on YouTube, with actual videos like movies or TV shows way worse. My OptiPlex 7010 with 16GB RAM has no problem. Was told on DSLReports forum it had to do with CPU, RAM, network card or graphics card.