because with the side panel off your at least allowing the ambient air to flow into the case due to a matter of slight pressure change in the air surrounding the case and the cpu fan being able to suck in some of the colder air.
Your case is definitely the culprit for those temps.
btw: why is it that when I came back there where 5 more posts in front of my first response than there were when I logged off last night? lol