I wouldn't worry about taking it off. Go ahead and take off both panels, just beware that dust will be able to easily get into your case, as well as other hazards. Just don't let any pets get near it, if anything put the case on a desk to where nothing can get to it.
nothing wrong with taking the side off. cools it down really well. my dad hates it when i do it. i need to make my own computer so i can do what i want with it.
The only problem with taking the side panels off is if you have a good cooling system. It wont allow the air to get to where you want. It will be too generalized (the air you are wanting to pump to the CPU will be spread everywhere) and wont be as effective as with the sides on.
If you dont have a good system though, then taking the sides off would help.