No there are not any downsides unless! its crossfire(ATI) or SLI(NVIDIA), so if you buy a crossfire board you can use one graphics card yes and have the other slot empty, although if you put an nvidia card in and want to add another then crossfire wont work. I do think there are hacked drivers to make it work though but obviously not the best way to do it.
so in conclusion: yes you can have sli or crossfire mobos and still only use 1 card and leave the other slot empty.