yeah, not sure why you wouldn't want a "server-based" OS. It's exactly the same thing without the GUI related packages. They use the same repos, same kernel, etc etc.
I'm a fan of CentOS -- You can use the Minimal installer and get nothing but the base packages required to have a working system. Then, if you want, it's relatively easy to install packages required for a GUI installation:
yum grouplist
^ lists all available install groups
yum groupinstall "Name", ie
yum groupinstall "GNOME Desktop"
or
yum groupinstall "Cinnamon"
will install the Gnome / Cinnamon desktops and all required packages.