Network wide upgrades...

C0RR0SIVE

Golden Master
Messages
9,213
Location
Lexington, KY
Ok, so, this may be a bit difficult to explain/show and get some GOOD input on.
First, below is a crude drawup of my network as a whole.
network_zpsf7387235.jpg


Now, as you can see, I have pushed everything onto one single machine known as Server-2, and it's starting to choke rather badly, not because of the CPU or RAM limitations for what it is doing, but because of the disks, and network feed.

My current plants are to make the rest of my network 10/100/1000 + 1x 10Gbit feed on one of the switches.

Now, I stream a crap-ton of 1080p HD content all over the home, that is the most basic network map I could come up with, but I have several more machines on the network that stream, and that also perform backups to Server-2. Needless to say, some files can cause enough data through put when streaming that it takes the whole 1Gbit feed for it self, at this point I can forget about doing anything else on the network for the most part, ESPECIALLY if someone upstream (closer to the router) starts streaming, as most the network is stuck behind a 100Mbps pipe.

As you can tell, this also causes issues when accessing the internet.

Now, why have I pushed so much into one machine? Well, it's due to electrical costs... I can't afford to be powering several machines when one is, for the most part, capable of running everything when I can use VM's.


Now, here is what I need to accomplish.

1: Server-2 NEEDS to be rebuilt entirely, with MASSIVE storage options, but still be highly efficient, and not pull too much power. It still needs to be high powered for what I am doing.
I also need a board that has x4-x8 slots instead of x1, good controllers are hard to find at x1, and x1 bogs down arrays too much. I need the ability to have 3-5 arrays, all arrays NEED to be Raid-6. I also need this server to have 1x 10Gbit port

Please remember, the goal is to keep it as low powered as possible, while performing previous tasks, but expanding upon how capable it is in my network.

2: I need a new switch, needs to have one 10Gbit port that is compatible with the 10Gbit port on Server-2

3: I need to replace Server-1 with a very low powered system, but will have the ability of having 10-20TB worth of storage. This system would be nothing more than a location for the entire network to back up to on a weekly basis, removing that role for the most part from Server-2.

Suggestions? Ideas? I have around $9,000 going into this project, but, the cheaper it is, the better I am off, just remember the goals, large amounts of storage that need to be capable of putting out for every device easily on my network, Server-2 needs to have all its current roles figured into the upgrade path.

Also, SERVER-2 will need a new case, potentially in the future I will have to have upwards of 20 SATA/SAS drives in it, same goes for SERVER-1.


I have been going over this for months trying to figure out the most cost effective way in the long run. Don't get me wrong, Server-2 is great as is, but the motherboard lacks in the area of me not being able to get GOOD SATA controllers added in along with a faster NIC. If the board on this rig had x4 or x8 slots I would be fine.

Just remember, low cost, low powered, but high performance if at all possible guys. Discrete GPU is not needed provided integrated can handle todays 1080P video easily.

If you wish to help, try to list what machine you build out in detail along with the name of the machine according to my diagram. Please provide a reason for it all in the end guys... Because this is such a big task and I have fallen behind on the times when it comes to building a machine that has to meet these kinds of goals.

Assume that at any given point in time, all machines on the network could be accessing the media drive array for reads and reads only. No one is permitted to write except the server, so read performance is a must. But write performance is a must on the backup arrays.


Don't be afraid to restructure my network, just note that between the two switches is around 130 foot, if you can put more machines in the network at a lower price along with reducing power consumption or keeping it the same, then do so. I expect consumption to go up due to the increase in arrays some.
 
Last edited:
Hello,

Is this your primary goal? Why not consider NAS'es with RAID configured, you may store Terabytes of files and they are very affordable.
large amounts of storage that need to be capable of putting out for every device easily on my network
You may use the second server for your VMs and Server Roles including database role but again if you will be storing massive files I'd utilize NAS.

Of course...just my thoughts, others might have different input than mine. :wink2:
 
A high quality NAS capable of handling 10Gbit worth of streaming various types of data is nearly impossible to find. It is cheaper to build an actual PC for these functions. Most sub-$500 NAS get lucky to push 30MB/s at most, I am needing far more than that on the connection.
 
Ok, so I have decided to do the following in the end.

Upgrade main switch to full Gigabit + 10Gigabit
Rebuild HTPC in all rooms for a lower overall power consumption.

Build a "Master" server that will handle every single thing in my network efficiently, and provide ample storage space for the entire network in the long run.
After 1 year build new server for redundancy.

Below is the current purchase list:

6x 2TB Western Digital Caviars: N82E16822136792 (Will create different arrays, will add more drives in future)
1x Netgear Prosafe GSM7328FS-100NAS : N82E16833122208
1x Intel E10G41AT2 : N82E16833106043
1x crappy GPU : N82E16814125404
1x Silverstone ST1000-p : N82E16817256057
1x Corsair Vengeance 32GB 8*4GB kit : N82E16820233263
1x Gigabyte GA-X79-UP4 : N82E16813128562
1x Intel Core i7-3820 LGA2011 : N82E16819115229
1x Rosewill RSV-L4000 : N82E16811147155
1x Samsung 830 128GB SSD : N82E16820147163


Total comes to around $4200, stupid switch might as well be half the cost.... This doesn't include all the new cables I plan on running, nor the closet I am building. Also, can someone recommend a GOOD UPS that will power a modem and the above listed equipment for ~5-10 minutes in the event of a short power outage?

Does anyone notice anything incompatible with any of that build? Especially between the NIC and Switch I picked out? I kinda wanna stay away from fiber as much as possible, but if a switch with 10Gbit fiber and a nice with 10Gbit fiber is cheaper, let me know. Never had to look at things of this nature in the past.
 
router you selected needs the optional 10gbit module to support 10gbit connections.

take a closer look at description of the router and it reads 10 gbit bays.

have you considered using link teaming method?

an intel 4x1gbit nic would provide 4gbit aggregated connection, along with the nic on motherboard for total of aggregated 5gbit bandwidth.
not sure which OS you plan on using, but Link aggregation - plantforms support seems to be pretty good.

with such a big project i assume you'll be using the ZFS file system with the samsung SSD as cache? if not, its something to think about.

and lastly, quite frankly I am surprised you said some files you stream can take up the whole 1gbit link by itself. i find 1080p content encoded at modest 6mpbs to be quite acceptable.
 
Last edited:
I have thought about teaming with multiple gigabit nics, but that's just more cables, and still need a good switch or router capable of handling such a thing.

I have one file that is well beyond 1080p (actually above 4k if I remember right) that murders my network, have to watch it from the server directly with an SSD, even then quiet often it has issues playing and it's due to disk output. I don't plan on having such content added in the future, but being able to pump out 1080p to every machine on the network, and still do network backups at the same time would be nice. But I still have a few 1080p files that for some reason, most likely due to how it was encoded, enjoy flooding my gigabit connection, could also be a software glitch with the media server software and client end causing the flooding.

The purpose of the SSD is more for the OS that will be installed, along with the media software packages that the house uses, PLEX Server after a certain point, due to cache reasons bogs down heavily because it can't pull data quickly enough.

Got a suggestion for a good gigabit switch that can handle teaming that wont break the bank like the 10Gbit switch/nic theory would? Keep in mind, I will have 2-3 of these servers built next year on top of the first one, so the switch would either need a decently high port count (48 port ones exist... but at what cost?) and be capable of such bandwidth, or I would need multiple switches that can use some form of backbone connection if I am thinking right.

I also don't think I could put the motherboard NIC into a team with the Intel add-on, could I? Or does the motherboard NIC actually support that particular feature?

Like I said earlier, never bothered with things of this nature till I noticed issues on the LAN here recently with so many people in the home pulling from one machine.

Trying to stay away from "server" grade motherboards for one reason, overly expensive, and processors are way too pricey for what I am doing IMO.

Thanks for the help so far fat.clown, also, I was planning on using Server 2k3 as all my software is 100% compatible with that in the end, moving to any other OS would be very difficult, some of it really hates newer systems, though, I guess I could use a different base OS, and put the software that I use in a VM, but that would potentially create issues I would assume.

If you know of an OS that can handle the following all at one shot, let me know.
Stealthbot 2.7 CB (C++/Java varient, rather picky, runs a few hundred custom built visual basic scripts)(Due to how it is built and works, it MUST run in a VM to isolate it from WSUS and the web server services.)
Webserver
WSUS
PLEX Media Server
PS3 Media Server
FTP
Networx for monitoring WAN traffic on the router while ignoring all local traffic.
And generally everything else this server would be doing, such as teaming of the nics, the raid array as it will be software driven most likely in the end due to the size requirements until I decide to hop for a pair of dedicated controllers for something rather fancy.

In the end, this server will have to handle data from 14 different machines, not all at once in most cases, but it is possible that all devices will be in use at one time.



EDIT: You think this switch would be acceptable since it seems I will be going the teaming way with several gigabit ports per NIC? It appears to be capable of supporting teaming, has plenty of ports in the end, and appears to be capable of supporting enough total bandwidth in the network. http://www.newegg.com/Product/Product.aspx?Item=N82E16833122079
 
Last edited:
That switch looks like it'd work. but honestly, I think your project is out of my league. so you may want to into it more yourself.

Like you, I am not an expert with networking and such. all of points i made in my previous post was the results of my research when I needed a NAS myself. In the end I went with the easy way out and purchased a Synology NAS.

at least you didn't purchase that netgear and find out it doesn't support 10gbit connections out of the box. :)

PS for those few problematic videos you may try checking out Net Meter to see if it is indeed your connection. or fragmentation? something else?
 
It is indeed a bandwidth issue, either the switch, a PCI-e bus, or just the nic on the server that is in use. Connected at gigabit, but all tests show I can only get ~600Mbps on it.

I would go the NAS route, but the cost of one that would meet my performance and space requirements would likely cost just as much as the server with out drives would.

Just looked at a few, far cheaper in the long run to build my own set of servers, though, I like how some the synology rack units can link together and create larger drive arrays, that may be in my far future though when 4k video and larger becomes standard.
 
Last edited:
What do you use to stream whatever your streaming from the server? And what do you put on the server to need that much space O.O

I have like 30 different shows of all the seasons on one 500gb external o_O

Just curious I guess to what all this is needed for.
 
I have close to 800 movies, and 15 tv shows, currently have 90 complete anime series, most all is 1080p. I use PLEX, and various other programs as listed earlier.

I also stated how fast my library grows earlier. I also tend to keep a 1:1 backup of everything on a large raid array. When dealing with stuff the way I do, you tend to not have all the disk space that you would have, due to redundancy. In theory, ~20TB of disk space would only be ~15TB of storage for me if done in R5, which I am planing on doing a R55 which means I loose even more disk space, but, have decent redundancy in the event I have several drive failures.

When you and all your friends get together to create a massive library as I have you will use more than 500GB, and externals suck when it comes to having several machines streaming from one machines drive array.
 
Last edited:
Back
Top Bottom