Performance & Scalability Considerations

[Steviebone]

I am designing a small load balanced system of web servers which will each be running a single web application that is database search intensive. My initial thoughts are to have single multiprocessor machine with a large striped raid array handling the data mining and several less expensive front end boxes serving the requests (and static data) in a load balanced configuration.

Am I correct in assuming that since a networked high performance share would be handling the intensive file i/o that the cpu's in the front end servers would benefit from noticeably lower loads? Or would the difference be offset by network traffic overhead?

Furthermore, how much of a performance loss will I take reading from a network share raid as opposed to a local one of the same configuration?

For example, lets assume a dual processor machine and a hardware raid stripe w/out partity reading from multiple SATA 3.0 drives? Assuming a GB(1000) nic/switch, at what point does the speed of the network transfer limit the transfer from the raid and diminish the speed gained by reading from the stripes?

 

[root]

sounds like you are giong to have to shell out quite a lot of money regardless of your final solution...


At the last place I worked we setup a system simillar to this for the purpose of an online point of sale system to server the entire of europe, (many thousands of outlets).

so I'll give you a couple of suggestions...

Data minning is always going to be resource intensive.

this is my suggestion.
first use linux.
get two duel processor servers each with stacks of Ram, each with onboard SCSI drives.
then get an external SCSI drive with two SCSI connections, (something like a Dell Power Vault).

then get a really fast database server (something like Oracle). and set up a cluster of the database servers with shared harddisk storage (you'll need at least two network cards in each Database server, 1 in each for outside connectiovity, on to connect to the other (with a crossover) for the heartbeat to keep the database syncronised.

congratulations you just spent,
£1000 per server
£1000+ on a power vault
£9000 on a database program,

BUT... you'll have a premium setup that will easily serve several hundred concurrent connections over a massive database, (which could be thousands of concurrent website connections).

Now for the webservers...
it's difficult to say what you'll need here, it really depends on the machines that you have, and howmany connection that you want to serve...

Apache
look in the default httpd.conf file as to the active threads allowed, now you can increase this (to as much as even four fold the default). but performance will suffer, it's probably best to thikn about how many concurrent connections that you'll have and got from there with a new server.
e.g. say you want to have 50 active threads.
but expect 130 connections at any one time, get three servers.

Load balancing is something that I'm not too hot on so I'll refrain from suggesting anything.