Moved this from the contact page:

Luke Halls Sep 19th, 2007 at 2:55 pm Edit

We are currently looking at installing a SAN in our production company and i am looking at getting some idea of how much it would cost to use 10GbE. We have 8 macs that we would like to share about 10TB of storage between. I would also like to have an idea of the benefits / drawbacks of using 10gbE compared with fibre. Is this something you could help us with? if not could you point us in the right direction?

1st response from Small Tree:
Hi Luke,

The original idea behind the SAN scenario is that the data path and the meta-data path would be separated allowing more scalability. In theory, that works and I think for installations of your size, it probably works fine. As installations get larger, the notion of scaling the meta-data network and traffic gets tough. There are SAN solutions like Lustre that are trying to solve that. We’ll see.

10Gb to a server would be a great thing, except that with 8 clients, you’re looking at 80Gb/sec of bandwidth to and from the server. I don’t think you’ll get that through anything within your budget.

We’d need to understand what your clients are trying to do (edit SD video off a shared storage location)? and then we could figure out whether Gigabit or 10Gb could meet that requirement. I have customers today serving 4 or 5 stations happily with Gigabit off a server, no special SAN software required.

2nd response from Small Tree:
One really needs to understand the requirements of the customer before we can say that 10Gb ethernet is better (or worse) than a fiber channel install.

The theory behind fiber channel is that each machine has a direct connection to the data path. That’s a good thing. However in order for that to work, one must have enough fiber channel ports (expensive) and some kind of volume sharing or cluster fileserver type software (typically $1000 per station). Additionally, your clustered filesystem will also need a metadata server. The (perhaps slight) advantage of a meta-data server over a normal fileserver is that in a regular file server situation, the server has to be able to handle the entire data path requirement. 8 systems (as in this case) might demand 1.2GBytes per second (assuming roughly one HD stream per system).

The question is whether it’s cheaper/easier/more reliable to buy a really fast server that has a big data path or spend that same money on FC switch ports, cluster software and a meta-data server. Given the feedback and performance data so far, I’m still leaning towards servers.

Now here’s the rub: Customers are always willing to pay a little less to get a little less. Money is generally tight and people like find ways to trim spending. When I talk to customers and they hear the price tag for HD bandwidth to every station all the time, they generally tell me that they don’t need that just yet. Usually, it turns out they need 3 or 4 compressed streams of SD with a total bandwidth of less than 50Mbits/sec.

Given this scenario, I advise most customers with less than 8 clients to consider a server with a lot of memory and striped storage. This generally works pretty well, especially with the latest final cut pro (6?). It does a much better job of buffering than in the past and with 10.5, I expect that solution to get even better. It’s extremely inexpensive compared to any sort of cluster or volume sharing set up with fiber channel.