[SGVLUG] Distributed filesystems

[Chris Smith]

On 6/23/05, Emerson, Tom <Tom.Emerson at wbconsultant.com> wrote:
> OTOH, I imagine that those bargain-basement cards will actually be WORSE for your 
> environment (driver issues notwithstanding) as I suspect these will be (effectively) 
> "winmodem" style NICS, i.e., they make heavy use of your own CPU to do the work that
> you bought them for in the first place [think of it this way: if you were do hook up the 
> interrupt line to the clock line, you'd overclock your CPU... ;) ]

Interrupts are a huge issue with gigE, but realistically unless you
are unloading TCP/IP processing on to the NIC (read: spend more money
than you should), the overhead of processing the ethernet frame isn't
nearly as big an issue as you might think, unless your NIC is worth
more than your computer. ;-) Instead the big deal tends to be getting
jumbo frame support. This tends to be easy with NICS, but a bit
trickier with the bottom end of the switch market. Dodge the netgear
switch and get SMC's switch that actually supports jumbo frames. It's
not much more expensive and will let you get *way* better performance
out of your network.

> > Copying files from one system to another
> > is not inordinately slow, but anything that requires fast
> > disk performance obviously suffers from use over NFS.
> 
> I'd look into your interrupt rates, cpu load, and other overhead tasks -- while NFS imposes 
> it's own penalty, "simple copying" doesn't do much more than read and write the disk, but 
> you may be running at 99% of your CPU(s) [one, the other, or both...] -- in other words, 
> you've got a little room to spare.

If NFS is pegging your CPU, you either have a CPU that is woefully
underpowered for the network you're on, or you're doing something
wrong (admittedly top can be misleading in this regard). Now, you
could have a problem with just handling the network interrupts, which
is not really a protocol specific issue. See above.

> final thought: the folks that produce Cinelarra (a.k.a. "broadcast2000") also maintain 
> something they call the firehose (or something like that...) which is a method of connecting
> multiple USB-2.0 and/or FIREWIRE ports between two computers to get an outrageous 
> "combined" network speed -- i.e., 5 usb2.0 ports connected between machines should get 
> you 2gigabits or more (isn't usb2 rated at 400mbs, or is that firewire?)

Realistically the only advantage of Firewire or USB over using gigE
these days is that you don't have to worry so much about setting up
your network. Once you do bonding, you have to anyway, so you are just
as well off going with multiple gigE instead of multiple Firewire or
USB.

-- 
Chris