[Dirvish] slow file transfers

[Paul Slootman]

On Tue 22 Feb 2011, Dave Howorth wrote:
> 
> Some basic parameters for the two filesystems and two others that are
> involved in my tests are:
> 
> data    2.0 TB, xfs,      part of 4 disk md RAID10
> backup  1.2 TB, reiserfs, part of 3 disk 3ware 9650SE RAID5

3ware, there's your problem...

At work we had a 24-disk storage system, with 2 x 3ware 9550 RAID.
We had had trepidations about using hardware raid, as all our earlier
tests had shown that hardware raid basically sucks (OK, this beginning
of the century that we had done the tests). However there was not a lot
of choice when it comes to attaching that amount of disks.

It was terribly slow.

After struggling for a couple of months, I wiped it all, reconfigured
the 3ware controllers to just present 24 JBOD disks, and configured the
raid under linux with md.  That performed a hell of a lot better.

Initial tests with the 3ware controllers didn't show it to be too bad,
but as soon as we hit the system with the normal load (about 3 rsyncs in
parallel) the performance really hit rock bottom. Apparently the 3wares
can't really handle concurrent stuff well.

Recently the cabinet underwent an upgrade, and the motherboard and 3ware
controllers were replaced. We now went with an Areca controller. That
does work quite well; well enough that we haven't bothered with linux md
raid this time.


> I measured the disk speeds with hdparm:
> 
> # hdparm -tT /dev/mapper/vg--backup-lv--storage  207.82 MB/s

Of course, the question is whether the 3ware controller is really
syncing to the disks when it reports that it does. I suspect it's not,
so I don't trust this number.

Also we've given up on reiserfs in the last couple of years, it's never
really been stable with 2.6 kernels (on 2.4 kernels we never had
problems). We're now happy with ext4, although I'm experimenting with
btrfs and besides a number of crashes when I was experimenting with
snapshots (and not really understanding what I was doing ;-) that works
quite well as well.


In short, what I'd do:
- don't use the 3ware raid features, turn it into a JBOD and
  do raid with linux.
- replace reiserfs with ext4 or btrfs. Perhaps xfs if you mainly deal
  with large files.


Paul