Hard Disk IDs in Linux

Nicholas Leippe nick at leippe.com
Fri Mar 15 09:37:04 MDT 2013


On Thu, Mar 14, 2013 at 4:25 PM, Daniel Fussell <dfussell at byu.edu> wrote:
> Did you have the disk caches turned off?  I read a few months ago that
> the md driver does outperform hardware raid, but only with the caches
> turned on.  When they were turned off, it's speed dropped to be the same
> as hardware raid, if not slightly worse.

It depends greatly on the workload and particular brand of hw raid
controller. Years ago I did some extensive benchmarking with a few
different brands and models and discussed the results on some of the
mailing lists. (This was 8+ years ago.) The consensus was that so long
as you had spare cpu cycles in your workload, md raid5 was nearly
always faster than hw--due to the fact that you had a 1+GHz x86 cpu
doing the parity calculations vs a 200MHz arm cpu doing them. For
raid1, md was faster than the Dell PERC controllers simply because
their firmware was terrible. Even with doubling the transfers across
the PCI bus, md significantly outperformed their hw raid
implementation (using their card as a JBOD controller to the same
disks!). Reflashing their cards with LSI firmware (when possible) gave
the same cards a big performance bump, but still wasn't enough. The
explanation at the time was again CPU speed: a 1+GHz x86 cpu
processing SCSI tag-command queues faster than a 200MHz arm cpu could
do it.

I did many of the tests with write-through, write-back, and no cache
modes--all depending on what the hw controller cards allowed--some
didn't expose all or any of those controls. I don't recall specific
results, but that in general, yes it did affect the results. In some
cases significantly, but it was very work-load specific. Of course the
cache mode can present additional risks in failure scenarios. I didn't
try to test these scenarios.

My results are obviously not current. For all I know the hw raid
vendors have greatly improved their products since then. There are
features of hw raid that can be useful: battery-backed caches,
multi-path, hot-swap PCI{,-X,-E}, and of course, off-loading some of
the I/O and parity calculations. If your workload demands it, they can
be a real win. IME, their firmwares and management tools were very
lacking--quite disappointing given their "enterprise" status and cost.

Some of the hw raid controllers implemented "raid 1.5" years before md
implemented its raid10 driver (same thing). In theory the hw could
outperform md in this case (and also raid1) simply by reducing bus
transfers for writes in half. In practice (that I've seen), the PCI
bus had plenty of spare bandwidth given its massive bandwidth limit
compared to disk write speeds at the time. If you had other I/O
competing on the same bus it could become a significant factor--again,
always workload specific.

</rambling>


More information about the PLUG mailing list