My main question here is this: what tools can I use to determine what is going on inside the H710P? For example, if I knew how busy its processor and cache are I could avoid some of the guesswork I'm doing.
The details: I built a nice Linux NFS server (using CentOS 6.4) with a 720xd, H710P raid controller, and for now I have 5 Micron P300 SSD's in a RAID 0 set. Unfortunately, the write speeds never get above about 250MB/sec even with a dozen NFS clients writing to it over 20Gb Infiniband (sadly, even with 1 gigabit they seem to be able to easily hit the limit).
I realize that there a lot of layers here, but I have a pretty good understanding of the Linux side. However, the H710P is a black box to me -- are there any tools to help get a better insight as to whether or not the controller is at its peak performance? I have done a ton of benchmarking, but the most interesting test results are with iostat and soft irq's. Here is what "iostat -xk 1" has to say about it:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 51804.00 92.00 3748.00 2808.00 221544.00 116.85 4.45 1.16 0.23 87.20
Sorry if the fonts above are showing up proportional even though I set them to monospaced! The important columns are: wrqm/s 51804, wkB/s 221544, util 87.20%
This is about as fast as it can go. It is interesting that top shows a ton of Soft IRQ's (%si) -- it routinely gets above 75% but all on the first CPU core.
Also interesting - before I started NFS I did a raw write test to the device with 4 disks in RAID 0, and it was getting about what I would expect: 1.3GB/sec (individual disks top out at about 350MB/sec). However, when I re-created the raid with 5 disks, the raw write speed slowed down to 1.1GB/sec when I was expecting > 1.6GB/sec. What can cause this kind of slowdown? Does the H710P hate odd numbers of disks, or do I possibly have a flaky SSD, or "other"?
Thanks!