current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.
up vote 11 down vote favorite

Dell PowerEdge 2950 with two 1 Gbps NIC's going to two 1 Gbps ports on a switch which then goes to a NetApp with four 1 Gbps NIC's that present as one virtual interface. 24 drives, 7200k SATA, NetApp RAID-DP. I've mapped each host NIC to the NetApp using MPIO in the Microsoft iSCSI initiator. Testing with SQLIO my write throughput appears reasonable at about 200 MBs, but my reads are closer to 100 MBs.

Shouldn't my reads be closer to 200 MBs just like my writes? Is this a configuration problem or is there a fundamental storage issue I don't understand?

enter image description here

Update: Here are IOPS for the random workload. The reads make sense, though, I'm not sure what to make of 20000 for the writes. SAN cache is 3.2 GB. SQLIO tests are against a 25 GB file.

enter image description here

share|improve this question
1  
Do you have results for a random workload also? –  Mark Storey-Smith May 29 '13 at 14:59
3  
What's your cache on the NetApp device? Do you have a SAN admin who can pull some metrics for you? We have a NetApp and were able to identify a few issues with a combination of reports and warning logs. Ultimately, our situation was a bad fiber card, but NetApp support was quite helpful in helping us root cause. –  swasheck May 29 '13 at 15:07
2  
It might be worth looking into the configuration of your aggregates and volumes to make sure your disks are being used correctly (feel free to post your configuration, though I'm not sure how many of us are NetApp experts). It's normal for writes to be faster than reads, because writes can be cached on the filer before they are pushed to disk, but reads have to hit the disk unless they're already in the cache. –  Nathan Jolly May 29 '13 at 15:15
2  
@mrdenny Where does this "99% of IO in 64k blocks" notion arise from? Bob Dorr indicates otherwise, as does Wes Brown. Even if we were to ignore those two comprehensive articles, surely common sense dictates that you're going to see 8K IO on a platform that uses an 8K page size. –  Mark Storey-Smith Jun 10 '13 at 15:32
2  
@MarkStorey-Smith In my experience reads in 8k occur typically correlated with fragmentation. Also could indicate memory trashing, low page lifetime due to scans evicting pages (ie. most of the extent is still in memory). A well tuned system should show 64k reads. Writes of course depends on what is actually dirty. –  Remus Rusanu Jun 10 '13 at 17:08

2 Answers 2

up vote 7 down vote

Disk writes are actually going to memory (NVRAM) on the filer, to be flushed to disk later - on an idle filer, these will be incredibly fast, and iops of 20,000 are quite believable (you'll see similar speeds from most SSDs).

Reads, on the other hand, need to come from disk unless they already happen to be in the filer's read cache (which, unlike writes, are on volatile memory).

It's hard to pin storage vendors down on iops for spinning disks, but for a 7200RPM drive, 80-120 iops is quite believable. Considering that you've probably lost a couple of disks to NetApp's RAID-DP and/or spares, 2,200 iops is close to what you could expect from 22 disks performing around 100 iops each.

This may not explain your read speeds (your disks may not be doing the full 2200 iops when you're performing a sequential read), but it may at least help explain your write performance.

share|improve this answer
    
Thanks Nathan. Should I be expecting double the throughput with two NIC's and MPIO? –  Henry Lee May 29 '13 at 17:20
1  
Are you able to check the utilisation on your filer while you're running your sequential read tests? If it reaches 100%, your bottleneck for those is likely to be at the filer (either due to configuration or iops limitations on each disk) and MPIO / extra MPIO connections won't add anything. Your write throughput might increase further. –  Nathan Jolly May 30 '13 at 0:05
up vote 5 down vote

For posterity, after much trial and error, we figured out how to get the expected throughput.

As mentioned above, the NetApp had one virtual interface backed by four physical NICs. The host has two NICs and I had configured MPIO through the MS iSCSI Initiator so that there was a path from each NIC to the one virtual interface. The results were the throughput above - writes made sense at close to 200 MB or the speed of two NICs, but the reads were half that or the speed of one NIC.

Upon closer inspection, our SAN guy noticed that traffic was only flowing through one of the physical NICs for the reads. I'm not sure if there was a configuration mistake on our end, but there were two things we tried and both got us our throughput. One was to change from one virtual interface backed by four NICs to two virtual interfaces, each backed by two NIC's. Then map one host NIC to one virtual interface. The other thing we tried was to use "aliasing" on the SAN side to present multiple virtual interfaces. (I'm not a SAN guy, so hopefully I said that correctly.)

My take-away is that we just needed the SAN to present more than one interface so the Initiator truly saw multiple paths. Here is our throughput now:

enter image description here

share|improve this answer
    
why are the smaller writes slower now? –  Jack Douglas♦ Jun 10 '13 at 17:51
    
Not sure, we haven't been able to figure that out yet. I will post back if I figure it out. –  Henry Lee Jun 10 '13 at 19:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.