current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top
up vote 14 down vote favorite

Summary

How can you configure Linux to both read from a local disk/filesystem and write to a network share at the same time, as opposed to reading while no data is going over the network, then sending that data over the network while the local disk is idle?

It is much faster to read and write at the same time instead of only performing one operation and then the other in an alternating fashion.

Details

I am moving a large amount of data from local disks on a Linux machine to a NAS device.

I am using rsync to basically copy /srv/data into /mnt/nas, which is a CIFS mount.

It started off well, reading at 100MB/sec and writing to the NAS at 100MB/sec (limit of gigabit network), with both reading and writing happening simultaneously.

However now, a few hours later, I am finding that it is reading from the local disk, then stopping the read while it writes to the NAS, then when there is no more data to write to the NAS, it resumes reading from the disk again. The network is idle while the disk is being read, and the disk is idle while the network is in use.

Needless to say, reading 200MB then writing 200MB takes much longer than reading and writing that 200MB at the same time.

How can I configure the kernel such that it sticks to the earlier behaviour of reading and writing at the same time, rather than alternating between reading then writing, performing only one operation at a time?

Some observations: When the local disk reads at 100+MB/sec everything seems to happen in parallel just fine, but once the disk slows down (seems to be going at only 20MB/sec now for some reason) that's when this read/write switching seems to happen.

I can also run sync manually every few seconds to get the writes happening in parallel with the reads (though obviously at the reduced speeds) however putting sync in a while loop so that it runs every five seconds doesn't seem like the right solution...

The kernel seems to cache about 1GB of data and then write it out over the network as fast as possible - which is fine - I just don't understand why the slow disk needs to stop being read while the data is being sent out over the network.

share|improve this question
1  
Most unix tools are absolutely not optimized for bandwidth in this sense, not rsync, not even a simple cp. They are single-threaded apps using blocking IO. – peterh yesterday
1  
Somewhere around 100 MB/s is also what you can expect to see on modern common 7200 rpm rotational HDDs in purely sequential workloads. It goes down once you start seeking, such as for updating metadata or if the file system is fragmented, because you then become IOPS-bound. – Michael Kjörling 23 hours ago
    
can you install rsync on the NAS? – Jasen 14 hours ago
up vote 21 down vote

After some more investigation, it looks like this issue is less kernel related and more about how rsync and CIFS interact.

As far as I can make out, what is happening is that when rsync closes the destination file, CIFS (and probably any network filesystem) ensures the file is completely flushed and written to the remote disk before the close syscall returns. This is to assure any application that once the close operation completes successfully, the file has been completely saved and there is no risk of any further errors that could cause data loss.

If this wasn't done, then it would be possible for an application to close a file, exit thinking the save operation was successful, then later (perhaps due to a network problem) the data could not be written after all, but by then it is too late for the application to do anything about it, such as asking the user if they want to save the file somewhere else instead.

This requirement means that every time rsync finishes copying a file, the entire disk buffer must empty out over the network before rsync is allowed to continue reading the next file.

A workaround is to mount the CIFS share with the option cache=none which disables this feature, and causes all I/O to go direct to the server. This eliminates the problem and allows reads and writes to execute in parallel, however a drawback of this solution is that the performance is somewhat lower. In my case, network transfer speed drops from 110MB/sec to 80MB/sec.

This may mean that if you are copying large files, performance may well be better with the alternating read/write behaviour. With many smaller files, disabling the cache will result in fewer cache flushes each time a file is closed so performance may increase there.

It seems rsync needs an option to close its file handles in another thread, so it can start reading the next file while the last one is still being flushed.

EDIT: I have confirmed that cache=none definitely helps when transferring lots of small files (brings it from 10MB/sec up to 80MB/sec) but when transferring large files (1GB+) cache=none drops the transfer from 110MB/sec down to the same 80MB/sec. This suggests that the slow transfer from many small files is less about the source disk seeking, and more about having so many cache flushes from all the small files.

share|improve this answer
2  
That it a very interesting problem, and thanks for posting the explanation. rsync does read the file in a different thread (actually, different process) because it's designed so that one copy of rsync is running on each side of the network, even though in your case both copies are on the same side (and the filesystem is hiding the fact that there is a network). I guess it doesn't help, because the reader process very very quickly fills up the pipe while the writer process is blocking on a close(). rsync would perform better if you were using rsync on the wire, not CIFS. – Celada yesterday
1  
I would imagine that another solution is you can't run rsync on the NAS would be to use rsync over the network (like rsync -a files localhost:/dest/path) while somehow artificially introducing a huge buffer (like, multple megabytes, at least) into the network connections. Not sure what the best hack for doing that would look like. – Celada yesterday
    
@Celada: Thanks! Yes I imagine running rsync on the NAS box itself would work around the issue as well. Bit more complex though (weird NAS permissions, have to drop symlinks, etc.) but if I had a little more data to copy it would be worth the time investment to do that I think. – Malvineous 23 hours ago
2  
Possibly unrelated to your case: I had a similar problem a few years ago writing the output of dump(8) to a NAS mounted over NFS. At the time I diagnosed the problem as maxing out the CPU on the NAS, because of the combined effect of the NFS server and a firewall running on the NAS (the box wasn't rooted, and the firewall couldn't be disabled completely from the web interface). The problem went away when we replaced the NAS with an old PC. FWIW. – Sato Katsura 22 hours ago
    
@SatoKatsura: Definitely a possibility for older NAS devices, although in that case I imagine you would see a slower overall transfer rather than a bursty one like this? My NAS is a dual-core Atom (~2GHz) which sits at about 30% CPU use when maxing out one gigabit NIC without jumbo frames so should be ok there. – Malvineous 13 hours ago

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.