current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 3 down vote favorite
3

I would like to use the free memory for write cache. Linux does that automatically, but it has some weird flushing algorithm that I would like tuned for my use.

I have recently used ddrescue to copy a disk. It reads a block and writes the block to disk.

What I would like is to have the reading read as much as possible until there is no more free then block until some space is freed, while at the same time the writing writes as fast as possible freeing up memory.

So if the writing disk is faster than the reading disk no memory will be used, and if the writing disk is slower than the reading disk then all the memory will stay used until the copying is complete.

In both scenarios the memory usage remains constant (unless other programs take some of the free memory).

But what I see is this:

Write cache

The reading fills up the cache (while little to no writing is done) and when the cache hits a limit (or after er period of time) then the reading is blocked while all the cache is written to disk. It is almost as if a troll is running 'sync' every now and then.

I have tried changing the /proc/sys/vm/dirty_* to no avail: The only change I have been able to provoke is how tall the tops are. So an answer that mentions /proc/sys/vm/dirty_* will have to provide tested values that measurably flattens out the saw tooth graph.

A tuning that will tell Linux to start writing ASAP but never block reading or writing if there is free space, will most likely suit my purpose.

I am aware of tools like mbuffer. They will flatten out the graph, but they are not dynamic (i.e. releasing memory when the free mem is low) and they do not allow for seeks, so they will not work with programs like ddrescue (which has to seek back and forth).

share|improve this question
    
Which filesystem are you testing against? What kernel version are you using? –  Kyle Jones Sep 29 '13 at 0:52
    
No file system: Raw device IO. Linux 3.2.0. –  Ole Tange Sep 29 '13 at 0:59
    
This seems to me to be a case of premature or misguided optimisation - and the best you can hope to achieve is to make sure that reading the source disk is continually stopping and starting (as small amounts of cache become free) rather than long sequential reads. –  cas Sep 29 '13 at 2:18
    
@CraigSanders Let us say the disk is 1 TB. Let us say we can read at 100 MB/s and write at 50 MB/s. The current setup it will take 1000000/100+1000000/50 = 30000 seconds to copy. If I can get my setup it drops to 20000 seconds as read and write happens simultaneously. A 33% speedup is hardly premature optimization. Also I will avoid the system blocking other disk I/O while the sync-like writing is taking place. –  Ole Tange Sep 29 '13 at 8:34
    
33% is wildly optimistic. your read and write will chug along at a decent speed only until the buffers are full. then your read speed will be maximum of as fast as you can write (i.e. as fast as you can clear buffer space for incoming reads) but more likely a lot slower than that. the problem being that a lot of stop-start reads meaning that the reads will never get anywhere even close to max throughput and is more likely to result in slowdown due to rotational delays (waiting for the sector to come around again). –  cas Sep 29 '13 at 9:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.