current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 9 down vote favorite
2

Possible Duplicate:
Bash while loop and reading from pipe

I come from web programming background, and find myself interested in one peculiarity of using a local shell. I understand that when a program reads from file, it can read at whatever rate's necessary. But I'm curious to know, how does it work when a program gets other program's input piped in and can't process it in realtime?

A good example would be video encoding. Suppose I point a decoder to video file, then it's output gets piped in as encoder's input. The total size of decoded video is more than ram+swap, so I guess there's no way to have it buffered altogether. I found about read and write calls to stdin and stdout, but I'm interested to know what actually happens when this example's encoder can't cope with all the data at once. Does it somehow informs the decoder about desired rate? Does decoder program needs to be specifically prepared for such a signal and modify it's processing speed accordingly? If not, how does it all gets balanced in the end?

share|improve this question

marked as duplicate by Gilles, jasonwryan, warl0ck, Jim Paris, rahmu Oct 30 '12 at 9:30

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
    
+1 for thinking about how things work. Always encouraging to see. –  itsbruce Oct 29 '12 at 14:36

2 Answers 2

up vote 5 down vote accepted

When a writer writes to a pipe and the pipe is full (its size is limited to a few kilobytes), its process blocks until one of the readers frees some space. Similarly, when a reader reads from a pipe, its process blocks until there is something there.

There are also asynchronous writes and reads that a programmer can use to queue up these reads and writes.

I highly suggest reading Beej's Gudes, starting with Beej's Guide to Unix Interprocess Communication

You call it with two arguments, the first is a sleep time for the writer, the second is a a sleep time for the reader. Try calling it with args 0 3 and 3 0

#!/bin/sh
write_sleep=$1
read_sleep=$2
writer(){
  echo writing output >&2
  echo hi
  sleep $write_sleep
  echo writing output >&2
  echo hi
  sleep $write_sleep
  echo writing output >&2
  echo hi
  sleep $write_sleep
}
reader(){
  while true; do
    echo getting input >&2
    read input
    [ $input ] || { echo input is empty >&2 &&  break; }
    echo $input
    sleep $read_sleep
  done
}

writer | reader
share|improve this answer
up vote 4 down vote

Pipes are usual unix descriptors at the programmatic level. When you setup two programs to communicate through a pipe, all they see at first is the standard output and standard input descriptors, which they interact with, because the shell did setup them that way. There are some peculiarities with these descriptors when setup this way, but processes interact with them as with any other file descriptor (it's the whole unix philosophy, you know).

Basically, the writer process may write as much as he wants in the pipe, but, because the pipe has a maximum storage limit (a buffer), it may be blocked if the pipe is full, or the request may be answered as "try again later" if the writer didn't want it to block.

Conversely, the reader may read as much as he wants, but it may also be blocked (or answered with "try again later" as well) if the pipe is empty.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.