This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.
3
votes
0answers
44 views
Read everything in a pipe's buffer without waiting
Is there a way to read everything already in the buffer of a pipe, pipe them to another program, and exit immediately without waiting for any more input?
It has to be binary safe. It's ok to require ...
-1
votes
0answers
55 views
Files in ram (tmpfs) representing a single-producer multi-consumer buffer [closed]
constraints: My system has 250MB RAM memory available. The amount of
data generated by the producer lies around 5MB/sec. I need to provide
a buffer of around 30 seconds. So around 150MB...
...
2
votes
1answer
22 views
Is it possible to give caching preferences to the filesystem?
I have a directory (/srv/data) with a large number (20,000,000) with a number of small (4-40KB) files. These files are all located in subdirectories of /srv/data, where the concatenation of the ...
3
votes
1answer
62 views
Unbuffered socat command to connect serial ports in remote machines and log the data
I'm currently using the following command. It reads from serial ttyUSB0 in local machine, and bidirectionally connects to ssh, through two tee commands for logging. On the remote end, socat connects ...
3
votes
1answer
64 views
How to get the file path of a buffer?
I know that the register % contains the full path of the current buffer. But how to get the full path of another buffer by its number?
Is there such a function/command in VIM?
I want to explain how ...
0
votes
0answers
24 views
How to use the Linux possibility to expand buffer or redirect buffer to a file before running the python script? [duplicate]
My script is sending output to its stdout or stderr pipes. The operating system will buffer some data then block the process forever when the pipe fills.
How to use the Linux possibility to expand ...
0
votes
2answers
28 views
locate(1) buffers its output [duplicate]
I'm using locate(1) from GNU findutils for a little task and it seems as if it buffers its output. I am piping the output of locate to another task that will process the lines as locate finds them. ...
2
votes
1answer
38 views
regular awk buffering issue when calling shell commands
Some precisions:
regular awk, NOT gawk
AIX 6.1, and old shell: GNU bash, version 2.05b.0(1)
I am trying to display some things in the right order, hence I do pipe outputs through the shell's "sort" ...
1
vote
2answers
46 views
Misbehaving irssi Pagination in tmux
I'm experiencing some strange console pagination behaviour in tmux when SSH'd to a Ubuntu host (EC2 instance on AWS). I've demonstrated this behaviour by running irssi inside the tmux and hitting PgUp ...
1
vote
1answer
87 views
buffer (not cache) usage growing, find the process using it
I've a system where the buffer usage is growing over the time.
I trace it while monitoring the memory usage with dstat ex:
# dstat -t -m
----system---- ------memory-usage-----
time | used ...
9
votes
1answer
122 views
“Leaky” pipes in linux
Let's assume you have a pipeline like the following:
$ a | b
If b stops processing stdin, after a while the pipe fills up and writes on stdout from a will block (until either b starts processing ...
1
vote
1answer
110 views
Help Avoiding Buffering With Output Redirection
I'm trying to write some functions I can utilize in my scripts to essentially tag all output as Error vs Standard output, throw date/time stamps on it, and also include the name of the function that ...
0
votes
3answers
198 views
piped command after grep not working
I want to monitor a file. So I have tailed a file using below command, to execute a script per line.
tail -3f logfile.log | grep "action.*add" | sed -u -e "s/^/'/" -e "s/$/'/" | xargs -L 1 -P 5 bash ...
2
votes
1answer
62 views
How to reduce Linux' write buffer for removable devices?
When writing content to removable devices on Linux (USB sticks/HDDs, SD cards, etc), I often see incredible write speeds in the first few seconds, sometimes in the order of GB/s (filling of the write ...
1
vote
1answer
121 views
Filter out duplicate lines of tail -f [duplicate]
I often need to tail -f apache access logs for websites to troubleshoot issues- one thing that makes it annoying is that anyone loading a page once may cause 12+ lines to get written to the log, and ...
2
votes
1answer
101 views
non-blocking/multi-threaded cat
cat is blocking from what I see in the code, i.e. it uses blocking read() and then it uses blocking write().
I want to call some tool where I disable all stdout buffering by purpose (e.g. like ...
3
votes
2answers
76 views
stdbuf supposed behavior for subprocesses
I didn't exactly find something about the following in the man-page. How is the supposed behavior in subprocesses spawned by a process which was itself spawned by stdbuf?
E.g.:
stdbuf -oL myprog
...
0
votes
0answers
60 views
Is there a way to edit video buffer size in google chrome in linux?
Its easy to do in windows but how to do it in google-chrome in cent OS 7.
NOTE :I think google-chrome here is not using adobe flash perhaps it does not even need it but still i can play videos.
3
votes
0answers
44 views
Does Linux perform buffering twice when hosting a VM?
I ran a KVM VM on my CentOS host and found that after performing a HDD-intensive operation, it used all of the host's memory I assigned to it. However, free on the guest suggests that the memory's ...
4
votes
4answers
4k views
Restrict size of buffer cache in Linux
Is there a way to tell the Linux kernel to only use a certain percentage of memory for the buffer cache? I know /proc/sys/vm/drop_caches can be used to clear the cache temporarily, but is there any ...
4
votes
3answers
285 views
use line buffering when piping output to file
I have a perl program that prints to stdout, which I then pipe into a file, like this:
./skript.pl > file.txt 2>&1
This makes the output of that skript buffer. However, I would like to be ...
3
votes
2answers
350 views
Avoid output buffering for a command group (curly braces) in bash script
I have a bash script containing a group of commands in curly braces { ... }. This group contains some initial echo commands and then one loop. At each iteration the loop executes various slow commands ...
1
vote
2answers
854 views
How to identify cause of large buffer memory usage?
I have a production machine running a 2.6 kernel (Centos 6.7) with 64 GB of RAM of which 18 GB is apparently used for buffer memory.
I am interested in identifying specifically what that 18 GB is ...
0
votes
2answers
144 views
awk output has extra carriage returns and is buffered
My ultimate goal is to get a zenity progress dialog to work while encoding with fdkaac.
I first started with some code that is working when I encode "aa.wav" to "aa.mp3" with lame. This results in a ...
2
votes
0answers
199 views
Understanding inotifywait, pipes and buffers
I want to monitor every file change in a directory with inotifywait. inotifywait shall write to a FIFO buffer, which then can be read, leisurely. While experimenting with relatively huge amounts of ...
0
votes
1answer
105 views
Wanted: nonassociative array in AWK
I am looking at speeding up some code in AWK by serially processing an
internal string representation instead of serially processing
an array representation. (For Examples 1 and 2 below assume ...
4
votes
3answers
241 views
How to append data to buffer in shell script?
I'd like to do following using shell script: (for simplicity, I use same data for INPUT. In real case, the data changes with loop label jj)
#!/bin/sh
for jj in `seq 100`; do
cat INPUT.file >&...
8
votes
1answer
2k views
Increase FIFO size limit
Thanks to the answers to my other question, I now understand that FIFO on Linux, i.e. /dev/xconsole has a buffer limit of 64 KB.
How can I increase this limit to 128 KB?
Apparently, I will need to ...
3
votes
1answer
64 views
How do I ensure all data to an SSH pipe is sent, despite lengthy disconnects?
I'm on a laptop with intermittent internet connectivity. (i.e. I sometimes don't have network for a week.) I want the output of a process on my laptop to end up on my server. It all needs to get ...
-1
votes
1answer
105 views
Why readdir() only reads 32K of directory entries at a time?
readdir() only reads 32K of directory entries at a time.
Why does it read only 32K entries? Is it dependent on buffer or any other parameters?
Can I change that value so that I can read as many ...
5
votes
3answers
595 views
How to forward between processes with named pipes?
The /tmp/in, /tmp/out and /tmp/err are named pipes, already created and opened by some process (for reading, writing and writing, respectively).
I would like to create a new process that pipes its ...
1
vote
1answer
35 views
Why combining two unbuffers give this error?
I am reading this answer here and run test code
unbuffer xxd -ps /usr/bin/telnet | unbuffer -p less
but get Missing filename ("less --help" for help).
Why combining two unbuffers give this error?
5
votes
3answers
512 views
Why does this one-liner fail to produce output? [duplicate]
The following produces nothing in bash:
while true ; do upower -d ; sleep 1 ; done | grep percentage | uniq
I've discovered that it doesn't matter what the last- or even the second to last- program ...
0
votes
0answers
63 views
Backup server with tar, encrypt and directly transfer the result, AFTER BUFFERING IT
So I've been investigative how to make full backups with tar, and I found that tar is a good tool if the backup is to be made once, and if the server has enough disk space to handle the output file.
...
1
vote
1answer
641 views
Formatting grep output with awk. Simple case and background case
I'm struggling with this problem. I have this line:
mplayer *.* 2>/dev/null | grep Playing
which just launchs mplayer and only the line matching Playing filename is printed on screen.
This works ...
1
vote
1answer
115 views
Function tracing per thread
I have a complex multi threaded application running on a Cent OS 5.8
the application is coded using C and C++
I am searching for a FUNCTION TRACING tool which can help me do the following.
Trace all ...
1
vote
0answers
307 views
How can I find which process is eating my memory? [duplicate]
My Linux system sometimes reaches 90% memory usage.
We are running some Java processes, and top shows overall usage as 45%.
If I stop the running processes, usage reduces to 5-10%. I want to list ...
2
votes
4answers
688 views
Completely buffer command output before piping to another command? [duplicate]
Is there a way to only execute a command after another is done without a temp file?
I have one longer running command and another command that formats the output and sends it to a HTTP server using ...
3
votes
3answers
181 views
What to use instead of `buffer` for 1GB buffer? [duplicate]
buffer(1) seems to be old-ish and have hard-coded values preventing it to cache large amount of data.
$ buffer -m 1G
max_shmem 1 too low
// it doesn't even understand gigabytes
$ buffer -m 1000M
...
21
votes
2answers
2k views
Pipes, how do data flow in a pipeline?
I don't understand how the data flows in the pipeline and hope someone could clarify what is going on there.
I thought a pipeline of commands processes files (text, arrays of strings) in line by line ...
1
vote
1answer
507 views
Is it useful for a proxy to buffer less data than Linux RCVBUF can?
I am looking at how HTTP proxies and reverse proxies deal with slow client problems. The idea is that the upstream server only have a limited slots for clients and if the client is slow to receive ...
0
votes
1answer
1k views
How to enable a non-root user to empty the linux buffer cache
I'm required to empty the linux buffer cache in a python script, that runs on a Debian wheezy VM.
As root I run sync; echo 3 | sudo tee /proc/sys/vm/drop_caches, but the script is run by a user, ...
3
votes
3answers
1k views
Buffering (named) pipe in GNU OS
In the GNU OS a process can only write data to a pipe if another process reads the same data (from the same pipe) at the same time.
Is there something like a pipe which lets the 1st process write and ...
0
votes
1answer
39 views
:bnext not cycling through all buffers in vim
For some reason :bnext is not cycling though all the buffers shown by :buffers
I have 5 buffers open but only 3 buffers are visited when I used :bnext repeatedly.
How can I cycle through all the ...
4
votes
1answer
98 views
Disable buffering in a VM configuration
I am executing some benchmark using a DBMS and virtual machines. And I am using this command
free && sync && echo 3 > /proc/sys/vm/drop_caches && free
on each Guest, and ...
4
votes
1answer
250 views
Meaning of fsync() in sshfs+LUKS setup
Background: I'm investigating methods for encrypted storage on untrusted machines. My current setup uses sshfs to access a LUKS-encrypted image on the remote machine, which is decrypted locally and ...
1
vote
1answer
87 views
Use saved input for scripts
I have a few scripts that I constantly enter in input as below:
[support@ccfcore-3 exporttool]$ ./sendExamToRepo.sh
Enter the Study Instance UID:
2.25.98472613238667427310842065102689842379
The ...
1
vote
2answers
147 views
Specifying height for vertical splits in vim
Usually I am editing two files, A and B, split horizontally. I set it up, so that one buffer would only occupy the first 2 lines at the top of the screen, and the rest is for the other buffer.
...
3
votes
1answer
293 views
Does “mount -o remount,ro” flush filesystem buffers?
On Linux, does mount -o remount,ro flush filesystem buffers/cache, or should I also run sync to achieve that?
6
votes
2answers
2k views
What should be the buffer size for the sort command?
I have a machine with 2 TB of RAM and I am running a sort command on a file of size 150G where I have specified the buffer-size as 1000G, after doing my bit of reasearch on google, I got this piece of ...
