This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.
-1
votes
1answer
33 views
Why readdir() only reads 32K of directory entries at a time?
readdir() only reads 32K of directory entries at a time.
Why does it read only 32K entries? Is it dependent on buffer or any other parameters?
Can I change that value so that I can read as many ...
5
votes
3answers
222 views
How to forward between processes with named pipes?
The /tmp/in, /tmp/out and /tmp/err are named pipes, already created and opened by some process (for reading, writing and writing, respectively).
I would like to create a new process that pipes its ...
1
vote
1answer
30 views
Why combining two unbuffers give this error?
I am reading this answer here and run test code
unbuffer xxd -ps /usr/bin/telnet | unbuffer -p less
but get Missing filename ("less --help" for help).
Why combining two unbuffers give this error?
...
5
votes
3answers
372 views
Why does this one-liner fail to produce output? [duplicate]
The following produces nothing in bash:
while true ; do upower -d ; sleep 1 ; done | grep percentage | uniq
I've discovered that it doesn't matter what the last- or even the second to last- program ...
0
votes
0answers
39 views
Backup server with tar, encrypt and directly transfer the result, AFTER BUFFERING IT
So I've been investigative how to make full backups with tar, and I found that tar is a good tool if the backup is to be made once, and if the server has enough disk space to handle the output file.
...
1
vote
1answer
131 views
Formatting grep output with awk. Simple case and background case
I'm struggling with this problem. I have this line:
mplayer *.* 2>/dev/null | grep Playing
which just launchs mplayer and only the line matching Playing filename is printed on screen.
This works ...
1
vote
1answer
38 views
Function tracing per thread
I have a complex multi threaded application running on a Cent OS 5.8
the application is coded using C and C++
I am searching for a FUNCTION TRACING tool which can help me do the following.
Trace ...
1
vote
0answers
121 views
How can I find which process is eating my memory? [duplicate]
My Linux system sometimes reaches 90% memory usage.
We are running some Java processes, and top shows overall usage as 45%.
If I stop the running processes, usage reduces to 5-10%. I want to list ...
2
votes
4answers
88 views
Completely buffer command output before piping to another command?
Is there a way to only execute a command after another is done without a temp file?
I have one longer running command and another command that formats the output and sends it to a HTTP server using ...
1
vote
0answers
49 views
What to use instead of `buffer` for 1GB buffer?
buffer(1) seems to be old-ish and have hard-coded values preventing it to cache large amount of data.
$ buffer -m 1G
max_shmem 1 too low
// it doesn't even understand gigabytes
$ buffer -m 1000M
...
20
votes
2answers
1k views
Pipes, how do data flow in a pipeline?
I don't understand how the data flows in the pipeline and hope someone could clarify what is going on there.
I thought a pipeline of commands processes files (text, arrays of strings) in line by line ...
1
vote
1answer
151 views
Is it useful for a proxy to buffer less data than Linux RCVBUF can?
I am looking at how HTTP proxies and reverse proxies deal with slow client problems. The idea is that the upstream server only have a limited slots for clients and if the client is slow to receive ...
0
votes
1answer
262 views
How to enable a non-root user to empty the linux buffer cache
I'm required to empty the linux buffer cache in a python script, that runs on a Debian wheezy VM.
As root I run sync; echo 3 | sudo tee /proc/sys/vm/drop_caches, but the script is run by a user, ...
2
votes
3answers
105 views
Buffering (named) pipe in GNU OS
In the GNU OS a process can only write data to a pipe if another process reads the same data (from the same pipe) at the same time.
Is there something like a pipe which lets the 1st process write and ...
0
votes
1answer
23 views
:bnext not cycling through all buffers in vim
For some reason :bnext is not cycling though all the buffers shown by :buffers
I have 5 buffers open but only 3 buffers are visited when I used :bnext repeatedly.
How can I cycle through all the ...
3
votes
1answer
66 views
Disable buffering in a VM configuration
I am executing some benchmark using a DBMS and virtual machines. And I am using this command
free && sync && echo 3 > /proc/sys/vm/drop_caches && free
on each Guest, and ...
4
votes
1answer
201 views
Meaning of fsync() in sshfs+LUKS setup
Background: I'm investigating methods for encrypted storage on untrusted machines. My current setup uses sshfs to access a LUKS-encrypted image on the remote machine, which is decrypted locally and ...
1
vote
1answer
85 views
Use saved input for scripts
I have a few scripts that I constantly enter in input as below:
[support@ccfcore-3 exporttool]$ ./sendExamToRepo.sh
Enter the Study Instance UID:
2.25.98472613238667427310842065102689842379
The ...
1
vote
2answers
108 views
Specifying height for vertical splits in vim
Usually I am editing two files, A and B, split horizontally. I set it up, so that one buffer would only occupy the first 2 lines at the top of the screen, and the rest is for the other buffer.
...
3
votes
1answer
169 views
Does “mount -o remount,ro” flush filesystem buffers?
On Linux, does mount -o remount,ro flush filesystem buffers/cache, or should I also run sync to achieve that?
6
votes
2answers
874 views
What should be the buffer size for the sort command?
I have a machine with 2 TB of RAM and I am running a sort command on a file of size 150G where I have specified the buffer-size as 1000G, after doing my bit of reasearch on google, I got this piece of ...
10
votes
1answer
1k views
What exactly is scrollback and scrollback buffer?
What are "scrollback" and "scrollback buffer" in programs like bash and screen, and how do they relate to the tty, the programs being run, and stdin/ stdout/ stderr?
Here is the only definition of ...
3
votes
1answer
120 views
Artificially prime buffer cache?
Is it beneficial to artificially prime the buffer cache when dealing with larger files?
Here's a scenario: A large file needs to be processed, line by line. Conceptually, it is easy to parallelize ...
5
votes
3answers
465 views
Piping commands, modify stdin write to stdout
echo Hello World | nano - | less
I want to modify Hello -> Goodbye while in the the text editor.
It does not have to be with nano.
I am stuck with nano reading stdin but requiring me to write to a ...
1
vote
1answer
348 views
How to control buffering of pipes in bash scripts?
I need my script to be executed with stdbuf -i0 -o0 -e0 otherwise user will wonder why there is no output (becuase of pipe buffer). How can I acheive it?
#!/usr/bin/stdbuf -i0 -o0 -e0 /bin/bash
...
0
votes
1answer
224 views
Preventing a new instance of a program to open when calling it
My problem is with FBI (frame buffer image viewer) for Linux, but my question and any answers are perhaps applicable to more than only that program. I don't know. Anyway…
While things are loading on ...
2
votes
1answer
556 views
Difference between “blockdev --flushbufs” and “sync” on Linux?
Is there a difference in practice between running blockdev --flushbufs and sync(1) on Linux? (apart from blockdev flushing for a specific device, and sync being system-wide).
sync(1) manpage says it ...
6
votes
1answer
992 views
`unbuffer` or `stdbuf` for removing stdout buffering?
Is there a difference between unbuffer(1) and stdbuf(1)? From what I gather, unbuffer makes more than the "best effort" of calling the libc function set(X)buf at the beginning, and then letting things ...
0
votes
2answers
3k views
How to reduce buffers\cache
A monitoring system keeps alerting that my machine is reaching\breaking through it's RAM utilization threshold which is 15 GBs.
I've done some reading and understood that the apparent RAM utilization ...
2
votes
1answer
91 views
Why don't I see a file being copied right away with ls?
Is it due to the buffering in memory? I did cp /network/large_file . & and I kept doing ls -lh to monitor progress, but the file didn't appear until it was about 400MB. Linux 2.6 kernel, Bash 4, ...
8
votes
3answers
3k views
vim - how can i do a 'change word' using the current paste buffer
I have some text in my paste buffer, e.g. I did a yw (yank word) and now I have 'foo' in my buffer.
I now go to the word 'bar' and I want to replace it with my paste buffer.
To replace the text ...
4
votes
3answers
882 views
Are there any ways or tools to dump the memory cache and buffer?
As the title, I want to check out what files are loaded in buffer. Just so it's clear, the buffers & cache I'm referring to are what show up when you run the free -m command:
$ free -m
...
0
votes
1answer
1k views
very high buffer memory usage [closed]
Is this normal - I've never seen buffers so high?
$ cat /proc/meminfo
MemTotal: 16332808 kB
MemFree: 1112264 kB
Buffers: 10630124 kB
Cached: 699400 kB
SwapCached: 0 kB
Active: 3262924 kB
Inactive: ...
1
vote
1answer
547 views
Daemontools multilog loses log line time information. How to fix it?
I am using daemontools to monitor a process and its output log. I am using multilog to write the logs to disk.
The run script for the log is:
#!/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
cd ...
4
votes
2answers
541 views
Does sysctl kern.bufcachepercent not work in OpenBSD 5.2 above 1.7GB?
I run OpenBSD/amd64 5.2 on a system with 12GB of RAM, and I want to use about 6GB to 8GB for filesystem caching.
By default, 5.2 amd64 comes with sysctl kern.bufcachepercent set to 20 (20%); I've ...
4
votes
1answer
824 views
Ksh loses data after piping 16K bytes
I recently found that ksh may lose some data after printing more than 16K bytes to the stdout if it is blocked for a couple of seconds.
This test.sh script prints out 257*64 (16448) bytes:
...
3
votes
0answers
87 views
xdelta3 fails to decode on block device
I'm trying to use xdelta3 to bring a remote block device into sync with a local one. I'm able to easily generate a xdelta3 patch file using the command:
xdelta3 -e -B 33554432 -v -9 -I 0 -s ...
4
votes
1answer
568 views
Switching between split buffers in vim
recently starting trying to use VIM as my full-time text editor since I spent a lot of time SSH'ed anyways. Recently installed NERDTree so I can quickly swap between files in a project.
Easy question ...
6
votes
2answers
2k views
linux stdbuf - line-buffered stdin option does not exist
The man page for the stdbuf command states that line-buffered mode is invalid as a standard input buffering option.
What is the reason for this?
tail -f access.log | stdbuf -iL cut -d' ' -f1 | uniq
...
4
votes
2answers
6k views
Alternate FIFO device for Linux with a (way) bigger buffer while still having filesystem semantics?
I am developing something that consists of a program I made generating output, and a program I did not make (and would very much like not to have to alter it) consuming it. I am using a named FIFO ...
12
votes
4answers
2k views
Is there a way to sync only one partition?
Is there a way to sync only one partition instead of all partitions?
Something like "sync /dev/sdc1".
1
vote
5answers
1k views
Display output to console while grep is used
In vim I use the following command to compile a tex file:
pdflatex\ \-file\-line\-error\ \-shell\-escape\ \-interaction=nonstopmode\ $*\\\|\ grep\ \-P\ ':\\d{1,5}:\ '
this works in terms of getting ...
9
votes
4answers
1k views
Utility to buffer an unbounded amount of data in a pipeline?
Is there a utility that I can stick in a pipeline to decouple read and write speeds?
$ producer | buf | consumer
Basically, I want a utility buf that reads its input as fast as possible, storing it ...
5
votes
2answers
1k views
Prevent unwanted buffers from opening
I'm using emacs for my daily javascript editing, to switch between buffers I use C-x LEFT and C-x RIGHT and I'm fine with this (even if I find it difficult to know the path of the file I'm modifying).
...
12
votes
1answer
8k views
Buffer size for capturing packets in kernel space?
Going through the man page of tcpdump, it seems kernel can drop the packets if the buffer is full. I was wondering if:
that size is configurable and/or
where can I see the size for my distro?
From ...
53
votes
4answers
51k views
How big is the pipe buffer?
As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote:
Another reason to avoid | true is that if the command ...
16
votes
1answer
2k views
Is there truth to the philosophy that you should sync; sync; sync; sync?
When I was first introduced to Linux, working at Cisco Systems in 2000, I was taught the merits of the sync command, used to flush buffers to disk to prevent filesystem corruption / data loss. I was ...
3
votes
1answer
303 views
Ex and Vi: buffers and regexes
I want to put regex matches to buffer but cannot get even simpler example y :.put myNewBuffer return msg [line] [put] [buffer] working ie how to put something to buffer?
So to regular expressions ...
14
votes
2answers
2k views
How to change the name of a vim buffer
Is it possible to change the name of a buffer in vim?
Specifically, I'm using Conque Shell to open shells in vim (each shell is in a buffer) and with multiple shells, I see:
10: bash - 1
11: bash - ...
