Unix systems tend to favor text files, often consisting of one record per line. Most unix configuration files are text files. Unix systems come with many tools to manipulate such files. Most tools process the file in a stream: read a line, process it, emit the corresponding output; this makes it ...
0
votes
3answers
73 views
Count number of lines in files then compare which has more (BASH)
I need to count the number of lines in x files and compare which has more.
The one I've done only takes two files and compares them. Any idea how to make it x amount of files?
echo Enter the ...
6
votes
1answer
81 views
A shell script for joining two files
I want to write a shell script that get two files A and B, and get a result like this:
File A:
user_a tel_a addr_a
user_b tel_b addr_b
File B:
process_1 user_a
process_2 user_a
process_3 user_b
...
2
votes
3answers
72 views
Sorting columns from two files with awk
I have two input files
28a39a289906c01159f999a68996091a [email protected]
274d1d2c7e931fb55ac0c91dd41f2be7 [email protected]
44d25d3b1b70b240d5058f1be1cef576 [email protected]
2227a768f6d253b7bf81bb4ecb15b52d ...
1
vote
1answer
36 views
Tools to dump entries from arbitrary sets of log files between 2 dates
I had a outage incident a few days ago and the support team has asked for any available logs for the period, while thinking about how to do this, it occurred to me that it would be generally handy to ...
3
votes
3answers
91 views
How to perform an action only on the first line?
sed 's/.\(.*\)/\1/' myfile
Say myfile contains:
abc
def
ghi
I want to remove the first character only from the first line but the above removes it from all the line.
2
votes
2answers
59 views
Grep: search and replace full line
The command
grep "foo" myfile.txt
prints all matching lines in my file.
Now I want to replace the full line with another string. How can I do that?
0
votes
2answers
63 views
What is the python equivalent of grep -v?
I like grep -v. I use it all the time. But I am also doing some text processing in python, and there is one crucial thing that I lack.
Usually, I use grep -v to take extraneous stuff out of text.
...
4
votes
1answer
73 views
How to output file & ignoring lines that start with “?”?
I do svn status --show-updates and then I want to either
Q1:
ignore (not to display) lines that start with ?
Q2:
display only lines that start with * Note that there are few spaces before * occurs. ...
3
votes
4answers
197 views
How to add newline to a file?
Using version control systems I get annoyed at the noise when the diff says No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid off those messages?
2
votes
4answers
56 views
separate a file into several small files according to columns
I have a data file, which can have N rows, and each row is composed M elements separated by white space. Currently, I want to separate each row into several segments. In other words, assume the ...
2
votes
6answers
121 views
How to swap columns in such a file?
I have a text file, each line is stored like this :
"Video or movie" "parent" "Media or entertainment" "1" "1" "1" "0" "0"
I want to swap the columns 3 with 2, i.e.
"Video or movie" ...
1
vote
3answers
67 views
grep equivalent of the kwrite regex [A-Z][A-Z]+
So, it took me ages, but I finally learned to think in terms of regular expressions, thanks to using them in kwrite.
But I still don't know how to translate that knowledge to grep. I love my grep, ...
2
votes
2answers
66 views
need shell script to grock a csv
Need some pointing in right direction on script to fetch and regex or sed.
Site24x7 provides a URL with a CSV list of their source IP's used for monitoring. (they also provide other formats, CSV ...
2
votes
2answers
55 views
How to sort (by whatever key) BibTex entries in `.bib` file?
I work with LaTeX and do versioning with Git. For bibliography management I use Mendeley.
The problem is that each time Mendeley synchronizes it's .bib exports,
they are in different order, what ...
3
votes
2answers
54 views
How to expand tabs based on content?
I've got some tab-delimited data coming out of a Unix pipe. I'd like to format this data into a compact human-readable table.
How can I expand these tabs into spaces, and automatically set the tab ...
4
votes
2answers
44 views
Is it possible to use split to make character chunks out of Chinese unicode bytes?
For a while, I've been dealing with Chinese unicode text. Of course, the usual rules apply. I can grep for characters the same way I'd do so for words. This is very useful to me.
But there's one ...
2
votes
1answer
28 views
!fmt for current line only
When I want to format part of text with fmt in vim I usually select parts of interesting lines in Visual Mode, with: Ctrl+V Arrows, and than type !+f+m+t.
When there are couple of lines they are ...
6
votes
4answers
73 views
extracting certain info from output
I'm trying to extract certain info from ffmpeg output.
Sample ffmpeg output:
configuration: --enable-memalign-hack --enable-mp3lame --enable-gpl --disable-vhook --disable-ffplay --disable-ffserver ...
21
votes
3answers
778 views
How to remove duplicate lines inside a text file?
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).
What I need is to remove all the repetitions ...
4
votes
2answers
42 views
Extract part of the first line of a file
I am trying to configure some external commands in Gedit3, for compiling LaTeX files. All works well, except for the following. I have a LaTeX document, consisting of many parts. Each file begins with
...
2
votes
4answers
72 views
Linux equivalent of Cisco's “begin” command
In the Cisco IOS, you can filter and manipulate the output of commands using various built-in tools. Some of these have clear Linux equivalents. For example, 'exclude' and 'include' provide 'grep ...
16
votes
9answers
2k views
How can I write to the second line of a file from the command line?
I have an external program that produces an output file (largish, 20K lines possible).
I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
7
votes
5answers
255 views
Randomly draw a certain number of lines from a data file
I have a data list, like
12345
23456
67891
-20000
200
600
20
...
Assume the size of this data set (i.e. lines of file) is N. I want to randomly draw m lines from this data file. Therefore, the ...
5
votes
2answers
115 views
sort inplace - like sed --in-place - exists?
Am I blind or is there no option like --in-place for sort?
In order to save results to the input file, sed uses -i (--in-place).
Redirecting the output of sort to the input file
sort < f > f
...
9
votes
4answers
167 views
Delete whitespace for a set of lines in Vim editor
I have some text like the following in a file:
sample text
some random text
even more random text
text with no indent
worst indention
I need to delete the empty space before each ...
6
votes
1answer
92 views
How to detect and delete lines containing ˆ@
I have a simple problem:
In my file, the are lines containing the string ˆ@ˆ@ˆ@ˆ@ˆ@ˆ@. I just want to delete all lines with this string, using for example the sed or grep commands.
And I would like ...
9
votes
2answers
511 views
How can I find out how many lines a text file contains without viewing it?
How can I find how many lines a text file contains without opening the file in an editor or a viewer application? Is there a handy Unix console command to see the number?
3
votes
1answer
34 views
Wrapping long cells in a tsv to keep them in same column
Formatting a TSV with long cells with column makes its readability very low. It simply breaks lines like this:
$ python3 -c 'for i in (1,2,3): print(((str(i)*50)+"\t")*3)' | column -s $'\t' -t
...
1
vote
2answers
111 views
How to search and replace text in all php-files in a directory and it's subdirectories
I am looking for a shell script that recursively traverses all .php files in a directory and performs a search & replace of a particular text pattern.
The search pattern is quite long ( > 5000 ...
3
votes
2answers
67 views
Extracting bigrams with sed
How do I split an entire text consisting of only letters into bigrams.
For example:
odd even
od -> de
dd -> ev
This is what I have so far but it does not produce the desired result.
[some ...
1
vote
1answer
48 views
How to display TSV (csv) in console, when empty cells are missed by: `column -t -s $'\t' `
I have file with columns spearated with tab.
I have file when some rows have empty cells (on begining, in middle).
In such cases column -t -s $'\t' simply fails:
Input:
$ echo -e ...
3
votes
2answers
28 views
changing the order of items in a list
I have a file with plain text. This file contains a list like this:
@"Joe", @"211",
@"Bob Nelson", @"132",
@"Jack Sierra", @"3422",
@"Walt Robert", @"14",
The list goes on and on with thousands of ...
0
votes
2answers
59 views
list the difference and overlap between two plain data set [closed]
Possible Duplicate:
Linux tools to treat files as sets and perform set operations on them
I have two data sets, A and B. The format for each data set is one number per line. For instance,
...
2
votes
3answers
78 views
Count distinct values of a field in a file
I have a file contains around million number of lines. In the lines I have a field called transactionid, which has repeatitive values. What I need to do is to count them distinctly.
No matter of how ...
4
votes
6answers
225 views
How can I convert tab delimited data to comma delimited data?
I'm requesting a list of ec2 snapshots via amazon's ec2 command line tool:
ec2-describe-snapshots -H --hide-tags > snapshots.csv
The data looks something like this:
SnapshotId VolumeId ...
3
votes
2answers
116 views
use awk to replace parentheses?
How can I use awk to replace the parentheses in these 2 scenarios?
Change empty parens to the word "None"
192.168.0.24,Up,()
Remove parens to only host.domain.com is left
...
5
votes
3answers
155 views
Any (text based) plotting tools in unix toolbox?
I was wondering if there are any generic plotting tools available in the unix toolbox?
By plotting I mean simple bars or things you see in many curses based utilities.
And by unix toolbox I mean ...
2
votes
5answers
92 views
evaluate multiple patterns from program output and write into pattern specific files
I have a script outputting some value/numbers and I want to split those into two files. I am looking at something like:
./runme.sh | grep 'ook' >> ook.out | grep 'eek' >> eek.out
Where ...
1
vote
1answer
81 views
Reading two files into an IFS while loop — Is there a way to get a zero diff result in this case?
I have a text file full of several hundred lines of sequences like this:
b 29.
b 52.
c 84.
c 83.
c 94.
c 93.
c 61.
b 38.
c 81.
c 92.
c 28.
c 37.
...
1
vote
3answers
164 views
Print odd-numbered lines, print even-numbered lines
I want to print the odd-numbered and even-numbered lines from files.
I found this shell script which makes use of echo.
#!/bin/bash
# Write a shell script that, given a file name as the argument ...
2
votes
4answers
83 views
Compare two files
I have two files in UNIX.
1st file is Entity and Second File is References. 1st File has only one column named Entity ID and 2nd file has two columns Entity ID | Person ID.
I want to produce a output ...
9
votes
2answers
789 views
How to split stdout to go to several output files?
Say, I have a command command which prints huge number of lines to stdout:
line1
line2
.....
lineN
I want to save the output to disk, but not as a single file, but as a sequence of files each ...
2
votes
2answers
116 views
How to execute this particular shell command from Python?
OK, so, I have this non-functional shell script, which I am rewriting piece by piece in python, except I am getting an "unexpected "|" error" from the shell (see below):
#/bin/sh
LINES=`cat $@ | wc ...
6
votes
4answers
223 views
Efficient in-place header removing for large files using sed?
The commands below may takes minutes depends on the file size. Is there any more effient method?
sed -i 1d large_file
2
votes
1answer
74 views
How to follow (a la “tail -f”) a binary file from the beginning?
Is it possible to follow a binary file from the beginning, a la tail -f?
This is useful in some cases, for example if I'm scping a file to a remote server, and at the same time I want to feed it to ...
8
votes
6answers
214 views
Delete lines beginning with #
How do I delete lines beginning with a #, given that there can be whitespace on the left and right of the #?
# Master socket provides access to userdb information. It's typically
4
votes
2answers
95 views
How to search and replace multiple needles by one word via one expression?
Assume you have a text file:
foo fnord bar
bizz foo poit
And now I would want to replace both "foo" and "bar" into "narf".
I know I could use:
sed -e 's/foo/narf/g' -e 's/bar/narf/g' fileName
...
2
votes
4answers
156 views
Trouble getting awk output in for loop
I'm trying to create a script that will check a website for a word. I have a few to check so I'm trying to input them via another file.
The file is called "testurls". In the file I list the keyword ...
5
votes
2answers
91 views
How can I wrap text at a certain column size?
I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
6
votes
4answers
100 views
Fastest way to determine if two sorted lists contain unique elements
I have two sorted files A and B where the size of A is much greater than B, e.g. A is 100GB while B is 50MB. I want to quickly determine if there are any lines in B that are contained in A, stopping ...
