Manipulation of text by programs, scripts, etc.
0
votes
1answer
21 views
cut with variable used to provide column to cut [on hold]
I want to cut a column and all columns after, where the column is given as a variable. This is what I've tried:
Var=3
cut -f "$Var"- ~/file.csv > ~/fileNew.csv
I've tried looking into this ...
-1
votes
2answers
28 views
How to use awk and print 3rd column that contains both 'db2' and 'SAP' strings?
cat ost_lvdd01_inbound_DD990_East.csv | awk -F',' ~ /db2/&/sap/ ; print $3
Input
/ost_lvdd01_inbound/1472930000/WPW-DBS003795.amer.warnerbros.com_1472938570_C1_IM:1472938570:/PROD_SQL_TXN_LOG:4:...
2
votes
1answer
21 views
Join two files each with two columns that have multiple columns are matched
I have two files
File A
MUREX_2089975 :SBE Zinc Non-Option 118510 metals USD -308410
MUREX_2097300 :SBE Zinc Non-Option 118510 metals USD 7751
MUREX_2097474 :SBE Zinc Non-Option ...
1
vote
1answer
28 views
Splitting files in Unix using values in a file
I have a txt file which has multiple XML records. I have to separate it using vendorname. I modified the commands as per my requirement. However, it's giving me 2 error messages:
awk: sf.awk:1: /<...
1
vote
1answer
28 views
Count number of values within a range in a specific column
I have to count the number of values that are between 0 and 0.05 in column 11 of a dataset. How do I go about doing this?
-4
votes
3answers
40 views
number lines in a file [on hold]
So I have a file text.txt with lines of some characters
example:
abc
abc
yxz
abc
now I would like to number the lines like this and copy it to another file.
End expected result:
1 abc
2 abc ...
0
votes
0answers
17 views
linear table to matrix format [migrated]
I would like to convert a linear table to a matrix format.
My input table looks like this and is called "linear_table.tab":
transcript ortho
Transcript_1 ORTHO_1
Transcript_2 ORTHO_2
...
3
votes
2answers
22 views
fold and text columns
Can fold be set to recognize characters instead of bytes? Traditional Chinese characters appear to be encoded in three bytes each (in UTF-8 at least), which means that if fold's -w is not a multiple ...
1
vote
1answer
22 views
Extract specific section
I have multiple sections, each consisting of multiple lines.
e.g:
#--- Section 1 ---
Start
Lookup.mdc|123|xyz|
String
End
#--- Section 2 ---
Start
Lookup.mdc|123|xyz|
Integer
End
My ...
0
votes
3answers
30 views
Add brackets for latitude and longitude
I have a CSV file which has longitudes and the latitudes in it:
input.csv
103.771 1.44849,103.771 1.44894,103.771 1.4489,103.772 1.44802,103.772 1.4479,103.774 1.44948,103.775 1.4501,103.778 1.4512,...
2
votes
2answers
17 views
Read columns from file, then column into an exsisting csv file
Essentially, I have a csv file that contain multiple columns, called cols.csv
1,a,100
2,b,200
3,c,300
4,e,400
and I have a new csv file that has one column, called col.csv
f
g
h
i
I want to copy ...
5
votes
4answers
165 views
how to calculate average of values in a column by considering the information from another column?
file 1:
HOGBRM443983 -2522.00 19800826
HOGBRM445985 -2389.00 19801101
HOUSAM1891409 -1153.00 19811228
HOUSAM2004289 -650.00 19860101
HOUSAM2005991 -843.00 ...
1
vote
2answers
59 views
How to convert row to column
Please help me to convert the row values into column.
My input file.dat is:
first two fields are alpha string and numbers respectively and input will always have same number of fields i.e., 6.
...
1
vote
1answer
18 views
awk - grep multiple rows and print in a specific manner
I have the following content and a file called file.txt
TIMESTAMP 200 J8899
INSTANCES 19 x86_64 False i-xxx1a1a t2.medium True subnet-93b224fb hvm vpc-7b801
STATE 16 running
...
1
vote
2answers
35 views
How to insert the content of a file into another file between two matching lines?
I have an xml file process.xml and I want to insert the content of tmp.xml into it. But the caveat is, I need those contents to be inserted between two matching patterns. Here is a snippet of the ...
3
votes
1answer
40 views
What Type Of Data does the Input File Contain?
I have a folder with more than 10000 text files. The files can be of two types:
Type1: called “DNA”
Format:
header information starting with “>”
Line 2 onwards contains only the following letters:...
5
votes
7answers
298 views
Insert newline before each line matching a pattern unless the previous line is already empty
I need to add a new line before any line containing a pattern where we can assume that the pattern is always the first string of the current line. For example
This is a
pattern
This is a
pattern
I ...
1
vote
1answer
34 views
Check the count of records from the source file and the loaded target
I have some source file suppose mydata.csv and my target table.I want to validate the record count whether it's same between the source file and the target table .The target table is in hive.
I have ...
3
votes
5answers
89 views
Get the query string from URLs in a log file
I have a server log file that contains a number of log entries in the following format:
193.1.172.46 - - [23/Mar/2008:03:57:38 +0000] "GET /robots.txt HTTP/1.0" 404 289 "-" "gsa-crawler (Enterprise; ...
0
votes
2answers
37 views
Remove lines with duplicated less than 5
I have a data looks like this, for each SNP, it should repeat 5 times with different beta. But for SNP rs11704961, it only repeat twice, so I want to delete SNP rows that repeat less than 5 times. I ...
-1
votes
1answer
39 views
Is there go up line character? (Opposite of \n)
In awk is there a go up character? I found something similar here but it doesn't work.
For below:
97
ARE
3
NL
1303
1
140
USA
If record is not numeric need to print it as below
97,ARE
3,NL
1303
1
...
1
vote
2answers
49 views
Taking information from logs using grep, sed, awk and cut
I'm trying to use simple commands to extract data from a log file.
Here's a sample line from the file
128.101.35.183 - - [23/Mar/2008:00:03:10 +0000] "GET /javascript/email.js HTTP/1.1" 200 359 "...
2
votes
1answer
25 views
Record of highest NF
Using awk for the following input file, how could I take the record with highest NF for the same last field.
File:
5541,55004,919843,IND
5542,5541,55004,919843,IND
7903,790287,RUSE
7905,7903,...
1
vote
1answer
33 views
how can I count transactions based on datetime
my file format is below
123456 2016-07-05 08:08:00
123456 2016-07-05 10:23:45
125678 2016-07-05 09:23:54
127890 2016-07-06 10:23:00
127890 2016-07-06 13:00:00
how can I get just transactions that ...
1
vote
2answers
49 views
UNIX: Need to replace last line on the file and also remove extra blank line in the end
I need to replace - WOR to VWOR in end of the file.
I have tried
sed s/WOR/VWOR/g
But this does not work until I have new blank line in file.
So I used
( cat FileName && echo ) | ...
0
votes
2answers
48 views
Need to parse a csv file and it should store in variable
I need to parse a csv file and store the value in a variable. Below is the sample csv file
And below is the ./script.sh
#!/bin/bash
D="";
P="";
./xyz --project "$P" --displayname "$D"
For ...
0
votes
1answer
17 views
subset files in a folder based on a list
I have a folder "all_images/" with more than 1000 image files named as "Image1.tif", "Image2.tif" and so on..
I have a text file "extract_images_list.txt" which is a list of images that I want to ...
1
vote
2answers
34 views
How to convert uA/mA to uA in each lines of csv file using c shell?
I have a *.csv file as bellow. What I want to do is converting all units to one unit (uA). Anyone can help to do this in c shell??
1
vote
2answers
23 views
Whitespace delimited table to CSV with standard text processing tools
I have the following data table:
ssh 22/tcp 0.182286 # Secure Shell Login
linuxconf 98/tcp 0.000088
tacnews 98/udp 0.000560 # TAC News
from which I wish to generate a CSV ...
1
vote
2answers
77 views
sed or awk command to insert tabs (delimiter) at spcific positions
I am trying to parse a table with variable number of delimiters to convert into a table of equal columns
cluster=96\troot\tcellular organisms\tno_rank no_rank$
cluster=42\troot\tcellular organisms\...
3
votes
3answers
66 views
How to delete all occurrences of a list of words from a text file?
I have a file containing a list of words. I want to remove all occurrences of all the words in this file from a big text file.
Example:
File 1
queen
king
Text file sample
Both the king and queen ...
3
votes
4answers
62 views
How to check for null fields in CSV file?
My CSV file will look like this (pipe separated):
apple|banana|pear||grapefruit
lemon|lime|damson|jackfruit
|tangerine|nectarine|plum
apricot|orange|pineapple|coconut|
(Assume that any sort of ...
4
votes
4answers
161 views
Double quote value assignments stored in a CSV?
I have a file that contains text as follows:
dt=2016-06-30,path=path1,site=US,mobile=1
dt=2016-06-21,path=path2,site=UK,mobile=0
I want to convert it to text with double-quoted values in the key-...
1
vote
1answer
19 views
How to compare multiple columns of one file with one column of another and display matches from second file?
I have two files like this:
File 1
apple fruit 10 4
strawberry fruit 22 11
berry fruit 33 33
ball toy 1 12
straw thing 2 33
File 2
strawberry straw berry
rawberry raw berry
I want to compare ...
1
vote
1answer
36 views
How to extract only the header name in a data without listing the data itself
I have a data set that has 1127 columns, I only need to know the header without listing the data itself in each column.
For example, data as
name age
m 33
A 26
I need a code in UNIX that ...
1
vote
1answer
30 views
Compare two files line by line without comm (I need to maintain order of file 1)
File 1:
happy
sad
calm
palm
File 2:
palm
dream
calm
I want to compare the two files and display only those line that are common in both the files, but I want to maintain the order of File 2. My ...
0
votes
1answer
35 views
How to remove tcpdump first and last lines?
I'm coding a script based on tcpdump, but I want to get rid of the first lines that indicates the verbose parameters and the last lines with the number of captured packets:
I just need the output ...
1
vote
2answers
59 views
How do I split a large file into multiple smaller ones based on the value in a text column?
I have a large file that I need to split into workable pieces. (350 million records)
The key is the second column value cannot overflow into the next file.
Reading and writing take way to long, split ...
1
vote
1answer
47 views
Parsing .txt file to generate .csv
I have a text file with following contents
Torrent file : Linux.Format.-.October.2016.-.True.Pdf.-.Set.1001.[ECLiPSE].torrent
Metadata info : 9968 bytes, 412 pieces, 65536 bytes per piece
Torrent ...
-1
votes
2answers
47 views
text-editor which counts words not characters? [closed]
Is there a text-editor which counts words not characters in GNU/Linux ? I tried out various editors but couldn't figure out which one has it but none of the popular ones seem to have it. Looking for ...
-3
votes
3answers
44 views
How to display only directories from a file?
Hello i have a text file which contains data like below
/nrt/www/html/USA/anil.php
/nrt/www/html/USA/anil1.php
/nrt/www/html/USA/anil2.php
can any one tell me how to display only directories from ...
0
votes
1answer
45 views
Compute percentile and max value per variable
Bash Gurus, I need to compute the max and percentile numbers for each item in the list, using awk
aa 1
ab 3
aa 4
ac 5
aa 3
ad 2
ab 4
ac 2
ae 2
ac 5
Expected output
Item 90th percentile ...
0
votes
0answers
23 views
get a particular word in every line [closed]
line1: root 17001 1 0 Oct18 ? 00:00:33 node --optimize_for_size --gc_interval=100 --max-old-space-size=1024 stationManager/stationManager.js NODE_01 10.44.112.62 8097 0 0 10.44.112.62 ...
-3
votes
3answers
71 views
Continuously write terminal output to file?
Given that I have a script which prints out a random number between 100-999 every 10th second, how can I gather this information into, for example, a .txt file?
Is it also possible to only gather the ...
0
votes
3answers
55 views
How to modify file content? [duplicate]
Ubuntu 16.04
I'm pretty new to linux and this question may sound prety dumb.
I have some pretty large file so modifying it by hand is not possible. I'd like to change all occurences of
<tag1>...
3
votes
1answer
42 views
How can I print out the complete function declaration of any function which has a specific string inside it?
I have lots of functions in my bashrc, but for newly created ones I often forget the name of the function.
So for example, when I have defined this function in my .bashrc:
function gitignore-...
-1
votes
3answers
43 views
Extract values from key-value pair and output values alone separated by comma
Input file data
Name: aa
Age: 20
Address: zzzz
Name: bb
Age: 17
Address: cccc
I want to output like this
aa,20,zzzz
bb,17,cccc
0
votes
2answers
33 views
Convert comma delimited variable values into space delimited
I have a property file in which a particular key has comma delimited values.
$cat sample.prorerties
value=alex,raj,kaly,rema
In my shell script I implement this property file and use the key value ...
3
votes
4answers
74 views
paste and save the result to the 1st argument without using temp files
I would like to use paste to add new column in original file "first.xls".
My files:
first.xls
1
1
1
second.xls
2
2
2
I'm using:
paste -d'\t' first.xls second.xls >> first.xls
and I get
...
-1
votes
0answers
17 views
How to find a string and match with another file and return the value
I have two files:
gene-enzymes- and c_elegans_withoutexpression
gene-enzymes- contains info like this:
W07E11.1
gdh-1
(ddo-1 | ddo-2 | ddo-3)
And the other file contains info like this:
C47A10.5a ...
