Manipulation of text by programs, scripts, etc.
1
vote
2answers
85 views
How to loop a script over many files?
I have Gnuplot script (exe), inside this script I supply X_001 as an input file to be plotted by a certain function giving output file with X_001.png. It works well if I manually put the name of the ...
5
votes
2answers
158 views
How to get the text between two words specified by their indices?
Using awk, I can print the words of the given indices as following.
$ echo "The quick brown fox jumps over the lazy dog" | awk '{print $3, $7}'
brown the
But I also want to get the text between the ...
3
votes
4answers
61 views
Way to remove newline (^M) from variables only, not from file
I am loading a file into variables, the problem is that the file is formatted in Windows, I believe, so I get a ^M instead of a newline.
How do I modify it when the value is in the variable? I am ...
2
votes
2answers
28 views
Replace values in certain section in a file
I have a file which have several sections, I have to change the values of all parameters in section [Test2].
Input:
[Test_1]
abc=123
def=456
.
.
.
so on
[Test_2]
abc=123
def=456
.
.
.
so on
[...
0
votes
2answers
27 views
how to arrange your data in a way that missing values within a file don't affect the data? [on hold]
input file
BMN8400;12345;CMNP002;B123;GM
BMGN400;;CMBD600;;G2
BMD200;80088;;B808;G8
output file
BMN8400;12345;CMNP002;B123;GM
BMGN400; * ;CMBD600; * ;G2
BMD200;80088; * ;B808;G8
1
vote
1answer
20 views
Set theory with unix tools: “relaxed” intersection of two lists where item in set 1 is at least a subtring of items in set 2
I have a list of mac addresses of machines by means of a dhcp log. I need to filter on a set of mac address prefixes (the substring) which belong to a vendor.
How can I can filter the list of machine ...
0
votes
2answers
42 views
count number of occurrence of word between 2 fixed words
I have file as below
FHEAD
THEAD
TCUST
TITEM
TTEND
TTAIL
THEAD
TCUST
TCUST
TITEM
TITEM
TTEND
TTAIL
THEAD
TCUST
TITEM
TTEND
TTAIL
THEAD
TCUST
TCUST
TITEM
TTEND
TTAIL
I need to count number of ...
7
votes
4answers
754 views
How to print lines number 15 and 25 out of each 50 lines?
I have a large file and I would like print from each sequential 50 lines , the 15th and 25th lines.
sed -n '15,25p' inputfile
How to modify this command to print only lines 15 and 25 and to loop ...
1
vote
1answer
24 views
How to find the min of a column in every nth intervals of a file, using sed, sort, tail?
I want to find the minimum of the 5th column of a file in every 12th interval of that and save the associated line into a new file.
To find the minimum of the last 12th line I can use
tail -n 12 $...
-2
votes
0answers
35 views
Problem converting windows formatted text file to unix formatted text file - dos2unix [on hold]
I used a program hashmyfiles to generate hashes of all my files on windows 7 and windows 10.
I can view the files in windows fine.
I want to edit them on ubuntu.
I ran dos2unix on ubuntu as follows:...
1
vote
1answer
21 views
Sort the directories and files in each directory in ascending order and export data to csv
I have 30 directories i have more than 300 text files in each, all text files in each directory have same naming format
regional_vol_GM1.txt
regional_vol_GM2.txt
regional_vol_GM*.txt
I would like ...
1
vote
5answers
41 views
List all unique strings in a specific column
I don't know what I have in my huge vcf.gz file that looks like this
CHROM POS ALT 12345
1 345632 T 0/1:4,4:8:99:105,0,106
4 032184 C ...
3
votes
3answers
34 views
Delete lines that contain the same information but in different order
Let's say I have two lines( in a txt file)
Monday, Tuesday, Week
Tuesday, Monday, Week
They contain the same information, but the information(the first and second column) is in a different order in ...
0
votes
1answer
36 views
Break a string of host names into columns
I have the output of a command that can list between 8 and 800 hosts.
I am trying to figure out how to break this into 3 or 4 columns.
Right now I am using "fmt | column" but the output is not as ...
0
votes
1answer
31 views
Dealing with duplicate hash data in text file - keeping one unique line and deleting duplicates
Thanks to this forum I was able to process a large text file containing filenames and hashes and end up with a list of files with identical hashes as shown below:
file_35.txt ...
0
votes
2answers
20 views
Text substitution with specific fields in a log file.
I have this:
00:05:40.005 id=32214483 Src=PIPE <[email protected]> (received) [email protected] relayed (1234 bytes)
I need to achieve this:
00:05:40.005 id=32214483 [email protected] <...
-4
votes
0answers
20 views
Fixed width to delimited file for more than 1000 columns
I have a fixed width file with more than 1000 columns. It needs to be converted to a delimited file along with column header with the delimiter as |(pipe symbol).
Control file of excel type(.xlsx) ...
0
votes
1answer
25 views
How to split my log file in multiple log files according to column values?
My connections log file is structured as follows:
hostname direction timestamp bps
Here's a fragment of my log file:
www.youtube.com DOWNLOAD 1479897661131903 23508910
www.youtube.com UPLOAD ...
-1
votes
3answers
41 views
How to look for a pattern in a file and print something if it finds a pattern?
Example -
test.log:
ConductID: null; RunOpSequenceCode: 12345678; ClientID: null; FescoOfferID: null; PriorityCode: null; RowNumberStart: 0; FetchSize: 0; ChannelTypes: ER,PART; NigoIndicator: null;...
0
votes
0answers
16 views
Transmitting a content of a text file from one server to another? [on hold]
What's the best and easiest way to transmit the content of a text of file or a file itself of the size around 100Kb from one server to another programmatically? That is, using a programming language ...
6
votes
3answers
804 views
How to get the number of bytes in just one line of a file?
I am wondering how I can get the number of bytes in just one line of a file.
I know I can use wc -l to get the number of lines in a file, and wc -c to get the total number of bytes in a file. What I ...
3
votes
2answers
61 views
Grep a paragraph that matches words from a file
I have a file (myfile) organized in paragraphs, i.e. with empty lines separating the entries. I want to retrieve some of these paragraphs according to match.
Now, it's all fun and games when the ...
1
vote
1answer
38 views
Difference between echo and print in the Korn shell
In Ksh, echo hello world and print hello world will return the same output (hello world). What is the difference between these text processing commands with respect to the Korn Shell?
I know how they ...
-1
votes
1answer
44 views
compare two csv files and fetch matching data [duplicate]
I have two .csv files namely file1.csv and file2.csv
file1.csv
ADIS
BAP3
Mercury_System
nxh-2003
DR_FeatureUP_PT
file2.csv
ADIS,projects.adis
EcoSystems,projects.ecosystems
em1xxxsw,projects....
4
votes
2answers
171 views
Replace “/U+[0-9A-Fa-f]{4}/” with proper unicode character in shell pipeline with sed eval flag
I am trying to properly visualize the existing characters that listed in the /usr/include/X11/keysymdef.h file.
It has lines like:
#define XK_onethird 0x0ab0 /* U+2153 VULGAR FRACTION ONE THIRD *...
4
votes
1answer
44 views
Text file containing filenames and hashes - extracting lines with duplicate hashes
I have generated a large text file containing filenames and sha-256 hashes using the format below - new line at the end of each line after the hashes.
file_1.txt ...
-1
votes
2answers
55 views
Extract text from a file based on some criteria
I have a file which contains records like those:
434419\Teclu\Tudor\1501\9502
187650\Cosma\Sorin\1504\9253
239474\Teclu\Daniel\1502\5245
844936\Gaman\Mihai\1505\4074
942341\Avram\Tudor\1505\4543
...
1
vote
2answers
20 views
return lines containing numbers in a range in a certain column
Task: return lines containing numbers in range from x to y in column z.
Example, selecting lines with numbers in range 140-160 in column 2.
1 1 some data 150
2 150 some data 1
3 151 some data ...
0
votes
2answers
31 views
How to evaluate special characters to see them in the text editor?
Let's say I have a text "line1\n\nline2\nline3" and I want to see it in a console or pipe it to a text editor so it looks like
line1
line2
line3
So special characters got evaluated.
Can I do it ...
-1
votes
2answers
60 views
compare two csv files and fetch matching data into a new csv file
I have two csv files with similar content/data.
file1.csv
ADIS
BAP3
Mercury_System
nxh-2003
DR_FeatureUP_PT
file2.csv
ADIS,projects.adis
EcoSystems,projects.ecosystems
em1xxxsw,projects.em1xxxsw
...
2
votes
2answers
30 views
Remove Nth line from both files if line in one file matches
I have two files, file1 and file2. They both have X lines. Some lines in file2 are empty(they match ^$). If line I is empty in file2, I want to remove line I from both files. Example:
File1:
ad
dada
...
-1
votes
2answers
32 views
How to use awk and print 3rd column that contains both 'db2' and 'SAP' strings?
cat ost_lvdd01_inbound_DD990_East.csv | awk -F',' ~ /db2/&/sap/ ; print $3
Input
/ost_lvdd01_inbound/1472930000/WPW-DBS003795.amer.warnerbros.com_1472938570_C1_IM:1472938570:/PROD_SQL_TXN_LOG:4:...
2
votes
1answer
28 views
Join two files each with two columns that have multiple columns are matched
I have two files
File A
MUREX_2089975 :SBE Zinc Non-Option 118510 metals USD -308410
MUREX_2097300 :SBE Zinc Non-Option 118510 metals USD 7751
MUREX_2097474 :SBE Zinc Non-Option ...
1
vote
1answer
36 views
Splitting files in Unix using values in a file
I have a txt file which has multiple XML records. I have to separate it using vendorname. I modified the commands as per my requirement. However, it's giving me 2 error messages:
awk: sf.awk:1: /<...
1
vote
1answer
33 views
Count number of values within a range in a specific column
I have to count the number of values that are between 0 and 0.05 in column 11 of a dataset. How do I go about doing this?
-4
votes
3answers
40 views
number lines in a file [closed]
So I have a file text.txt with lines of some characters
example:
abc
abc
yxz
abc
now I would like to number the lines like this and copy it to another file.
End expected result:
1 abc
2 abc ...
0
votes
0answers
17 views
linear table to matrix format [migrated]
I would like to convert a linear table to a matrix format.
My input table looks like this and is called "linear_table.tab":
transcript ortho
Transcript_1 ORTHO_1
Transcript_2 ORTHO_2
...
3
votes
2answers
23 views
fold and text columns
Can fold be set to recognize characters instead of bytes? Traditional Chinese characters appear to be encoded in three bytes each (in UTF-8 at least), which means that if fold's -w is not a multiple ...
1
vote
1answer
26 views
Extract specific section
I have multiple sections, each consisting of multiple lines.
e.g:
#--- Section 1 ---
Start
Lookup.mdc|123|xyz|
String
End
#--- Section 2 ---
Start
Lookup.mdc|123|xyz|
Integer
End
My ...
0
votes
3answers
31 views
Add brackets for latitude and longitude
I have a CSV file which has longitudes and the latitudes in it:
input.csv
103.771 1.44849,103.771 1.44894,103.771 1.4489,103.772 1.44802,103.772 1.4479,103.774 1.44948,103.775 1.4501,103.778 1.4512,...
2
votes
2answers
20 views
Read columns from file, then column into an exsisting csv file
Essentially, I have a csv file that contain multiple columns, called cols.csv
1,a,100
2,b,200
3,c,300
4,e,400
and I have a new csv file that has one column, called col.csv
f
g
h
i
I want to copy ...
5
votes
4answers
189 views
how to calculate average of values in a column by considering the information from another column?
file 1:
HOGBRM443983 -2522.00 19800826
HOGBRM445985 -2389.00 19801101
HOUSAM1891409 -1153.00 19811228
HOUSAM2004289 -650.00 19860101
HOUSAM2005991 -843.00 ...
1
vote
3answers
93 views
How to convert row to column
Please help me to convert the row values into column.
My input file.dat is:
First two fields are alpha string and numbers respectively and input will always have same number of fields i.e., 6.
...
1
vote
1answer
23 views
awk - grep multiple rows and print in a specific manner
I have the following content and a file called file.txt
TIMESTAMP 200 J8899
INSTANCES 19 x86_64 False i-xxx1a1a t2.medium True subnet-93b224fb hvm vpc-7b801
STATE 16 running
...
1
vote
2answers
37 views
How to insert the content of a file into another file between two matching lines?
I have an xml file process.xml and I want to insert the content of tmp.xml into it. But the caveat is, I need those contents to be inserted between two matching patterns. Here is a snippet of the ...
3
votes
1answer
43 views
What Type Of Data does the Input File Contain?
I have a folder with more than 10000 text files. The files can be of two types:
Type1: called “DNA”
Format:
header information starting with “>”
Line 2 onwards contains only the following letters:...
5
votes
7answers
312 views
Insert newline before each line matching a pattern unless the previous line is already empty
I need to add a new line before any line containing a pattern where we can assume that the pattern is always the first string of the current line. For example
This is a
pattern
This is a
pattern
I ...
1
vote
1answer
37 views
Check the count of records from the source file and the loaded target
I have some source file suppose mydata.csv and my target table.I want to validate the record count whether it's same between the source file and the target table .The target table is in hive.
I have ...
3
votes
5answers
91 views
Get the query string from URLs in a log file
I have a server log file that contains a number of log entries in the following format:
193.1.172.46 - - [23/Mar/2008:03:57:38 +0000] "GET /robots.txt HTTP/1.0" 404 289 "-" "gsa-crawler (Enterprise; ...
0
votes
2answers
38 views
Remove lines with duplicated less than 5
I have a data looks like this, for each SNP, it should repeat 5 times with different beta. But for SNP rs11704961, it only repeat twice, so I want to delete SNP rows that repeat less than 5 times. I ...
