All Questions
Tagged with csv shell-script
151
questions
1vote
1answer
68views
Filter one very large CSV based on values from another CSV
I am processing some CSV files that do not fit in RAM.
The 2 CSV files have the following structure:
first.csv
id
name
timestamp
serial
str
yyyy-mm-dd hh:mm:ss
second.csv
id
name
date
serial
str
...
2votes
2answers
77views
Comparing with csv file value
I have a csv file and I need to compare the value in 4th column of 2nd row with a string.
Sample csv file:
Col1,Col2,Col3,Col4
Val1,Val2,Val3,ValNeeded
I'm using the below code to perform this,
...
- 23
2votes
3answers
467views
Looping through columns
I have a csv file that looks something like this:
first_name,surname,email,
first_name2,surname2,email2,
....3,..
....4,...
The number of rows in each column are the same.
I want to run a command ...
- 21
0votes
2answers
260views
How do I loop through all csv files in a directory, select a range of columns and combine to single csv?
I have a large number of CSV files in a specific directory. All have at least 41 columns with matching headers, but can be as wide as 200 columns. I need to grab just the first 40 columns and append ...
1vote
3answers
369views
Generate templates in yaml from a CSV file
I am trying to create yaml files from a template using my variables.
My yaml template look like this
number: {{NUMBER}}
name: {{NAME}}
region: {{REGION}}
storenum: {{STORENUM}}
clients: {{...
- 11
0votes
2answers
116views
Organizing a CSV file with missing columns
Problem
I have a .csv file that was not filled in properly, which has missing column information.
The file looks something like this (whitespace is for clarity):
C1, C2, C3, C4,
R1C1,R1C2,R1C3,...
- 19
0votes
3answers
100views
Performing CSV Split effectively keeping limit into consideration
I have to split a large CSV file into multiple small files with a limit of 1000 records in each small file. However, I don't know beforehand how many records will be there in that large CSV file. I ...
- 1
0votes
2answers
316views
How to convert xlsx file to csv using shell script? [closed]
I have tried with libreoffice, ssconvert, xlsx2csv to convert the file but I am getting "command not found". Could you please help on that?
xlsx2csv
ssconvert file.xlsx file.csv
libreoffice -...
- 1
1vote
1answer
232views
Write values with embedded commas from arrays into CSV file using BASH
I am trying to write different arrays into a csv file ($file) columns.
Each array has the same number of values.
For example, arr1 and arr2 have 3 values each
arr1=( 23 45 34 )
arr2=( "abc,d"...
- 11
0votes
1answer
146views
Reading in a CSV file using while IFS= but not treating the comma as the separator if the field is in quotations
I have a large CSV file where columns are separated by commas in each case. However, about 5% of the time the first column contains a comma within itself and is denoted this way by the fact that these ...
- 41
1vote
2answers
28views
Converting files in multiple directories from tab separated to comma separated
I am running Mac OS.
I have a directory /Users/sethparker/Documents containing several subdirectories /Users/sethparker/Documents/dir1,/Users/sethparker/Documents/dir2,/Users/sethparker/Documents/dir3....
- 75
1vote
3answers
71views
Concatenate N number of csv files to one single csv file
I have a requirement to build a shell script which would concatenate all available CSV files into one. All these CSV files have a similar structure (i.e same headers and number of columns) and these ...
3votes
1answer
208views
Bash Script : Problem with variables from csv file
I'm having a little problem getting variables from a csv file on my Linux machine and using them in an if. I have the following csv:
Name;Age
Marc;18
Joseph;10
I'm trying to get this information from ...
2votes
1answer
94views
awk to return percentage by day of the week [closed]
So I have a dataset of flights in a CSV file and I want to get a percentage of flights delayed (column 6) ordered by day. 0 is for not delayed and 1 is for delayed 15 or more minutes.
Input:
...
- 145
-1votes
2answers
45views
comparison to get only lines with changed values on a column
Given the following two csv files, previously sorted:
file1
a,1,val1
b,2,val2
c,3,val3
d,4,val4
e,5,val5
f,6,val6
g,7,val7
file2
a,1,val1
b,2,val2
c,3,val3x
e,5,val5x
g,7,val7
h,8,val8
How can I ...
-1votes
2answers
79views
Compare two csv files containing floating-point data and print their difference statistics
I have two csv files: a.csv and b.csv. Both these files contain same set of rows. For example:
a.csv:
Yield
5.68
1.34
1.47
9.01
7.59
b.csv:
Yield
5.68
2.01
0.78
7.98
8.17
I need to compare these two ...
- 169
6votes
1answer
557views
How can SQLite command ".import" read directly from standard input?
How would you modify function csv_to_sqlite so that sqlite3 command .import reads directly from standard input instead of from a temporary named pipe?
#!/bin/bash
function csv_to_sqlite() {
local ...
- 385
0votes
1answer
154views
Combine multiple CSV files with Same Header into multiple group files
Hi I am looking for a fastest solution to process loads of csv files.
Situation:
I have multiple csv files in a single folder with different headers
I have already done pre-processing on them to ...
- 101
-1votes
2answers
56views
Copy column B if column E or F is empty or comes with value 0
I have a CSV file, and I want to use a bash with command awk or anything else to modify this file.
For example, I want to copy column B if column E or F comes empty or with the value of 0
AAAA, BBBB, ...
- 3
1vote
1answer
79views
Replace several characters in a text file using specific row and column values of a csv
I have a .txt file with different entries such as region1a, region1b, region2a, region2b etc.
I have an excel file with different column and row values. I want to use a particular row and column value ...
- 13
-1votes
3answers
115views
Issues related to grep command looping for multiple csv files
I made a for loop script as shown below,
for file in *.csv
do
grep raxA $file > new_${file}
done &&
mkdir raxA && mv new_* raxA &&
for file in *.csv
do
grep raxB $file > ...
- 129
1vote
1answer
125views
In Shell Script SSH connection is working only for first line data in CSV file and command failing for remaining servers
My target is to just validate the login is successful for list of servers in CSV file. For SSH Password, Userid and IP address, I am passing these values via CSV.
And want to capture the result of SSH ...
- 11
0votes
2answers
98views
Remove all rows in a CSV file without corresponding/matching timestamp in another CSV file
I have 2 CSV files which look like follows:
file1.csv
col1,col2,col3,col4,date,time,col7
1,2,3,4,2019-07-20,12:40:00,0
1,2,3,4,2019-07-20,12:43:00,0
1,2,3,4,2019-07-20,12:44:00,0
1,2,3,4,2019-07-20,12:...
- 111
-1votes
1answer
144views
grep and re-direct to csv
This is the campaign log that I am parsing. Based on call-id I do grep.
Orginal Progress-log as follows:
FilePlayer::run:: Finished audio file is /voip/VoiceCampaigSetup/wildfly-10.1.0.Final/...
1vote
1answer
187views
How can i put a variable in an awk command
with my code im trying to sum up the values with the specific name of a column in a csv file, depending on the input of the name.
Here's my code:
#!/bin/bash
updatedata() {
index=0
while IFS=...
- 11
1vote
1answer
37views
Get a list of file name, and compare to a csv file while adding 2nd and 3rd column of the same 1st column
I have a csv file that looks like this:
user1,400,300
user2,250,250
user3,400,300
user1,400,300
user4,400,300
user2,250,250
and a directory that has a empty file named after the user
$ ls /dir/*
/dir/...
0votes
5answers
1kviews
how to check validation of csv file with two fields
I have simple csv file that should contain only two non-empty fields as the following
This is example of right csv file
$ more file.csv
why_we_need_help,log_low=53687091200
whats_is_going_on,...
- 10.6k
0votes
1answer
375views
How to convert all text files in directory to one csv
I want to convert all text files in a directory to one csv file. I want the input to the csv to be the text in the text file labeled by the author of the text file found in the filename. For reference,...
- 35
0votes
3answers
79views
not able to find a file name in the directory using file name as a variable from a temporary file
I have sample CSV file contains file names
f1
f2
f3
I want to search these file name in a directory, below is my code
while read f1
do
ls -t | grep "$f1"
if[ $? = 0 ]
then
echo file ...
0votes
2answers
1kviews
Remove URI prefix (http:// and https://) by using awk command or shell script
I have the below data (Actual output)
http://localhost:5058/uaa/token,80
https://t-mobile.com,443
http://USERSECURITYTOKEN/payments/security/jwttoken,80
https://core.op.api.internal.t-mobile.com/v1/...
1vote
1answer
34views
Compare name of files to a column in a table and if a correspondence is found rename it according to another column
I have an extensive set of files in a directory (in bash) that are named (inconveniently) as follows:
COMMONSTUFF_X_Y_**VARIABLE-IDENTIFIER**_COMMONSTUFF
I also have a table in which one column ...
- 11
0votes
3answers
468views
Is there a way to accurately parse a true CSV file? [duplicate]
I'm using bash shell. I want to parse a CSV file, in which the CSV file observes true CSV formats. From this thread -- https://stackoverflow.com/questions/4286469/how-to-parse-a-csv-file-in-bash , I ...
- 1,760
0votes
1answer
1kviews
How to write data to a csv file with comma in string values
I need to hit a database with basic select queries and write the result output in a file. using sqlcmd I execute the queries and write the results to the output csv file.
While trying to open this ...
- 3
3votes
3answers
175views
How to parse Stdout as a mix of CSV and JSON?
I'm currently working on a class that has us submit our code to an auto-grader and then returns our results. The format it returns is kind of hard to visually parse, so I would like to write a script ...
- 181
0votes
6answers
137views
Convert line-wise block of program output to CSV while removing line headers
Need your help, my command output is :
Server1:
CPU(S): 8
RAM: 24
Socket: 2
Server2:
CPU(S): 16
RAM: 32
Socket: 5
Desired Output:
Server1:,8,24,2
Server2:,16,32,5
- 37
0votes
4answers
620views
Shell : How to replace character only inside double quoted text in a file?
I have following line in text file
abc|45|"Do not replace | in this"|0.23
I want a way to replace | character only in text in double quotes, resulting
abc|45|"Do not replace in this"|0.23
I have ...
- 115
0votes
1answer
490views
Read csv from shell script and print the column header based on closest match to user input
I Have a csv file. It has multiple columns. I want to develop a shell script which prompts the use for a numerical value, find the closest match in the csv and print the header of the column. For ...
- 369
-1votes
1answer
267views
Check if a two columns are completely numeric in a csv using awk
I have a csv with columns A & B as below which has numeric or alphanumeric data.I need to write column C such that if,
A or B is blank -> write "Error" in C
A or B not numeric -> write "Error"...
1vote
1answer
852views
Export command output specific lines and put it into CSV file
Could you assist me how to retrieve specific lines and put it into CSV file?
How to process the output via Bash and Python3?
Remark:
"objectID" represents each found server
"An instance of Path" ...
- 43
0votes
1answer
173views
How to execute multiple commands from an input file and string it to a new output file
I want to execute multiple commands (one after the other) from an input.txt file and output it to a new .csv file.
All the commands run successfully when executed individually, but nothing happens ...
- 11
0votes
2answers
296views
Merge CSV Files based on ID even when Unmatched IDS and Indefinite Number of Columns
I'm writing a bash script and part of the process requires to piece csv files together whilst preserving the integrity of the columns; for example I have two files in the following format:
F1
ID,MD,...
- 67
0votes
2answers
46views
converting variables in a shell script from strings to numbers
I have a file laid out like this
gender,yearsExperience,yearsSchool,wage
female,9,13,6.3152956461
female,12,12,5.4797699786
female,11,11,3.6421699174
female,9,14,4.5933365997
female,8,14,2.4181574607
...
- 75
0votes
3answers
464views
Compare 2 comma separated files and output the difference
I would like to compare 2 similar files on a common column and output the difference. The files will have identical headers. And i am looking to see the difference of particular column.
Second file ...
- 13
-2votes
2answers
67views
How can I manipulate contents of a text file into a new output file
I have an input text file that looks like this:
table columns are:
number of vertices
total surface area (mm^2)
total gray matter volume (mm^3)
average cortical thickness +- standard deviation (mm)
...
- 11
1vote
1answer
2kviews
create a folder from dynamic path location
I would like to create a folder which location is available in abc.csv file. abc.scv content the following fields id,path and file name we should read the file name and move the file in the path ...
3votes
2answers
338views
Parsing csv file to filter rows basing on a matching set of characters from a column value
Consider the following csv file:
A,3300
B,8440
B,8443
B,8444
C,304
C,404
M,5502
M,5511
The real csv file is really big (around 60,000 rows) but I only included a small version for ...
- 31
0votes
1answer
2kviews
Sybase query: save output to a file
I´ve created the following script:
#!/bin/bash
isql -U databasename_dba -P password -b <<EOF!
select quantity, date from name_table where numer_id="1234"
go
quit
EOF!
...
- 109
2votes
7answers
7kviews
How can I search a CSV file for a value, then get another value associated with it?
myfile.csv
445,aLabel
446,anotherLabel
447,aThirdLabel
In a bash script, I want to search for the presence of "anotherLabel"
(in a case-insensitive way) within myfile.csv and get the value ...
- 67
0votes
1answer
42views
Get data form CSV to move a file
I have a CSV file with two columns and this information:
Column1 | Column2
USA | Chicago
USA | Dallas
FR | France
AUS | Sydney
AUS | Perth
I have to move all the files where ...
- 1
1vote
2answers
349views
Script to combine separate csv files
I have two separate files containing information on clients that I would like to merge into one. The first file contains global information, the second is for options.
Basically, every client has an ...
- 159