All Questions
291
questions
3
votes
1answer
247 views
Reducing the amount of duplicated code (python) - cricket matches [closed]
I'm a beginner and self taught and just want to see if there are neater ways of doing what I am doing, or whether there are bits of code that are poorly written. I think Lines 100-204 are the most ...
4
votes
1answer
37 views
GUI that reads data and generates/ saves charts
I have a program that uses pandas to read csv files and then generates and saves graphical charts. I have been trying to follow the SOLID principles so I have tried to seperate responsibilities.
So ...
2
votes
1answer
43 views
Calculate students' GPAs and sort them, based on two CSV input files
Problem:
Given two csv files science_courses.csv and other_courses.csv, which has students name and their respective course ...
1
vote
2answers
51 views
Python script for parsing and using CSV data
I am given a CSV file of stations, with data like this:
...
3
votes
1answer
63 views
Mapping CSV columns
I have the following code where I am trying to map columns from the input file to output.
I have written it using multiple loops. Is there a way to write this more efficiently?
input.csv:
...
3
votes
1answer
59 views
Manipulating Pandas Dataframe with vaccination data from CSV to display on matplotlib
I have some code that manipulates a Pandas Dataframe containing Covid-19 vaccine data and displays it on Matplotlib.
The data is here: https://covid.ourworldindata.org/data/owid-covid-data.csv (...
1
vote
1answer
46 views
Processing CSV files with filtering
I have 4 CSV files that i need to process (filter out Part segment and then merge them) but the problem is that they do not fit in my memory. So I decided to: [open - filter - write out] each one of ...
5
votes
2answers
131 views
Mapping column names and values of a csv using another csv
I have two csv files, pricat.csv which contains objects I need to populate my DB with, and mapping.csv which specifies how the ...
2
votes
2answers
2k views
Python script to extract columns from a CSV file
I have a very basic python script that reads a CSV file. I'm importing the csv module setting some empty list variables and getting the data I want. The problem is that I know there is a much more ...
-2
votes
1answer
46 views
what is the best practice concerning opening CSV files in python [closed]
I am looking for the best way to save regular data from a series of sensors to a CSV file.
the program will be reading every sensor once a minute 24/7 and needs to report the results to a CSV file ...
3
votes
0answers
166 views
Reorder the Columns in a CSV File in Descending Order
I wrote a script to reorder the columns in a CSV file in descending order and then write to another CSV file. My script needs to be able to handle several tens of millions of records, and I would like ...
5
votes
1answer
182 views
Concatenate several CSV files in a single dataframe
I have currently 600 CSV files (and this number will grow) of 50K lines each i would like to put in one single dataframe.
I did this, it works well and it takes 3 minutes :
...
3
votes
1answer
72 views
Sending a CSV file for a client list from a database
What can I improve about this code in order to be production-ready? I'm not worried about security but about errors that could occur.
What exceptions should I catch? I feel overwhelmed about ...
3
votes
1answer
39 views
Count the number of riders in each category
I have a code to get riders from certain teams and calculate the number of riders in those teams in that category:
...
3
votes
1answer
80 views
Calculating the total daily amount of confirmed cases of Coronavirus
I'm writing a small program to plot new COVID-19 infections. As of right now, I have it so the program reads the given data file, pulls out the daily cases and dates for each country, and adds ...
1
vote
0answers
46 views
Converting multiple HTML files to csv lowers down the speed
New to python.
I am converting multiple html files to csv. Developed a full code. It can easily convert 50+ files at a time but after 200+ files, the code takes too much time and with 1000s to ...
2
votes
3answers
662 views
Efficiently convert 60 GB JSON file to a csv file
Description
Simply take a JSON file as input and convert the data in it into a CSV file. I won't describe the functionality in too much detail since I have reasonable docstrings for that. As you can ...
1
vote
1answer
525 views
I made a Python program to calculate price based on Inflation Rate
I made a program that calculates how much the value of money has decreased due to inflation over a given set of years.
This takes a principal amount, a start and end date, and with the help of the ...
2
votes
2answers
103 views
Receive data from SQL and CSV and send it to MySQL database using Python
Details:
I have a code, which does the following:
Get data from CSV and insert it into MySQL table
Get data from SQL Server and insert it into MySQL table
Run this code every 240 seconds - to ...
2
votes
1answer
85 views
Parse org-mode clocktable CSVs into time sheets
For my student job, I have been logging work times with the org-mode in emacs for quite some time. Now since I can only work from remote, I figured it would be nice ...
1
vote
3answers
75 views
Scraping the Bundesliga table and saving it to CSV
Wrote my first mini-script to get data from the german football league table and save the data to a csv file. Do you like the approach or could I be more efficient? Thanks in advance!
...
9
votes
3answers
217 views
Performance - Read large amount of XMLs and load into single csv
I am dealing with a large amount of XML files which I obtained from here https://clinicaltrials.gov/ct2/resources/download#DownloadAllData. The download yields around 300.000 XML files of similar ...
0
votes
2answers
70 views
Read .csv files and convert the rows into dict for looping
This script tries to parse .csv files and returns an iterable object to fetch one row at a time to do some process. It works fine but I think it needs improvements ...
2
votes
0answers
171 views
application to get git commit and write data into csv file
I am writing a small application to get commits between two tags from different env (apple, banana) and write the result into a csv file.
For security, I have to replace some sensitive information ...
3
votes
1answer
92 views
Get data from pipedrive API and insert it into Snowflake
I am a new python programmer.
I have created the code to put API data from pipedrive to Snowflake database.
Here are the steps in my code.
Delete the csv file if it exists.
Make an API call put all ...
9
votes
2answers
1k views
Are there ways to speed up this string fuzzy matching in Golang?
I have a piece of python code doing fuzzy matching which works very well and is pretty fast. For reference, it uses the following files:
https://raw.githubusercontent.com/datasets/s-and-p-500-...
1
vote
1answer
83 views
File size when parsing XML
The code below takes a directory of XML files and parses them into a CSV file.
Right now parsing around 60 XML files is fast and the output is a CSV file that is around 250MB.
That is a really big ...
4
votes
1answer
1k views
Reading multiple csv files in a single dataframe
I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now:
...
1
vote
0answers
47 views
Optimized AWS Lamda purpose [closed]
I'm looking for optimized code with faster execution time. This writes CSV to parquet with the max size of 64 MB chunks.
we are trying to concatenate the csv file contents being written as an parquet ...
8
votes
2answers
710 views
Analyzing records of London Underground outages
You have a file that describes historic data of the London Underground showing dates when stations have been unavailable. The file contains these fields, separated by a pipe character:
Date: ...
30
votes
6answers
4k views
Python web-scraper to download table of transistor counts from Wikipedia
I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it?
...
2
votes
1answer
832 views
Multithreading to process requests and save results in python
I was presented with a task to come up with a script that generates a CSV with POSTAL codes via bruteforce (I'm in Brazil, so that means CEP to us).
Points to note:
I'm using an external library, ...
2
votes
1answer
69 views
Remapping property IDs when inserting CSV data to a database table
My code is functioning the way I'm expecting, with the if. However, it is not pythonic to me, and that bugs me.
The multiple if in the code below are supposed to ...
3
votes
1answer
1k views
Exporting data from PostgreSQL as CSV to S3 bucket
I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
3
votes
0answers
524 views
JSON to CSV in python using json.loads and json_normalize
I am trying to convert a JSON file to CSV format using Python. I am using the JSON.loads() method and then using json_normalize()...
2
votes
1answer
41 views
Batch retrieve formatted address along with geometry (lat/long) and output to csv
I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City.
My goal was to output multiple csv ...
2
votes
0answers
93 views
Pricing data checker script
I was recently given a task to implement a data-checker script which given an input file containing (date, last_price) values should check for 3 different kind of errors - missing values, stale values ...
3
votes
2answers
88 views
Excel To CSV Converter Ch 9 ATBS
I am learning more about python using the book "Automate The Boring Stuff". The problem I completed was "Excel To CSV Converter". I want to kick bad habits to the curb, and make sure I'm learning the ...
4
votes
0answers
42 views
Combining CSV files of simulation results
This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
2
votes
1answer
34 views
Grouping sales transactions by person
I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
2
votes
0answers
27 views
Getting service rate based on time/day of service
I have a long roster where I'd like to categorize the rates by time/day of service. Most shifts ends the same date but there are some special cases where the shift ends on a different date. I have to ...
3
votes
3answers
172 views
Computing average duration per user from a large CSV file that doesn't fit in memory
We have a large log file which stores user interactions with an application. The entries in the log file follow the following schema: {userId, timestamp, actionType} where actionType is one of two ...
4
votes
1answer
49 views
Count words in a list of titles, with some cleanup
I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...
4
votes
0answers
108 views
Convert Amazon transaction data for import into accounting system
At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
10
votes
2answers
6k views
Read/write a pipe-delimited file line by line with some simple text manipulation
This code that I wrote is supposed to read/write a pipe-delimited file line by line to a new file with some simple text manipulation. (It also adds two new columns) and publishes a "Status Update" ...
7
votes
1answer
1k views
Python code to identify structure of a text file
We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
3
votes
1answer
75 views
Checking whether fund prices changed over multiple CSV files
A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes.
1st download - 2 Fund ...
17
votes
4answers
1k views
Sensor logger for Raspberry Pi in a stratospheric probe
I'm writing a Python Script for a Raspberry Pi to measure different sensors. We are planning to send the Pi with that Script running to the stratosphere, so the power usage for the Pi is limited.
I ...
3
votes
0answers
84 views
Generate CSV from multiple files on Regex Python Output
I have written a python script using regex to extrapolate a poorly formatted text file data table into an actual data table. I am currently printing the output using . C:\my file location\file.csv in ...
3
votes
1answer
129 views
CSV/JSON converter
This is my first Python script and I would like to hear the opinion of more experienced users, so I can do it right from the beginning instead of trying to correct my mistakes after months (years?) of ...
