All Questions
96
questions
0
votes
0
answers
57
views
Looking for best practices/design patterns when working with SqlAlchemy ORM
I am new to python and sqlalchemy and looking for help to cleanup below code or follow a popular design pattern when working with database.
Below sample code is for the use case "User clicks on ...
- 183
0
votes
0
answers
32
views
Synchronized insertion to InlfuxDb using Python
I am connecting to various topics of the exchange websocket to fetch the data into the callback function, where it will insert to the InfluxDb, however, I am afriad the way I do below has low ...
- 77
2
votes
1
answer
274
views
How to structure in handling websocket and inserting data to databse in Python
I connected to the websocket, and the data response in the callback function handle_message. I wish to insert all the data to influxdb using ...
- 77
0
votes
1
answer
118
views
Efficient way to read files python - 10 folders with 100k txt files in each one
i am looking for an efficient way to read and append texts of .txt files to a dataframe. I currently have 10 folders with 100k documents each.
What i specifically need to do is:
getting the names of ...
8
votes
2
answers
1k
views
High traffic website that shows users subscription status
I was in two minds as whether to post this question on stackoverflow or dba stackexchange but because I am asking for review, I thought of posting here.
I am new to Python and looking for feedback on ...
- 183
3
votes
1
answer
173
views
Summing categories of financial records per month in a query
My program is working properly but I'm unconfortable with code repetition.
...
- 183
3
votes
1
answer
341
views
Why are my read speeds so bad using sqlite?
I know very little about databases and even less about how to optimize them, but I have a problem which calls for a database so here I am...
I created a sqlite3 database using the following script:
<...
- 906
1
vote
0
answers
45
views
UPDATE on Newspaper Bill Calculator CLI with Python (3 of 3, Database)
Code is posted after explanation.
Due to the size of the project, this is being posted in three separate posts. This also ensures each post is more focused.
Post 1 of 3, Core: UPDATE 1 on Newspaper ...
- 358
3
votes
1
answer
75
views
Newspaper Bill Calculator CLI with Python (3 of 3, Database)
Code is posted after explanation.
Due to the size of the project, this is being posted in three separate posts. This also ensures each post is more focused.
Post 1 of 3, Core: Newspaper Bill ...
- 358
3
votes
1
answer
79
views
Acupuncture database builder
The following code builds a rudimentary acupuncture database by collecting data from the web.
I would like to hear suggestions about improvements to the database structure, code organization, web-...
- 417
1
vote
0
answers
28
views
Django data migration - worked, but far from performant
Background: I am learning web dev and decided on the Django framework as an excuse to learn Python at the same time. (Background is stats with lots of R and ...
- 11
4
votes
1
answer
93
views
Implementing a data extraction pipeline with Bash
I wrote the following scripts to extract data from BigQuery and InfluxDB using the bigquery cli and the influx cli. The code works as expected. I included all files for completeness, but I'm primarily ...
- 319
2
votes
1
answer
234
views
Get data from binance api and save to ClickHouse DB
I did the following
Created DB and table in prepate_table function
Extracted data from binance in GetHistoricalData function
Saved the data to DB
My code works fine but I want to optimize my code ...
- 123
7
votes
2
answers
496
views
A Hit Counter For AWS Python Lambda powered with DynamoDB
I'm learning how to use DynamoDB and various AWS services, so I decided to write a simple hit counter application for my education of how to use DynamoDB. This application deduplicates hits from the ...
- 875
3
votes
1
answer
268
views
Reducing the amount of duplicated code (python) - cricket matches [closed]
I'm a beginner and self taught and just want to see if there are neater ways of doing what I am doing, or whether there are bits of code that are poorly written. I think Lines 100-204 are the most ...
1
vote
1
answer
79
views
Data storage and processing in Python
I have a piece of code that downloads acupuncture data from Wikipedia and consolidates it into an acupoint, meridian and extraordinary meridian dictionary stored as .pkl files in the working directory....
- 417
3
votes
1
answer
900
views
Data API with Influx-DB and FastAPI
I'm fairly new to time series databases in general and Influx in particular.
My objective is to build a simple and general-use API that will allow me to write and read data from an Influx Database.
I ...
- 31
2
votes
0
answers
75
views
Compute damages and savings for motor insurance claims
I am working on some claims data, where I have features such as
The descriptions of the columns are: (A) the year of claim, (B) who's the insuring company (C) whether the vehicle is driven, (D) ...
- 21
1
vote
1
answer
62
views
Wanted to eliminate the repetition in the classes created to reduce duplication
Can some one help me on how to eliminate repetition to reduce duplication score on the below classes created in the code
...
- 11
1
vote
0
answers
22
views
processing webhook request coming from a 3rd party application
I need some help to evaluate whether I am doing it right. The scenario is, a 3rd party application is sending a webhook request after a successful payment but the problem is that sometimes this ...
- 11
1
vote
0
answers
85
views
Tiny persistent storable class
I would like to have a minimal storable class in Python that I can use as ORM. I quite thinking my approach is bad, but I would like to have your opinion.
I do not want to use SQLAlchemy or PeeWee as ...
- 975
4
votes
2
answers
283
views
Creating a database, a table within the database and inserting some values into it in one go
I've written a script which creates a new database, a table within the database, ...
- 2,471
3
votes
2
answers
387
views
Downloading and parsing research papers
I am trying to write a script which gets a research paper from a website by calling their API and then traverse it sentence-wise with some conditions.
The paper is accessible in XML format. I am ...
- 31
1
vote
1
answer
62
views
Covid-19 World Case Report Beta 0.1
I am a self-learned programmer and this is my first program. I would really appreciate any critique on my approach.
The program should automatically download the who data file to the folder these ...
- 11
8
votes
4
answers
1k
views
Generalization of any() function with switchable default parameter for empty iterables
I have some use-cases where I want the behaviour of the build-in any() function to be slightly different, namely to return True ...
- 5,492
2
votes
0
answers
1k
views
Providing different __table_args__ to SQLAlchemy mapped class depending on backend DBMS
I need to set different __table_args__ for the mapped class depending on the backend DBMS, e.g. mssql and ...
- 21
7
votes
1
answer
444
views
LinDB: A dictionary type database
As an exercise, I've decided to write a lightweight, dictionary type database. Below are some of the features I've implemented:
Overwrite Inserting: I allow the user to determine if they want to ...
- 9,832
3
votes
0
answers
1k
views
Product inventory database with attributes and variants
I have design a model for product apps. This is the first time I am trying to create a product's database schema and then model it in Django. My code handles the following:
A product type can have ...
- 211
3
votes
1
answer
282
views
Get future events from Pinnacle API
My program makes API requests to Pinnacle Sports to retrieve future events and saves them to the database.
Questions:
Do I manage the database resources correctly? It takes about 3 seconds to check ...
1
vote
0
answers
117
views
Database connector in Python
I've been trying to build an ORM. I started with the connector layer, which is responsible for the database server connections and executes raw SQL as the code below shows.
I have built the ...
- 11
2
votes
0
answers
34
views
Handling failures in DB transactions
This code is meant to retry the db operation when a DB connection breaks: OperationalError.
A transaction can fail for example when a db is being restarted and a commit fails. Or when a network error ...
- 121
1
vote
1
answer
2k
views
SQLite and Python: commit once every 10 seconds maximum, and not after every client request
In a Python Bottle server using SQLite, I noticed that doing a DB commit after each INSERT is not efficient: it can use 100ms after each client request. Thus I wanted to improve this in How to commit ...
- 244
4
votes
1
answer
997
views
Interface class for SQLite databases
I am learning the basics about managing an SQLite database by using Python and, mainly for practice purposes, I have then defined an interface class in order to let operations on database as clear as ...
- 223
3
votes
1
answer
3k
views
Python script for refreshing and preprocessing data in Excel report files
I wrote some code to create marketshare reports. However, I'm sure it can be improved. The tasks performed are:
Open the global data file, refresh the data, import its content as a DF.
Create a ...
2
votes
0
answers
33
views
Gathering data using sqlalchemy from database and making it available using simplified namedtuple object in an iterable class
I wrote the below code today:
...
- 840
2
votes
1
answer
256
views
Speed up Django migration that adds two fields into one
Currently trying to migrate two fields of information into one field. We have many, many objects to go through and it takes roughly 12 minutes to loop through all the objects. That would mean the ...
- 163
3
votes
2
answers
504
views
Access a database and an API, uses python generators, and matches the results based on names, interview assignment
Thanks in advance if you are reading this code.
I recently submitted this code as part of an interview (took about 4-5 hours). Unfortunately, they did not like the code and I received a form ...
- 31
3
votes
1
answer
165
views
Get data from pipedrive API and insert it into Snowflake
I am a new python programmer.
I have created the code to put API data from pipedrive to Snowflake database.
Here are the steps in my code.
Delete the csv file if it exists.
Make an API call put all ...
- 131
1
vote
0
answers
232
views
Python database powered by SSTable
I've written up a key-value store that runs on SSTables in python.
...
- 1,031
4
votes
2
answers
114
views
First program in Python for data analysis
I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
- 43
3
votes
1
answer
67
views
Simple User and Database Relationship
Hello and thanks for clicking!
I'm currently studying the relationship between a database and user objects. This is a very simple program and the "database" itself is a dictionary. This will be used ...
- 33
4
votes
1
answer
182
views
Django model for real estate
I have built a real estate management system with Django. I want to know if my design database was wrong. Please let me know how to improve it.
Should I make the models use more than two tables (...
- 69
2
votes
1
answer
2k
views
Conditional joining Pandas dataframes
I'm looking for an optimum way to join two Dataframes where the same record exists in both. Examples would be:
...
- 43
6
votes
1
answer
174
views
Polymorphic Data Socket Factory with Query Contingency Protocols in Python
I work on a small data team where we developed this tool when we began experiencing several job failures on our legacy big data systems. We have several data lakes, each with their own API, and until ...
- 533
3
votes
2
answers
136
views
Finding the duration of how long a search has been popular
Program:
I have designed a program in python that does the following (in order):
Webscrapes google trends for the most popular searches
Stores the searches and traffic of the searches in an in-memory ...
1
vote
1
answer
878
views
Isolate a Database Change Within Django Transaction.Atomic
I have been typically adding the @transaction.atomic decorator to my endpoints and appreciate if a request fails at any point the data completely rolls back. However, some of my endpoints will make ...
2
votes
1
answer
583
views
Improvement in file management system based on the file names
Context:
I have a program which stores data to the disk. The data is then reprocessed during some of the iterations. Thus, it needs to store, search and load set of data.
Let's consider the ...
- 199
2
votes
0
answers
223
views
Django social network application - database schema
I am writing a simple social media application in Django. I would like to know whether the following model structure is correct, or how can I improve the schema to get better performance. Do I need to ...
- 121
3
votes
1
answer
771
views
Script to store, search, and delete IPFS data with description in JSON file
Here is the script that I would like reviewed for the following:
Best practices and design pattern usage
Correctness in unanticipated cases
Better dictionary access
The script is made to store (IPFS)...
- 85
3
votes
1
answer
119
views
