All Questions
33
questions
32
votes
3
answers
28k
views
How to efficiently store big time series data?
I need to store and to be able to query some very large amounts time series data.
Properties of the data are as follows:
number of series : around 12.000 (twelve thousand)
number of data points, ...
- 445
23
votes
5
answers
27k
views
When should use of database be preferred over parsing data from a text file?
I was making a Python program to measure the growth of codereview.SE. My approach was to get the "Site stats" shown on the front page and store them on my hard drive. I plan to do this once every day. ...
- 3,004
8
votes
1
answer
3k
views
Making API calls with celery
I'm designing a system for a client where the requirements are:
they upload a JSON file (one object/line)
make a call to an API with the JSON object as the payload
record the state (success/failure) ...
- 101
7
votes
3
answers
469
views
Does it make sense to develop open source python library for database inspection? [closed]
Some time ago I came up with an idea for a library for database inspection. I started developing it and got some very basic functionality, just to check if that's possible. Recently however, I get ...
- 602
5
votes
3
answers
2k
views
Should we have a database independent SQL like query language in Django?
Note :
I know we have Django ORM already that keeps things database independent and converts to the database specific SQL queries.
Once things starts getting complicated it is preferred to write raw ...
- 359
4
votes
1
answer
4k
views
Enum or reference table when dealing with maps
I am storing information about Widgets in my database, and each Widget has one (non-unique) transformation function associated with it. My problem is how to associate Widgets with their transformation ...
- 151
4
votes
1
answer
117
views
Maintaining parallel libraries: Binary files access + Metadata Database ORM design
I'm dealing with a higher-level data abstraction that I would appreciate some input on.
I'm working on an application that uses a large data lake. The data lake is consisted of thousands of large ...
- 151
4
votes
1
answer
300
views
Class instance clustering in object reference graph for multi-entries serialization
My question is on the best way to cluster a graph of class instances around specifically marked objects (objects are the graph nodes and the references to each other are the directed edges of the ...
- 171
3
votes
2
answers
2k
views
Is a bad practice create database structure on the runtime?
I'm creating a crawler app that will always, in the beginning, when the Class constructor, ensure that the basic database structure exists.
Is this a bad practice? What is the advantage OS creating ...
- 157
3
votes
3
answers
279
views
How to DRY with calls to a database to execute?
I'm writing a python app that instructs a database to perform various processing instructions on partitioned tables. The processing can take long enough that there might be a timeout, so I surround my ...
- 171
3
votes
1
answer
401
views
optimize human-readable database with index
i need to store a large amount of data - about 10 million entries of the format
unique hash (64 chars), value 1 (5 chars), value 2 (9 chars)
i will be reading and deleting (but not updating) this ...
- 199
3
votes
2
answers
331
views
Best approach to microservice shared databse architecture
I have two microservices, one Flask (python) and one Spring (java), they currently share a database. The Flask microservice handles processing json files (~40mb) for each user (could be 100's or 1000'...
- 31
3
votes
4
answers
1k
views
Notify players every x seconds in a multiplayer game
I'm working on a realtime multiplayer game using Django and gevent-socketio, I'm facing some issues:
I need to send an update of the game state to connected players every X seconds (~4 seconds), so ...
- 165
3
votes
1
answer
962
views
Django: caching properties for non-changing entries
I am wondering if it is a good idea to do the following:
I have a Django model (which is related to a migration, therefore it has a database entry) with a bunch of properties. Accessing these are ...
- 2,256
2
votes
1
answer
251
views
two diffrent database in unit of work
I will explain my problem in the form of an example.
Suppose we want to use both databases in a transaction.
Data is edited in database 1 (for example, Postgres) and then added to database 2. Finally, ...
- 29
2
votes
1
answer
2k
views
How to organize ORM models and business logic
I have a Python application using SQLAlchemy. It uses around 15 database tables.
For example, I have an image table that lists image files manipulated by the application. Most operations involving ...
- 159
2
votes
1
answer
48
views
Setup pipeline to analyze data stored in web app DB
Background:
So there is a (Ruby) web app with a production Postgres DB (hosted in the cloud)
I would like to run some machine learning algorithms in a Python setting on the production data and (...
- 151
2
votes
2
answers
2k
views
Is it OK to use (WTF) forms to validate against stuff from DB?
Form classes are intended (IMO) for submitted data against rules. For example: are passwords equal, is end date later than start date.
submitted data--->|Form|
Is it okay for Form classes to ...
- 257
2
votes
1
answer
517
views
Transferring information between a web server and a Python backend?
I have been Googling my problem but haven't been able to come up with anything; here's the situation:
I have a PHP / MySQL website hosted with 1and1.com.
I also have a Python program that runs on a ...
- 21
1
vote
2
answers
156
views
Critique on design principle and validity of such in general
I was hoping you could give some feedback on an idea I had for designing functions.
I am trying to think of a unifying principle for choosing what functions should return. The specific project is ...
- 31
1
vote
2
answers
102
views
Dynamic web application hitting database on front page load to fetch profile image
Folks, I am in middle of writing a web application (Python/Flask) where home page has user profile image in the navbar which is coming from a database (blob), I am wondering if this is a good practice....
- 19
1
vote
1
answer
358
views
What is an apporpriate design pattern when dealing with Pandas and databases?
We're dealing with a lot of "data analysis", basically different sorts of data mangling, aggregations and calculations using Pandas. Usually, the data is time series data.
All underlying ...
- 13
0
votes
2
answers
242
views
How to query database effectively with the same program millions times
I got a proprietary software (something like a DMS system) to process the documents in file system. The software provide very limited API to manipulate the contents of the documents in DMS but it ...
- 11
0
votes
1
answer
94
views
Am I making this web app right?
I am currently making a newsletter website that will send you a email with the top ten manga for that week. Currently I have a web scraper that retrieves the top ten along with descriptions and other ...
0
votes
1
answer
168
views
How to use database adapters' cursors safely?
I started to use psycopg2 to connect my little python script to a PostgreSQL database few days ago. After some research I found that a lot of database connector, like psycopg, work using cursors.
I ...
- 396
0
votes
2
answers
1k
views
Is there a better way to trigger API calls from an on Prem SQL Server without using a job scheduler?
I'm not sure if this is the correct area to ask this question, quite honestly, I'm not sure how to phrase the question because I don't know if what I'm thinking is possible.
I'm trying to figure out a ...
- 1
0
votes
0
answers
87
views
Building a Feed Handler and Persisting JSON Data to Postgresql
I'm starting a pet project and would like some advice. I need to perform the following:
Scrape JSON Data from a REST API
Save the Data into my Postgresql table
Paginate through the rest API and ...
- 101
0
votes
2
answers
70
views
Is it worth to use a sql db for just two columns (possibly extensible)?
So, first of all hello to everyone, I hope you are having a good day :)
I will give you a little context. I coded a program for my job, in which I save to a database Errors ID and their solutions. It ...
0
votes
0
answers
103
views
Converting large files in python
I have a few files that are ~64GB in size that I think I would like to convert to hdf5 format. I was wondering what the best approach for doing so would be? Reading line-by-line seems to take more ...
- 109
-1
votes
2
answers
852
views
When should I use a database instead of a list? [duplicate]
New to Python and all things database related. Wondering when I should consider using a database and why?
I have what is essentially a list of objects that is around 30000 lines long. I'm developing a ...
- 121
-1
votes
3
answers
524
views
Is it better to override methods in classes or make methods general?
I am creating the backend of a microservice that will serve as a tool to see in real time how the company's employees are distributed by projects and what days they have assigned to each one. The ...
- 1
-1
votes
2
answers
460
views
Python/Database Website development [closed]
I want to create my own website that would allow me to acess a database in the server and do inserts and lookups in a user friendly way. I am a seasoned user of linux and C/C++, and also have ...
-4
votes
1
answer
248
views
Is it better (from a maintenance perspective) to dynamically construct complex SQL queries using Jinja or String concatenation? [closed]
Two possibilities. Which is better, with respect to readability, maintenance and clean code:
SQL injection can be disregarded.
Constructing complex parameterized sql queries using python string ...
- 5