Tell me more ×
Raspberry Pi Stack Exchange is a question and answer site for users and developers of hardware and software for Raspberry Pi. It's 100% free, no registration required.
up vote 3 down vote favorite

Currently I am reading data from a GPS receiver through USB on a raspberry pi and I would want to log this data on the SD card. I am collecting the data through Python by using

file = open("file.txt", "w")
while True:

    ###PULL DATA THROUGH USB###

    file.write( DATA )

Currently I am SSHing ( using Putty) into the RPi to initiate this program. However there is no constant network connection to the RPi and thus SSH connection loses communication at times and sometimes the data doesn't isn't written and saved. I have also tried running the program at start-up by changing the rc.local but when I pull the power and extract the SD Card, the data seems to be lost.

Is the program still running if the SSH disconnects (goes out of receiver range)?

I was wondering what the best way to log data when there is no way to send a stop command (like a keyboard interrupt to end python data collection program) is?

share|improve this question
add comment (requires an account with 50 reputation)

3 Answers

up vote 1 down vote accepted

to avoid losing data, you have to open your file in append mode:

f = open( 'file.txt', 'a')

then write your data and then close it:

f.write(DATA)
f.close()

when you close the file, your data fill be saved to SD and all file tables properly updated.

if you want to keep your file open, you may try to flush/fsync the contents:

f.flush()
os.fsync(f.fileno())

however in the flush/fsync case some data still might be lost, so i'd recommend using append/close approach above.

share|improve this answer
add comment (requires an account with 50 reputation)
up vote 2 down vote

Along with the append mode for file I/O (as answered by others), you can also try using screen command. This how, you are able to put any task into separate "screen" and don't need to worry about connectivity problems when doing stuff remotely.

I would also recommend using some other logging method, like SQL database (e.g. MySQL with Python), if there is a lot of logging going on. SQL data can easily get exported (e.g. to CSV file) and also any big data management is faster, like (conditional) searching, updating and deleting the data.

share|improve this answer
+1 I was just about to post this very thing. I wouldn't recommend MySQL on the pi. It is usable, but it's a bit overkill for most needs. Sqlite3 would probably be a better bet. – Jacobm001 2 days ago
add comment (requires an account with 50 reputation)
up vote 0 down vote

I think there are a few questions here- first, dropped ssh means processes in the shell will go away. Look into nohup.

Second, rc.local should work, ensure you have given proper paths since your environment variables won't be provided to the script. (look at /var/log/messages and know it's similar to the same problem happening in cron). Otherwise you might have buffering issues that keep it from writing.

My suggestion is to put a test program in rc.local that simply opens/writes/closes a file and make sure it runs properly, which will ensure your paths are correct. You can start it with /full/path/to/python /full/path/to/script.py. You can also add it to your crontab to ensure it works rather than starting the pi each time.

share|improve this answer
add comment (requires an account with 50 reputation)

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.