Tell me more ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 14 down vote favorite
11

I am trying to load and parse a JSON file in python. Although I'm stuck trying to load the file:

import json    
json_data = open('file')
data = json.load(json_data)

Yields:

ValueError: Extra data: line 2 column 1 - line 225116 column 1 (char 232 - 160128774)

I looked at http://docs.python.org/library/json.html but it's pretty discouraging to read through this horrible looking documentation.

share|improve this question
4  
How do you know for a fact the file isn't mis-formatted? Can we see the first few lines? – Martijn Pieters Sep 16 '12 at 23:02
2  
If the file is big try validating at jsontlint.com – RanRag Sep 16 '12 at 23:03
@MartijnPieters, Okay, thank you. I will delete the comment. – mjgpy3 Sep 16 '12 at 23:06
   
@RanRag: jsonlint.com only offers a textbox, and can take raw JSON or a URL. You can't upload a file, so if it's big jsonlint.com will not help. – Martijn Pieters Sep 16 '12 at 23:06
4  
You have a JSON dict on each line. – Martijn Pieters Sep 16 '12 at 23:07
show 4 more comments
up vote 33 down vote accepted

You need to parse your file line by line:

data = []
with open('file') as f:
    for line in f:
        data.append(json.loads(line))

Each line contains valid JSON, but as a whole, it is not a valid JSON value as there is no top-level list or object definition.

Note that because the file contains JSON per line, you are saved the headaches of trying to parse it all in one go or to figure out a streaming JSON parser. You can now opt to process each line separately before moving on to the next, saving memory in the process. You probably don't want to append each result to one list and then process everything if your file is really big.

share|improve this answer
1  
+1 Maybe it is worth noting, that if you do not need all objects at once, processing them one-by-one may be more efficient approach. This way you will not need to store whole data in the memory, but a single piece of it. – Tadeck Sep 16 '12 at 23:13
1  
@Tadeck: yup, was already adding that. :-) – Martijn Pieters Sep 16 '12 at 23:15
1  
@Pi_: you'll have a dictionary, so just access the fields as keys: data = json.loads(line); print data[u'votes'] – Martijn Pieters Sep 16 '12 at 23:26
1  
@Pi_: print the result of json.loads() then or use the debugger to inspect. – Martijn Pieters Sep 16 '12 at 23:31
1  
@Pi_: no; don't confuse the JSON format with the python dict representation. You are seeing python dictionaries with strings now. – Martijn Pieters Sep 16 '12 at 23:37
show 5 more comments
up vote 2 down vote

That is ill-formatted. You have one JSON object per line, but they are not contained in a larger data structure (ie an array). You'll either need to reformat it so that it begins with [ and ends with ] with a comma at the end of each line, or parse it line by line as separate dictionaries.

share|improve this answer
3  
With a 50MB file the OP is probably better off dealing with the data line by line anyway. :-) – Martijn Pieters Sep 16 '12 at 23:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.