I have multiple (1000+) JSON files each of which contain a JSON array. I want to merge all these files into a single file.
I came up with the following, which reads each of those files and creates a new object with all the contents. I then write this new object into a new file.
Is this approach efficient? Is there a better way to do so?
head = []
with open("result.json", "w") as outfile:
for f in file_list:
with open(f, 'rb') as infile:
file_data = json.load(infile)
head += file_data
json.dump(head, outfile)
headvariable. If you have a lot of data in those 1000+ files, you might start running out of memory. – alexwlchan Apr 18 at 14:00[]). Add commas and array markers as appropriate between items. Then you only have one file in memory at a time. – alexwlchan Apr 18 at 14:01