I am working with quite large files (pytables) and I am having problems with the Memory Error when I try to load the data for processing.
I would like some tips about how to avoid this in my python 32bits, since I am new working with pandas and pytables, and I do not know how to work splitting the data in small pieces.
My concern also comes when, if I get to split the data, how to calculate statistics like mean, std, etc without having the entire list or array, etc.
This is a sample of the code that I am using now, this works fine with small tables:
def getPageStats(pathToH5, pages, versions, sheets):
with openFile(pathToH5, 'r') as f:
tab = f.getNode("/pageTable")
dversions = dict((i, None) for i in versions)
dsheets = dict((i, None) for i in sheets)
dpages = dict((i, None) for i in pages)
df = pd.DataFrame([[row['page'],row['index0'], row['value0'] ] for row in tab.where('(firstVersion == 0) & (ok == 1)') if row['version'] in dversions and row['sheetNum'] in dsheets and row['pages'] in dpages ], columns=['page','index0', 'value0'])
df2 = pd.DataFrame([[row['page'],row['index1'], row['value1'] ] for row in tab.where('(firstVersion == 1) & (ok == 1)') if row['version'] in dversions and row['sheetNum'] in dsheets and row['pages'] in dpages], columns=['page','index1', 'value1'])
for i in dpages:
m10 = df.loc[df['page']==i]['index0'].mean()
s10 = df.loc[df['page']==i]['index0'].std()
m20 = df.loc[df['page']==i]['value0'].mean()
s20 = df.loc[df['page']==i]['value0'].std()
m11 = df2.loc[df2['page']==i]['index1'].mean()
s11 = df2.loc[df2['page']==i]['index1'].std()
m21 = df2.loc[df2['page']==i]['value1'].mean()
s21 = df2.loc[df2['page']==i]['value1'].std()
yield (i,m10, s10), (i,m11, s11), (i,m20,s20), (i,m21,s21))
As you can see, I am loading all the necessary data into a Pandas DataFrame to procoess it, just the mean and the std by now.
This is quite fast, but for a pytable with 22Millions of rows I get Memory Error
