I'm using node-csv-parser to read in csv data and store it in mongodb using mongoose. However I'm trying to speed up the import, and I want to evaluate using the native save method exposed by node-mongodb-native
, accessed in mongoose using Model.collection
. (This at the advice of a mongo engineer I spoke to at office hours at Mongo HQ).
node-csv-parser
fires a data
event every time it reads in a new line of the csv. Inside this event I read in the line of data, create a new data point from it, and save it in mongo. I can save the data point just fine inside the data
event using my mongoose model TestDataPoint
. However if I try to instead create a javascript object for the data point and save it using TestDataPoint.collection.save
, I get the error: RangeError: Maximum call stack size exceeded
.
I've tried calling this native save in all sorts of different ways, including getting the collection directly through mongoose.connection.db.collection("testdatapoints")
and by sending it out to a queue provided by the async module, but always with more or less the same result. I can successfully save a data point using the native driver elsewhere in my code, and even inside the end
event of the csv import, just not inside the data
event.
I have also determined through logging that on my current setup (Ubuntu 12.04 on a 64-bit AMD processor, 8 gb RAM), the code iterates through 154 lines of the csv before throwing the stack error, however no data is written to the db from inside this data
event. It seems there is some kind of recursion happening(?) unintentionally, or perhaps it is some kind of bug between node-csv-parser and node-mongodb-native.
To clarify, with my (edited/updated) sample code below, the log repeats 154 times:
about to call native save
just called native save
and then says:
in native save callback for dataPoint: 1
Native save failed, error:RangeError: Maximum call stack size exceeded
in native save callback for dataPoint: 2
Native save failed, error:RangeError: Maximum call stack size exceeded
up to dataPoint 154, then it repeats the "about to call/just called" another 154 times, then logs the errors for dataPoints 155-308 and so on (I have a lot of data points to import). This 154 number is pretty repeatable, thought I've observed it going through 155 lines at a time once or twice.
It was suggested that I wrap the save call in a process.nextTick()
to clear the stack. When I try that, in my log I can see that the save is called 154 times, then process.nextTick()
is called 154 times, then the RangeError is logged 154 times, and the sequence repeats.
I'm running node 0.8.2, mongoose 2.7.2, and mongodb 2.0.4.
csv()
.fromPath(path)
.on("data", (data, index) ->
# cellTest is an instance of a Mongoose model object
newDataPoint =
testId: cellTest.testId # this assignment was causing recursion in the native save
dataPoint: data[1]
testTime: data[2]/3600
cycleIndex: data[3]
console.log "about to call native save"
# TestDataPoint is my my mongoose model, which saves fine, but
# this call throws the RangeError: Maximum stack size exceeded
TestDataPoint.collection.save newDataPoint, safe:true, (err, dataPoint) ->
console.log "in native save callback for dataPoint: " + data[1]
if err
console.log "Native save failed, error:" + err
console.log "just called native save"
.on("end", (count) ->
newDataPoint =
dataPoint: 100 # dummy values
testTime: 200
cycleIndex: 300
# This call works, saves the data point
TestDataPoint.collection.save newDataPoint, safe:false, null
)
.on("error", (err) ->
console.log err
)
EDIT: SOLVED!
The assignment:
testId: cellTest.testId
was causing recursion on the save. Something to do with cellTest being an instance of another Mongoose model. Changing the assignment to:
testId: parseInt(cellTest.testId)
eliminated the recursion and allowed the saves to execute.
TestDataPoint.collection.save newDataPoint, safe:false, null
is returning any errors in the callback prior to theRangeError
? – Bill Jul 20 '12 at 5:23newDataPoint
right before you get the RangeError? – Bill Jul 20 '12 at 6:03TestDataPoint
have any embedded documents in it? The.save
call must be calling itself recursively for some reason. I assume if you remove the.save
acll and just log your data points everything works fine. – Bill Jul 20 '12 at 15:56