Tell me more ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 1 down vote favorite

Here's my problem:

Model:

{ application: "abc", date: Time.now, status: "1" user_id: [ id1, id2, id4] }

{ application: "abc", date: Time.yesterday, status: "1", user_id: [ id1, id3, id5] }

{ application: "abc", date: Time.yesterday-1, status: "1", user_id: [ id1, id3, id5] }

I need to count the unique number of user_ids in a period of time.

Expected result:

{ application: "abc", status: "1", unique_id_count: 5 }

I'm currently using the aggregation framework and counting the ids outside mongodb.

{ $match: { application: "abc" } }, { $unwind: "$users" }, { $group: { _id: { status: "$status"}, users: { $addToSet: "$users" } } }

My arrays of users ids are very large, so I have to iterate the dates or I'll get the maximum document limit (16mb).

I could also $group by

{ year: { $year: "$date" }, month: { $month: "$date" }, day: { $dayOfMonth: "$date" }

but I also get the document size limitation.

Is it possible to count the set size in mongodb?

thanks

share|improve this question
Do you have more than 16mb ids per user or do you have more than 16mbs of data for all of the records? If the latter condition exits, you can try flushing the result to an ouput collection. – cubbuk Jan 28 at 18:13
The users array/set size is bigger than a thousand, and user ids are similar to object_ids (50b9d949816e6e37060005c2). The previous version was using map/reduce and an output collection. It was terribly slower. Counting in memory is faster than writing an output collection. – user2019059 Jan 28 at 18:17
And how is the performance when you make a table scan and retrieve only application and userId fields. Of course counting in memory is faster but you have limitations with mongo, as far as I know if the output doesn't fit into memory flushing to disk or making a table scan is your only option. – cubbuk Jan 28 at 18:30
The performance is acceptable. I just wish for a way to count an array size and not return the entire content. – user2019059 Jan 28 at 18:40

2 Answers

up vote 0 down vote accepted

The following will return number of uniqueUsers per application. This will apply an group operation to a result of a group operation by using pipeline feature of mongodb.

{ $match: { application: "abc" } }, { $unwind: "$users" }, { $group: { _id: "$status", users: { $addToSet: "$users" } } }, {$unwind:"$users"}, {$group : {_id : "$_id", count : {$sum : 1}}}

Hopefully this will be done in an easier way in the following releases of mongo by a command which gives the size of an array under a projection. {$project: {id: "$_id", count: {$size: "$uniqueUsers"}}} https://jira.mongodb.org/browse/SERVER-4899

Cheers

share|improve this answer
up vote 0 down vote

Sorry I'm a little late to the party. Simply grouping on the 'user_id' and counting the result with a trivial group works just fine and doesn't run into doc size limits.

[
    {$match: {application: 'abc', date: {$gte: startDate, $lte: endDate}}},
    {$unwind: '$user_id'},
    {$group: {_id: '$user_id'}},
    {$group: {_id: 'singleton', count: {$sum: 1}}}
];
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.