Postgresql count+sort performance

As @GordonLinoff says, these queries are going to be slow regardless of the database involved, but it's helpful to know why. Consider how a database can execute this query:

SELECT table1.*, count(*)
FROM table1
JOIN table2 ON table2.id1 = table1.id
GROUP BY table1.id

Assuming table2 contains data for most rows in table1 and both tables are nontrivially sized, relational databases will tend to do the following:

• Scan table2, computing the aggregate on id1, yielding a { id1, count } result set.
• Scan table1.
• Hash join.

Adding or not adding an ORDER BY count doesn't materially change the amount of work: you've still got two table scans and a JOIN, you've just added a sort step. You might try to add an index on table2 (id1), but all that could improve is the aggregation step: now instead of reading two whole tables, you're reading one whole table and a whole index. Joy.

If you can eliminate most of the rows from consideration using indexes on one or both tables, by all means do so. Otherwise, the operation will always boil down to two scans, and as your data set gets larger, this gets less and less performant.

Incidentally, this is the effect of dropping the ORDER BY in your query: by leaving the LIMIT clause, you've told PostgreSQL you're only interested in the first N rows. This means it can pick N rows from table1 and do a nested loop against table2 -- for each of those N rows in table1, it finds count(*) in table2 for that specific ID using an index on that ID. This is what makes it much faster: you've excluded most of table2.

If your application commonly needs a count of associated records, the usual solution is to maintain a counter yourself. One convention (natively supported by Rails and several other ORMs) is to add a table2_count column to table1. If you index this counter, an ORDER BY ... LIMIT query will be extremely performant.

If your tools can't do this out of the box, or you're using a diverse set of tools to manipulate this database, triggers are a better option. You could put this in a separate summary table as @GordonLinoff suggests -- that might mean less contention in the base table, but it forces a JOIN when retrieving counts. I'd suggest adding a table2_count column to table1 first and breaking it out only if performance measurements indicate it's a win.

The queries that you have written are going to be slow in any database. The comparison to the query without the order by is interesting. The speed return suggests that an index is involved. If so, then it can find the counts from the index.

A fairer comparison is to the query without the order by and with no limit clause. That way, all the rows will be generated, just as in the version with the order by. Basically, the database engine must evaluate all the rows in order to find the top 10. The optimizer decides whether it needs to sort the data or take some other approach.

You have several options. The first is to see if you can speed up the performance of the query by changing parameters specific to Postgres. For instance, perhaps the page cache is too small and could be expanded. Or, perhaps there are sort optimization paramters that can help.

Second, you can have a summary table, as you suggest, which is built by a job that runs periodically. If slightly out-of-date data is no problem, then this is fine.

Third, you can have the summary table, but populate it using triggers rather than a job. When the data changes, update the various counts.

Fourth, you might experiment with other approaches. For instance, perhaps Postgres optimizes the window function COUNT(*) over () better than the aggregation. Or, it might optimize row_number() on the aggregated results better than an order by. Or, if you can live with only one value instead of 10, then MAX() is sufficient.

Based on the posted plans your row-count estimates are fine and the plans look vaguely sane. Your main issue is the big sort, probably necessitated by the ORDER BY:

Sort Method: external merge Disk: 64288kB

That's going to hurt even if you have fast storage. If you're on a single hard drive or (worse) a RAID5 array, that's going to be very, very slow. That sort goes away with Erwin's updated query, but increasing work_mem is still likely to gain you some performance.

You should increase work_mem, either for this query or (less) globally to get much better performance. Try:

SET work_mem = '100MB';
SELECT your_query

and see what difference it makes.

You may also want to play with the random_page_cost and seq_page_cost parameters to see if a different balance produces cost estimates that're a better match for your environment and thus causes the planner to choose a quicker query. For relatively small amounts of data like this where most of it will be cached in RAM I'd start with something like random_page_cost = 0.22 and seq_page_cost = 0.2. You can use SET for them like you do work_mem, eg:

SET work_mem = '100MB';
SET random_page_cost = 0.22;
SET seq_page_Cost = 0.2;
SELECT your_query

Do not set work_mem that high if you're setting it in postgresql.conf and you have lots of active connections, as it's per-sort not per-query, so some queries could use several times work_mem and just a couple at once could bring the system to memory exhaustion; you need to set it low enough that each connection in max_connections can be using 2x or 3x work_mem without your system running out of memory. You can set it per-transaction with SET LOCAL, per-user with ALTER USER ... SET, per-database with ALTER DATABASE ... SET or globally in postgresql.conf.

See:

• Resource consumption configuration parameters
• Tuning your PostgreSQL server