March 12 | PGConf.Online 2021 wrap-up: 57 talks, 30 international speakers, 2000+ registrants!

PGConf.Online wrapped up 10 days ago, and we want to share some important stats with you. We were glad to have welcomed 2000+ registrants this year with 700-900 of them joining us online each day! The total attendance/registration rate for all three days was near 75%, which is amazing!

March 8 | Celebrating Women Who Code in Postgres Pro

In early March, before International Women’s Day, we asked our female tech professionals a few questions and were surprised by some of them providing us answers :) Let’s celebrate women who code working here at Postgres Pro and have a look at what they are up to!

February 18 | Postgres Professional is now sponsoring Psycopg development!

Postgres Professional has become one of the main sponsors backing the development of the Psycopg library, the most popular PostgreSQL adapter for the Python programming language.

February 3 | Postgres Professional at FOSDEM 2021

It’s this time of the year again, and FOSDEM is coming! In 2021, Postgres Professional is really well-represented in the PostgreSQL devroom, as we have 5 talks accepted by the Committee. Let’s take a look at all presentations to be given by our team at this conference.

April 1 | WAL in PostgreSQL: 1. Buffer Cache

The previous series addressed isolation and multiversion concurrency control, and now we start a new series: on write-ahead logging. To remind you, the material is based on training courses on administration that Pavel Luzanov and I are creating (mostly in Russian, although one course is available in English), but does not repeat them verbatim and is intended for careful reading and self-experimenting.

This series will consist of four parts:

• Buffer cache (this article).
• Write-ahead log — how it is structured and used to recover the data.
• Checkpoint and background writer — why we need them and how we set them up.
• WAL setup and tuning — levels and problems solved, reliability, and performance.

Many thanks to Elena Indrupskaya for the translation of these articles into English.

Why do we need write-ahead logging?

Part of the data that a DBMS works with is stored in RAM and gets written to disk (or other nonvolatile storage) asynchronously, i. e., writes are postponed for some time. The more infrequently this happens the less is the input/output and the faster the system operates.

But what will happen in case of failure, for example, power outage or an error in the code of the DBMS or operating system? All the contents of RAM will be lost, and only data written to disk will survive (disks are not immune to certain failures either, and only a backup copy can help if data on disk are affected). In general, it is possible to organize input/output in such a way that data on disk are always consistent, but this is complicated and not that much efficient (to my knowledge, only Firebird chose this option).

Usually, and specifically in PostgreSQL, data written to disk appear to be inconsistent, and when recovering after failure, special actions are required to restore data consistency. Write-ahead logging (WAL) is just a feature that makes it possible.

...

March 19 | MVCC in PostgreSQL — 8. Freezing

We started with problems related to isolation, made a digression about low-level data structure, discussed row versions in detail and observed how data snapshots are obtained from row versions.

Then we covered different vacuuming techniques: in-page vacuum (along with HOT updates), vacuum and autovacuum.

Now we've reached the last topic of this series. We will talk on the transaction id wraparound and freezing.

Transaction ID wraparound

PostgreSQL uses 32-bit transaction IDs. This is a pretty large number (about 4 billion), but with intensive work of the server, this number is not unlikely to get exhausted. For example: with the workload of 1000 transactions a second, this will happen as early as in one month and a half of continuous work.

But we've mentioned that multiversion concurrency control relies on the sequential numbering, which means that of two transactions the one with a smaller number can be considered to have started earlier. Therefore, it is clear that it is not an option to just reset the counter and start the numbering from scratch.

...

February 9 | MVCC in PostgreSQL — 7. Autovacuum

To remind you, we started with problems related to isolation, made a digression about low-level data structure, discussed row versions in detail and observed how data snapshots are obtained from row versions.

Then we explored in-page vacuum (and HOT updates) and vacuum. Now we'll look into autovacuum.

Autovacuum

We've already mentioned that normally (i. e., when nothing holds the transaction horizon for a long time) VACUUM usually does its job. The problem is how often to call it.

If we vacuum a changing table too rarely, its size will grow more than desired. Besides, a next vacuum operation may require several passes through indexes if too many changes were done.

If we vacuum the table too often, the server will constantly do maintenance rather than useful work — and this is no good either.

Note that launching VACUUM on schedule by no means resolves the issue because the workload can change with time. If the table starts to change more intensively, it must be vacuumed more often.

Autovacuum is exactly the technique that enables us to launch vacuuming depending on how intensive the table changes are.

...

JsQuery

JsQuery – is a language to query jsonb data type, introduced in PostgreSQL release 9.4. It's primary goal is to provide an additional functionality to jsonb (currently missing in PostgreSQL), such as a simple and effective way to search in nested objects and arrays, more comparison operators with indexes support.

pg_probackup

Pg_probackup is a utility to manage backup and recovery of PostgreSQL database clusters. It is designed to perform periodic backups of the PostgreSQL instance that enable you to restore the server in case of a failure.

pg_variables

Functions for defining and using variables in client sessions.

mamonsu

An active monitoring agent for Postgres Pro. Based on zabbix, mamonsu provides an extensible cross-platform solution that can collect and visualize multiple Postgres Pro and system metrics.

Multi-master cluster with sharding

Multi-master cluster with sharding which provides read/write scalability as well as high availability is obviously one of most wanted DBMS features. Experience shows that we should move in step-by-step manner in order to have it in core one day. Postgres Professional company joins community efforts in this direction.

Better temporary tables

Temporary tables are often being used. Their performance could be increased if they are removed from system catalog. Also it is attractive to allow using temporary tables on read-only standbys.

Page-level data compression

PGLZ compression of individual values is used now in PostgreSQL. That’s good, but sometimes it’s possible to achieve significant compression only when compressing multiple values together. This is why we’re considering page-level data compression.