Skip to content
master
Switch branches/tags
Code

Latest commit

79186: changefeedccl: send different column families to different topics r=[miretskiy] a=HonoreDB

This change necessitated touching every sink, which was making me
cranky, so this commit refactors to pull topic name generation
out of sink and encoder code and into a new topic.go file.
The goal here is that when we add another target type, sinks won't
need to know about it.

As a side effect, this also addresses an old TODO to implement
full_table_name on all sinks. The other options that affect topic
name generation logic are all at the sink level, so they haven't
moved to global application by default, but it'd be easy to do.

One interesting consequence of having different topics per family
is that we lose the invariant enforced by some sinks that no new
topics get created after initialization. For split_column_families,
a topic is now created "lazily" when emitting to it. The topics shown
in e.g. SHOW CHANGEFEED JOB will in the case of split_column_families
be placeholders. A topic will display as foo.{family} when events
for foo will actually be emitted to foo.primary, foo.family2, etc.
For now, in Kafka, PubSub and SqlSink I've disallowed sending resolved
 timestamps with split_column_families since the change frontier
 doesn't have the full set of topics to emit them to. This is a solveable
problem but not trivial because of schema changes.

Arguably this ship already sailed with ALTER CHANGEFEED, but it
still seems worth calling out, as we could still special-case back
in the old behavior if there's still a need for it.

Release note (enterprise change): Changefeed messages for tables with multiple column families will append the family name to the table name in the default topic. For example, `CREATE CHANGEFEED FOR TABLE foo, TABLE bar FAMILY primary` will emit to the topics "foo" and "bar.primary". A changefeed created with the `split_column_families` option will create new topics if new families are added to a watched table that already had multiple column families. In cloud storage sinks such as s3, the filename will include the family separated by a dash (e.g. "bar-primary"). The full_table_name option is now supported for all sinks.

Co-authored-by: Aaron Zinger <zinger@cockroachlabs.com>
84e11ae

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
Apr 5, 2022
pkg
Apr 5, 2022
Oct 21, 2021


CockroachDB is a cloud-native distributed SQL database designed to build, scale, and manage modern, data-intensive applications.

What is CockroachDB?

CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

For more details, see our FAQ or architecture document.

Docs

For guidance on installation, development, deployment, and administration, see our User Documentation.

Starting with CockroachCloud

We can run CockroachDB for you, so you don't have to run your own cluster.

See our online documentation: Quickstart with CockroachCloud

Starting with CockroachDB

  1. Install CockroachDB: using a pre-built executable or build it from source.
  2. Start a local cluster and connect to it via the built-in SQL client.
  3. Learn more about CockroachDB SQL.
  4. Use a PostgreSQL-compatible driver or ORM to build an app with CockroachDB.
  5. Explore core features, such as data replication, automatic rebalancing, and fault tolerance and recovery.

Client Drivers

CockroachDB supports the PostgreSQL wire protocol, so you can use any available PostgreSQL client drivers to connect from various languages.

Deployment

  • CockroachCloud - Steps to create a free CockroachCloud cluster on your preferred Cloud platform.
  • Manual - Steps to deploy a CockroachDB cluster manually on multiple machines.
  • Cloud - Guides for deploying CockroachDB on various cloud platforms.
  • Orchestration - Guides for running CockroachDB with popular open-source orchestration systems.

Need Help?

Building from source

See our wiki for more details.

Contributing

We welcome your contributions! If you're looking for issues to work on, try looking at the good first issue list. We do our best to tag issues suitable for new external contributors with that label, so it's a great way to find something you can help with!

See our wiki for more details.

Engineering discussions take place on our public mailing list, cockroach-db@googlegroups.com. Also please join our Community Slack (there's a dedicated #contributors channel!) to ask questions, discuss your ideas, and connect with other contributors.

Design

For an in-depth discussion of the CockroachDB architecture, see our Architecture Guide. For the original design motivation, see our design doc.

Licensing

Current CockroachDB code is released under a combination of two licenses, the Business Source License (BSL) and the Cockroach Community License (CCL).

When contributing to a CockroachDB feature, you can find the relevant license in the comments at the top of each file.

For more information, see the Licensing FAQs.

Comparison with Other Databases

To see how key features of CockroachDB stack up against other databases, check out CockroachDB in Comparison.

See Also