Change data capture

Change data capture (CDC) is a design pattern used to track and capture changes made to schema and data in a database. evitaDB supports CDC through all its APIs, allowing developers to monitor and respond to data changes very easily in near real-time in their preferred programming language. This document explains how to implement CDC using our API.

The database maintains a so-called Write-Ahead Log (WAL) that records all changes made to the database. This log is used to ensure data integrity and durability, but it can also be leveraged to implement change data capture (CDC) functionality. Once the catalogue is switched to the ACTIVE (transactional) stage, clients can start consuming information about changes made to both the schema and the data in the catalogue.

There is also a special CDC available for the entire database engine that allows clients to monitor high-level operations such as catalogue creation, deletion, and other global events (for more details, consult the Control Engine chapter).

Change data capture is not available for catalogues in the WARMING_UP stage since the WAL is not being recorded during that phase. This phase is considered "introductory" and clients should not work (query) with the data in that phase anyway. Clients should wait until the catalogue reaches the ACTIVE stage and perceive all the data at that moment as a consistent snapshot of the first version of the catalogue.

Engine and catalogue-level CDCs cannot be combined into a single stream since they operate on different levels (engine vs. catalogue). Catalogue-level CDC is always tied to a particular catalogue (name). If you need to capture all changes across all catalogues, you need to subscribe to engine-level CDC and then for each catalogue separately to catalogue-level CDC. The engine-level CDC notifies about catalogue creation/deletion events, so clients can dynamically subscribe/unsubscribe to catalogue-level CDCs as catalogues are created/deleted.

The basic principle in all APIs is the same:

clients define a predicate/condition that specifies which changes they are interested in,
define a starting point in the form of a catalogue version from which they want to start receiving changes,
and subscribe to the change stream.

From that point onwards, clients will receive notifications about all changes that match their criteria. The changes are delivered in the order they were made, ensuring that clients can process them sequentially. The second step is optional — if no starting version is specified, the change stream will start from the next version of the catalogue.

Subscription lifecycle

Once subscribed, the change stream remains active until one of the following occurs:

the client explicitly cancels the subscription
the client can't keep up with the rate of incoming changes (backpressure)
the client throws an exception during processing
the client doesn't react within a timeout
the server shuts down, or the catalogue is deleted or replaced
the server doesn't react within a timeout
the subscription TTL (time-to-live) expires - see configuration settings
the position the subscription is asked to start at can't be served - see resuming a catalogue subscription

As you can see, there are many reasons why a subscription may end. Therefore, clients should be prepared to handle such situations gracefully. The standard approach is to implement the AutoCloseable interface in your subscriber and re-establish the subscription in the close() method or schedule a re-establishment by another application service. Your subscriber should also track the resume position of the last successfully processed event - for a catalogue subscription the catalogId, version and index of the capture - so that it can resume from the correct point when re-establishing the subscription. The criteria handle version and index as inclusive, so you should skip the first event after resumption if it matches the last processed version and index.

Resuming a catalogue subscription

A catalogue version means something only within one incarnation of the catalogue. Replacing a catalogue (for example at the end of a full reindex into a temporary catalogue), restoring it from a backup or duplicating it puts a catalogue with a different identity and its own version sequence under the name. Renaming a catalogue or restarting the server keeps both its identity and its versions. The resume position of a catalogue subscription is therefore the triple (catalogId, version, index):

every catalogue capture carries catalogId - the identity of the catalogue incarnation that produced it,
store it together with the version and index of the last processed capture,
and pass all three back in the request - as catalogId, sinceVersion and sinceIndex - when you re-subscribe.

A position the catalogue can't serve ends the subscription with . Its reason tells you why:

DIFFERENT_INCARNATION: The catalogId of the request differs from the identity of the catalogue that bears the name - the catalogue has been replaced, restored or duplicated since the position was recorded. The decision is driven by identity alone: the position is refused whether the versions of the new incarnation happen to be lower or higher than the requested one.
AHEAD_OF_CATALOG: The sinceVersion lies more than one version past the current catalogue version. A position recorded on this incarnation is at most one version past it (the version the next change will carry, which is also where a subscription without sinceVersion starts), so such a position was recorded elsewhere. This check applies with or without catalogId and is the only protection a request without catalogId has - and it catches only a stale position that happens to lie ahead. A stale position that lies behind the current version of the new incarnation would be served from an unrelated version sequence, so always pass catalogId.
OUTSIDE_RETENTION: The write-ahead log no longer holds the changes the position points at. The position belongs to this incarnation, but the consumer fell too far behind and the changes it missed cannot be replayed. This is detected when the subscription catches up from the write-ahead log, so it can also end a subscription that is already running.

The exception extends , so a handler written for data that is no longer available catches every reason. It also carries the catalogId and the currentCatalogVersion of the catalogue the position was checked against; the inherited catalogVersion is the oldest catalogue version still available, when it is known.

Whatever the reason, the consumer can't continue where it left off: the changes between its position and the present are either gone or belong to a different dataset that merely carries the same name. Don't rewind the position silently - a consumer that holds state derived from the stream (a cache, a projection) has a gap in it and must rebuild it:

drop the derived state together with the stored resume position,
rebuild the state from one consistent snapshot of the catalogue - the data read by a single session, which always reads one catalogue version,
resume right after that snapshot: with the snapshot's catalogId, sinceVersion one past the snapshot's catalogue version and sinceIndex 0,
if this position is refused too - the rebuild took longer than the write-ahead log retains changes - rebuild again.

Subscribing from the head of the stream (without sinceVersion) once the rebuild has finished loses changes: the head has moved on while the rebuild ran, so the changes committed in the meantime never reach the derived state. Alternatively, you can register the subscription first, buffer its captures while the rebuild runs, and apply them once the rebuild has finished - as long as applying a change the rebuild has already seen does no harm.

Hierarchy of mutations

Not all mutations operate on the same level, and some mutations may encapsulate others. For example, when an entity is upserted, it may contain multiple mutations within it (multiple attribute, associated data, price operations, etc.). The hierarchy of mutations is as follows:

(complete listing, available in engine change capture)
- (complete listing, available in catalog schema change capture)
- (complete listing, available in catalog data change capture)
(available in all change capture streams)

When you don't specify any filtering criteria, you will receive all mutations in flattened form, i.e. you will receive all mutations regardless of their hierarchy. So, for example, an entity attribute upsert will be delivered once as part of the entity upsert mutation and once as a standalone attribute upsert mutation. In practice, a client usually wants either high-level information about entity changes (so only entity mutations) or very specific low-level changes (e.g. only changes to attributes of a particular name). The approach with a simple flattened stream that is filtered by a single predicate covers all these use cases very well, and it is very easy to understand and implement.

Author: Ing. Jan Novotný

Date updated: 21.10.2025

Documentation Source