Bulk vs. incremental indexing

evitaDB is designed as a fast, transactional, read-optimized database that offloads work from the primary data store, which is usually some kind of relational database. It is therefore expected to operate in two distinct phases: initial indexing of a large dataset and then maintaining the index throughout its lifetime. These two phases have different requirements and, as such, receive special treatment.

Bulk indexing (WARM-UP phase)

Bulk indexing is used for rapid indexing of large volumes of source data from an external data store. At this initial stage of the catalog's lifecycle, we don't require transaction support or concurrency. The only goal is to index as much data as possible in the shortest time possible. This phase has the following characteristics:

Only a single client (single session) can be open at a time.
There are no transactions - a group of writes cannot be committed or discarded as a unit. A single entity write is still atomic on its own, so an error part-way through one leaves nothing behind (see Atomicity of individual writes).
All changes to indexes are kept in memory and written when the session closes; in case of a database crash, everything written since the last session close is lost.
A failure that cannot be reverted returns the catalog to the last state it published and makes it inactive (see Failures that cannot be reverted).

How much data you write between session closes is a deliberate trade-off. Closing the session is the only moment when index changes reach the disk, so frequent closes act as checkpoints — the work completed so far is durable, and a crash costs you at most the block still in progress. You pay for that in throughput: every close has to collect and persist the modified parts of each index the block touched, and the more data the catalog already holds, the more that costs — so the price is paid repeatedly and rises as the import progresses. Writing the whole dataset within a single session avoids almost all of that cost and gives the fastest possible import, but it keeps the entire result in flight: nothing is durable until the end, the memory held by the pending index changes grows for the whole duration, and anything that loses the session — a crash, an out-of-memory condition — puts you back at the beginning. A single rejected entity write is not such a failure: it is reverted on its own and the import simply continues (see Atomicity of individual writes). Prefer one large block for imports short enough to simply repeat, and periodic closes for imports long enough that losing all the work would hurt.

Once initial indexing is finished, the client is expected to finalize the warm-up phase by closing the session and executing the MakeCatalogAlive mutation, which transitions the catalog to the ALIVE phase (see next chapter).

Incremental indexing (ALIVE phase)

Incremental indexing is the phase in which we continuously synchronize changes from the primary data store into evitaDB. Multiple clients (sessions) may be open concurrently, some reading and some writing. Each read-write session defines a transaction boundary, and changes can be committed or rolled back atomically (see the chapter about transactions for ACID details). Write performance is considerably lower than in the bulk indexing phase because there is a cost associated with maintaining transactional integrity, concurrency, and durability. Read performance is not affected and remains very high.

Atomicity of individual writes

A single write — an upsertEntity or deleteEntity call together with all the index changes it implies (attributes, references, facets, prices, hierarchy placement, reflected references) — is treated as one unit of work. That unit is atomic, and it behaves the same way in both phases.

If applying an entity mutation fails part-way through — for example because it violates a unique constraint or another consistency rule after some of its index entries have already been written — the engine reverts exactly that entity's partial changes. The index entries already written for it are removed, any unique value it reserved becomes available again, and its stored body goes back to what it was before the call. Nothing half-applied is left behind, such as an orphaned facet or a phantom price.

The failing call throws an exception and the session stays usable. You may catch it, skip or retry the offending entity, and continue writing the rest of your data. Neither compensating on the client side nor rebuilding the catalog is needed because of a single rejected entity.

In the ALIVE phase the enclosing transaction is untouched by the failure: every entity written before the failing one remains valid, and you may commit afterwards — the commit publishes exactly the entities that succeeded. Rolling back still discards everything, as usual.

One thing is deliberately not rewound: the primary key drawn for a failed entity is not returned to the pool. Primary key sequences guarantee uniqueness, not contiguity, so a reverted write leaves a harmless gap in the numbering.

Failures that cannot be reverted

A single failed write is reverted on its own and costs you nothing beyond that one entity. Some failures reach further than one entity, and the bulk indexing phase answers all of them the same way — by returning the catalog to the last state it published.

Three failures behave like this:

A schema change refused by validation. The catalog schema is validated as a whole when a session closes, because a schema is routinely built across several steps whose intermediate states are allowed not to validate — a reflected reference may be declared before the reference it reflects, for instance. By the time the refusal is known, the change has already been applied to every entity collection it touches.
A failure while writing the collected changes at session close.
A failed revert of a single entity write, where the revert itself throws.

In each case the catalog stops accepting writes and stops publishing. That refusal is what protects the data on disk, and it holds from the moment the failure is detected. The engine then moves the catalog to the inactive state (see Control Engine).

Activating it again loads

the last published state from disk — the newest state that reached the disk, which a session close writes and a collection-level schema operation may write again mid-session. Its schema and its indexes are consistent with each other, because nothing is published unless the schema validates. Everything written after it has to be replayed.

Moving a catalog to the inactive state is itself an engine-level operation, and one such operation is refused while another is in flight for the same catalog. Where the engine cannot complete the move, the catalog is left refusing every write and every publication, and restarting the engine loads the same published state that activating it would have. The server log says which of the two happened.

Why the recovery is this coarse

The bulk indexing phase earns its speed by leaving out the machinery a narrower recovery would need.

There is no transaction to roll back. A schema change is applied to each entity collection separately, together with the structural work it implies — root nodes for an entity that becomes hierarchical, reference indexes, capability registries — and only then is the catalog validated as a whole. Sending a corrective schema change afterwards repeats that work on top of what already ran rather than reversing it, and nothing guarantees the result matches what a clean path would have produced. Reloading the last published state is the only recovery that is consistent by construction.

It also costs less than it may appear. Nothing written since the last session close was durable in the first place, so what the reload discards is exactly the work you had not yet checkpointed. How large that is, is the trade-off described at the top of this chapter: frequent session closes make each failure cheap and the import slower, while a single long session makes the import as fast as possible and each failure expensive.

Full reindex of the live catalog

There are situations when you need to reindex the entire catalog from the primary data store while still serving live traffic from up-to-date data. The recommended approach is to create a new temporary catalog and fill it with an initial set of data using bulk indexing. Once the new catalog is fully indexed, you can switch your application to the new catalog using the replace catalog operation. Replacing the catalog is a very fast operation that does not require copying any data - it updates the catalog name in the schema and renames a few files on disk. Even though the operation is quick, sessions using the old catalog will be closed during the process, and attempts to open new sessions will wait until the operation finishes. The switch is not entirely without impact, but the impact is very short-lived. The old catalog is deleted during the process; if you want to keep it, back it up before executing the replace operation.

Author: Ing. Jan Novotný

Date updated: 24.8.2028

Documentation Source