Change Data Capture (CDC) identifies inserts, updates and deletes in a database so those changes can be passed to another system. Instead of copying a whole table every time, a CDC pipeline moves the changes.
A warehouse can use it to stay up to date with an operational database, a search index can reflect changes to product details, or another application can respond when a record changes.
How Change Data Capture works
If the destination needs records that already exist, the pipeline usually starts with an initial copy, often called a snapshot. It then captures ongoing changes and sends them to the destination through a connector or ingestion service. The initial copy and ongoing capture need to connect properly so that changes made during the copy are not missed.
Each captured change identifies the affected record, the operation and the information needed to apply it. The destination uses that information to add, update or remove data according to how it stores records. Capturing a change is only part of the process: the destination still needs to receive and apply it before an application or report can use it.
The pipeline also records its progress so it can resume after an interruption, with the destination handling any repeated changes safely.
A simple insert, update and delete example
Imagine a destination table that holds the current state of each order. As order 103 changes in the source, the pipeline should produce the following result:
| Source change | What CDC captures | Destination result |
|---|---|---|
| Create order 103 with status Pending | An insert for order 103 | Add one row for the new order |
| Change its status to Completed | An update for order 103 | Change the existing row to Completed |
| Delete order 103 | A delete for order 103 | Remove it from the current-order view |
The order ID connects each change to the right record, so an update changes the existing order rather than creating another one. A deletion needs to be handled deliberately too, though the destination might mark the row as deleted rather than remove it.
A table holding the current state tells you what the order looks like now. If you also need to know how it changed over time, keep that history as well, according to your reporting and retention requirements.
When CDC is worth using
CDC is useful when another system needs changes sooner than your existing loads can provide them, or when full reloads spend too much time copying unchanged data. For example, repeatedly copying a large customer table may be unnecessary if only a small number of records change between loads. Capturing those changes can reduce repeated copying, although the overall benefit depends on the database, capture method and amount of change.
It is also worth considering when updates and deletes need to reach other systems reliably, when several destinations need the same changes, or when you want to retain a change history. These needs can justify CDC even if rapid updates are not the main goal.
A simpler scheduled load may be enough for a small table or a report that only needs yesterday’s data. Reloading the table in full may be easy to manage, while a timestamp-based load can work if you only need the latest values and have a way to handle deletes.
CDC and scheduling are not opposites. CDC describes how changes are identified; those changes can still be collected or applied in batches. A daily pipeline may use CDC to capture deletes or avoid an expensive full reload, without needing immediate updates.
The main ways to capture changes
Log-based CDC
Log-based CDC reads the database’s transaction log, where changes are recorded. With the right configuration, a connector can capture inserts, updates and deletes without repeatedly querying the source table to find what changed.
This can suit large or frequently changing tables, but it needs supported database features, permission to read the log and enough log retention to recover after an interruption. It still uses resources, so you need to check its effect on the source database.
Trigger-based capture
A database trigger runs when a specified insert, update or delete happens. It can write the change details to a separate table for the pipeline to read. This is an option when log access is unavailable, though it adds work to database writes and leaves triggers and change tables to maintain as the source changes.
Timestamp-based incremental loading
A simpler approach queries a column such as updated_at to find rows changed since the previous load. It is often considered alongside CDC, but it does not necessarily record every change. The timestamp must be updated reliably, and the load boundaries need to account for rows with the same timestamp or changes that become visible late.
If a row changes several times between loads, the query may see only its latest value. A physically deleted row will not be there to find at all, so deletes need another mechanism. This approach can work well when those limits fit what the destination needs.
What to check before choosing a tool
Start with support for your database and the changes you need to capture. Check how the tool handles added or renamed columns, type changes and interruptions. Find out how long change records remain available, because falling behind that limit may require a fresh copy rather than a simple restart.
Test how the destination handles retries and the sequence of updates. Receiving the same change again should not create an unwanted duplicate, and an older update should not overwrite a newer value. Someone also needs to monitor failures, storage, source impact and the freshness of the final output.
CDC carries source data problems downstream too. It will not fix an invalid amount or resolve conflicting business definitions, so data checks are still necessary.
Start with the changes you actually need
Choose one source table and one destination. Decide how quickly changes must arrive, whether you need the latest state or a history, and how deletes should appear. Test an insert, an update, a delete and a retry before expanding. That gives you a practical basis for deciding whether CDC solves your problem or a simpler load is enough.
If you are weighing those options, Insights Titan can help you compare them against your existing systems and how the data will be used.
