Confluent Kafka Series: JDBC Source Connector | by Manish Sharma | Medium

AI Summary Hide AI Generated Summary

Key Features of the Confluent Kafka JDBC Source Connector

The Confluent Kafka JDBC Source connector enables seamless data transfer from relational databases (supporting various JDBC drivers) into Apache Kafka topics. Its capabilities include:

Periodic SQL query execution to generate records from result sets.
Default behavior of copying all database tables into individual topics.
Automatic adaptation to new or deleted tables.
Incremental data loading by focusing on new or modified rows based on specified columns.

Data Handling and Advanced Settings

The connector supports various JDBC data types and offers flexibility in managing database interactions. Crucially, it allows for:

Whitelisting and blacklisting tables.
Customizable polling intervals.
Efficient incremental copying by tracking the latest retrieved record from each table. This ensures efficient restarts after failures.

Confluent Kafka Series: JDBC Source Connector

The Kafka Connect JDBC Source connector allows you to import data from any relational database with a JDBC driver into an Apache Kafka® topic.

This connector can support a wide variety of databases.

Data is loaded by periodically executing a SQL query and creating an output record for each row in the result set.
By default, all tables in a database are copied, each to its own output topic.
The database is monitored for new or deleted tables and adapts automatically.
When copying data from a table, the connector can load only new or modified rows by specifying which columns should be used to detect new or modified data.

Features

The source connector supports copying tables with a variety of JDBC data types, adding and removing tables from the database dynamically, whitelists and blacklists, varying polling intervals, and other settings.
However, the most important features for most users are the settings controlling how data is incrementally copied from the database.
Kafka Connect tracks the latest record it retrieved from each table, so it can start in the correct location on the next iteration (or in case of a crash). The source connector uses this…