Confluent Kafka Series: JDBC Source Connector

The Kafka Connect JDBC Source connector allows you to import data from any relational database with a JDBC driver into an Apache Kafka® topic.

This connector can support a wide variety of databases.

  • Data is loaded by periodically executing a SQL query and creating an output record for each row in the result set.
  • By default, all tables in a database are copied, each to its own output topic.
  • The database is monitored for new or deleted tables and adapts automatically.
  • When copying data from a table, the connector can load only new or modified rows by specifying which columns should be used to detect new or modified data.

Features

  • The source connector supports copying tables with a variety of JDBC data types, adding and removing tables from the database dynamically, whitelists and blacklists, varying polling intervals, and other settings.
  • However, the most important features for most users are the settings controlling how data is incrementally copied from the database.
  • Kafka Connect tracks the latest record it retrieved from each table, so it can start in the correct location on the next iteration (or in case of a crash). The source connector uses this…

Confluent Kafka Series: JDBC Source Connector | by Manish Sharma | Medium


Click on the Run Some AI Magic button and choose an AI action to run on this article