Confluent Kafka Series: JDBC Source Connector | by Manish Sharma | Medium


The Confluent Kafka JDBC Source connector facilitates importing data from relational databases into Apache Kafka topics, offering features like incremental data loading and dynamic table handling.
AI Summary available — skim the key points instantly. Show AI Generated Summary
Show AI Generated Summary

Confluent Kafka Series: JDBC Source Connector

The Kafka Connect JDBC Source connector allows you to import data from any relational database with a JDBC driver into an Apache Kafka® topic.

This connector can support a wide variety of databases.

  • Data is loaded by periodically executing a SQL query and creating an output record for each row in the result set.
  • By default, all tables in a database are copied, each to its own output topic.
  • The database is monitored for new or deleted tables and adapts automatically.
  • When copying data from a table, the connector can load only new or modified rows by specifying which columns should be used to detect new or modified data.

Features

  • The source connector supports copying tables with a variety of JDBC data types, adding and removing tables from the database dynamically, whitelists and blacklists, varying polling intervals, and other settings.
  • However, the most important features for most users are the settings controlling how data is incrementally copied from the database.
  • Kafka Connect tracks the latest record it retrieved from each table, so it can start in the correct location on the next iteration (or in case of a crash). The source connector uses this…

Was this article displayed correctly? Not happy with what you see?

Tabs Reminder: Tabs piling up in your browser? Set a reminder for them, close them and get notified at the right time.

Try our Chrome extension today!


Share this article with your
friends and colleagues.
Earn points from views and
referrals who sign up.
Learn more

Facebook

Save articles to reading lists
and access them on any device


Share this article with your
friends and colleagues.
Earn points from views and
referrals who sign up.
Learn more

Facebook

Save articles to reading lists
and access them on any device