11 Integrating external systems with Kafka Connect

Kafka Connect is presented as Kafka’s built-in, scalable, and fault-tolerant framework for integrating external systems, covering both source connectors that ingest into Kafka and sink connectors that export from Kafka. It treats external data as partitioned streams with offsets to ensure parallelism and progress tracking, aligning naturally with Kafka’s topic partitions. The chapter explains how Connect runs best in Distributed Mode—backed by Kafka topics for configs, offsets, and status—where multiple workers with the same group.id form a rebalance-driven cluster that spreads tasks for throughput and resilience. A lightweight Standalone Mode suits local development. Management happens through a JSON-based REST API that creates, inspects, pauses/resumes, and restarts connectors and their tasks.

Configuration is split between worker-level settings and per-connector configs. Workers define the Kafka cluster, internal topics, converter defaults (String/JSON/Avro/Protobuf), plugin path, listeners, and policies like auto topic creation and exactly-once source support; operational levers include timeouts, flush intervals, and producer/consumer overrides, with config providers to externalize secrets. Connector configs specify class, tasks.max, topic selection (explicit or regex), consumer group overrides for sinks, and whether exactly-once is required. Robust operations hinge on error-handling controls: skipping or logging malformed records, routing to a dead-letter topic, and retry/backoff for transient failures (plus connector-specific retry knobs like JDBC connection attempts). The chapter stresses naming conventions, recommended replication factor for internal topics, cautious use of the minimal FileStreamSource in production, and limiting human access to the REST API—favoring CI/CD automation.

The chapter illustrates Connect in practice: a simple file-to-topic example that tails appended lines and remembers offsets; a JDBC source that ingests relational tables using modes like incrementing, timestamp, or combined—paired with topic prefixes for clear ownership—while noting limitations (missed updates/deletes without suitable, monotonically increasing columns and the inefficiency of bulk mode). To capture changes reliably, it introduces CDC with Debezium for PostgreSQL, which reads transaction logs and emits events describing operations (create/update/delete), including before/after images and metadata; an SMT like ExtractNewRecordState can streamline payloads to just the latest state. Finally, Single Message Transformations (e.g., ReplaceField, ValueToKey + ExtractField, MaskField) let teams perform light, in-line shaping and routing of messages, with a clear recommendation to use purpose-built connectors over custom producers/consumers and to reserve complex transformations for Kafka Streams or similar frameworks.

Kafka Connect integrates third-party systems with Kafka.

A partitioned stream uses partitions for parallelization and offsets to remember the last read message.

Partitioned streams in databases: We can view tables as independent data streams and use, for example, sequential id-columns to track which rows have been already seen by Kafka Connect.

A Kafka Connect Cluster consists of one or multiple workers. Tasks are distributed across workers and Kafka Connect uses three topics to store its internal state.

Single Message Transformations can be used for both, source and sink connectors.

We want to move customer data from a relational database to Kafka

• Kafka Connect is a framework designed for scalable and reliable data integration between Kafka and other systems.
• It enables the movement of large amounts of data into and out of Kafka effortlessly using connectors.
• Connectors can be classified as source connectors (importing data into Kafka) or sink connectors (exporting data from Kafka).
• Each connector can be configured with various parameters to meet specific requirements.
• Kafka Connect uses a REST API for managing connectors and worker configurations, and can also be configured using a properties file at startup.
• The REST API can be used to create, update, and delete connectors, as well as to check their status.
• Worker configurations play a vital role in determining how Kafka Connect operates, including aspects like resource allocation, task distribution, and the overall resilience of the integration process.
• Error handling in Kafka Connect can be configured, including options for ignoring problematic entries via the errors.tolerance parameter.
• Logging errors is essential for troubleshooting, achieved through the errors.log.enable parameter.
• For sink connectors, data can be directed to a dead-letter queue for handling unprocessable records.
• Retry mechanisms can be configured for transient errors using parameters like errors.retry.timeout and errors.retry.delay.max.ms.
• JDBC connectors have specific configurations for managing database connection retries and backoff strategies.
• The operational mode of connectors can be set to either incrementing, timestamp-based, or both, depending on change tracking requirements.
• Change Data Capture (CDC) can be implemented using connectors like Debezium to monitor database changes.
• PostgreSQL requires specific configurations (e.g., setting wal_level to logical) for effective change data capture.
• Kafka Connect can automatically create Kafka topics based on database tables, ensuring seamless integration.
• The topic.prefix parameter organizes topics generated by connectors and prevents naming conflicts.
• Connector configurations can specify database connection details and table filtering using parameters like table.include.list.
• Single Message Transformations (SMTs) enable simple data modifications like renaming fields, masking data, or moving values to keys during data transfer.
• SMTs work for both source and sink connectors but aren't suited for complex ETL tasks.
• SMTs are set up via JSON in the Connect API, specifying transformation order, Java class types, and configurations. Common SMTs include ReplaceField, ValueToKey, ExtractField, and MaskField.
• The framework’s flexibility allows for tailored solutions for various data integration scenarios, enhancing data flow management.

FAQ