Articles

Migrate Data Across Kafka Cluster Using mirrormaker2 in Strimzi

,

In this article, we will discuss a use case where data from one kafka cluster has to be migrated to another Kafka Cluster. Here the target is strimzi and the source is a standalone Kafka cluster.  Target means where data has to be copied and the source is from where we want to copy/migrate data. I have an article on how to use mirrormaker with apache kafka clusters about mirrormaker version 1. This article is about mirrormaker 2, which has more features than mirrormaker1.

At the time of writing this article, the latest version of strimzi is 0.22.1 and can be downloaded from here.

Source de l’article sur DZONE

/par

Deploying an Apache Kafka mock service with Microcks

,

Developers are working with new applications every day using Apache Kafka as the backbone to implement an event-driven architecture (EDA) to support distributed systems. However, this adds new challenges when sharing across teams, even within the same organization. What endpoints are available? What is the structure of the message? That’s why payload examples became critical to speed up development. For this reason, having a reliable and enterprise-grade service to mock Apache Kafka should be an item in your EDA checklist. This post will do a quick review of the Microcks General Availability (GA) version and their support to Kafka.

What is Microcks?

Source de l’article sur DZONE

/par

How To Propagate Context Information Throw Spring Batch

,

Introduction

While developing applications using Spring batch, especially in a micro-service project, we sometimes face one or most of the following cases:

  • The necessity of getting the security context inside the batch items to call methods that require authorizations inside the same micro-service or perform remote processing by calling other micro-services using Feign Client (HTTP) or  Spring Cloud Stream (broker like Kafka, RabbitMq …)
  • Propagating Sleuth trace Id and span Id in order to enhance logs traceability inside all the application components including other micro-services so the trace will not be lost if we use Job.
  • Getting the connected user Locale (i18n) in order to generate internationalized output otherwise, all the Job outputs will be generated in the default server language.
  • Retrieving objects stored inside Mapped Diagnostic Context  (MDC) for tracing purposes.

The following schema illustrates remote calls that can be performed in a micro-service-based application and the context information that String Batch items can propagate.

Source de l’article sur DZONE

/par

Integrating Red Hat Single Sign-On With AMQ Streams for Auditing Events

,

Here I am again with another take from the field.

Red Hat Single Sign-on (RH-SSO) is the enterprise-ready version of Keycloak, and one thing that is most commonly asked, especially for big customers is, « How do we audit all the events? »

Source de l’article sur DZONE

/par

App Modernization + Hybrid Cloud Architectures With Apache Kafka [Video]

,

Hybrid cloud architectures are the new black for most companies. A cloud-first is obvious for many, but legacy infrastructure must be maintained, integrated, and (maybe) replaced over time. Event Streaming with the Apache Kafka ecosystem is a perfect technology for building hybrid replication in real-time at scale.

App Modernization and Streaming Replication With Apache Kafka at Bayer

Most enterprises require a reliable and scalable integration between legacy systems such as IBM Mainframe, Oracle, SAP ERP, and modern cloud-native applications like Snowflake, MongoDB Atlas, or AWS Lambda.

Source de l’article sur DZONE

/par

Apache Camel Integration with Kafka

,

This article covers Apache Camel Integration with Kafka.

Setup:

Kafka Setup

Source de l’article sur DZONE

/par

Streaming Data From Files Into Multi-Broker Kafka Clusters

,

There are multiple ways to ingest data streams into the Apache Kafka topic and subsequently deliver to various types of consumers who are hooked to the topic. The stream of data that collects continuously from the topic by consumers, passes through multiple data pipelines and then stream processing engines like Apache Spark, Apache Flink, Amazon Kinesis, etc and eventually landed upon the real-time applications to deliver a final data-driven decision. From finances, manufacturing, insurance, telecom, healthcare, commerce, and more, real-time applications are becoming the best solution for organizations to take immediate action, gain insights from the updated data. In the present day, Apache Kafka shapes the central nervous system that brings data from all aspects of the business to the large information operational hubs where choices are made.

The text files contain unformatted ASCII text and are commonly used for the storage of information. Each line of the file represents a data record and can be updated continuously to store. Every insert of a new line or lines on the text file can be considered as new data insertion on the file. Henceforth, every addition of a new line or lines on the text file continuously either by humans or applications (no modification on the already inserted line)and subsequently moves or sends to a different location can be considered as data streaming from the file. Every addition of a new line or row in the text file can be analyzed continuously by exporting the new line/lines to the Kafka topic and importing them by consumers that hooks up with the topic.

Source de l’article sur DZONE

/par

Implementing MongoDB to Elastic Search 7.X Data Pipeline

,

In this article, we will see how to implement a data pipeline from an application to Mongo DB database and from there into an Elastic Search keeping the same document ID using Kafka connect in a Microservice Architecture. In recent days and years, all the microservices architectures are asynchronous in nature and are very loosely coupled. At the same time, the prime approach to have minimum code (minimum maintenance and cost), no batch systems (real-time data), and promising performance without data loss fear. Keeping all the features in mind Kafka and Kafka connect is the best solution so far to integrate different sources and sinks in one architecture to have very robust and reliable results.

We will Depp drive and implement such a solution using Debezium Kafka connect to achieve a very robust pipeline of data from one application into Mongo and then into Elastic cluster.

Source de l’article sur DZONE

/par

Change Data Capture From PostgreSQL to Azure Data Explorer Using Kafka Connect

,

This blog post demonstrates how you can use Change Data Capture to stream database modifications from PostgreSQL to Azure Data Explorer (Kusto) using Apache Kafka.

Change Data Capture (CDC) can be used to track row-level changes in database tables in response to create, update and delete operations. It is a powerful technique, but useful only when there is a way to leverage these events and make them available to other services.

Source de l’article sur DZONE

/par

Ports and Adapters Architecture With Kafka, Avro, and Spring-Boot

,

In this post, we will be implementing a Kafka Producer and Consumer using the Ports and Adapters (a.k.a. Hexagonal) architecture. We will also be using the KafkaAvroSerializer to send specific Avro types using Kafka and the Kafka Schema Registry.

Overall Flow

The overall workflow is something like this: a domain object (Person.java) is sent from BusinessDomainService to Kafka where it gets serialized to Avro object (PersonDto.java); which is then consumed from the Kafka topic via a Kafka Consumer and translated from Avro object (PersonDto.java) back to domain object (Person.java) before being sent back to BusinessDomainService for some arbitrary post-processing.

Source de l’article sur DZONE

/par