Articles

Building a Data Pipeline Using QuestDB and Confluent Kafka

,

A data pipeline, at its base, is a series of data processing measures that are used to automate the transport and transformation of data between systems or data stores. Data pipelines can be used for a wide range of use cases in a business, including aggregating data on customers for recommendation purposes or customer relationship management, combining and transforming data from multiple sources, as well as collating/streaming real-time data from sensors or transactions.

For example, a company like Airbnb could have data pipelines that go back and forth between their application and their platform of choice to improve customer service. Netflix utilizes a recommendation data pipeline that automates the data science steps for generating movie and series recommendations. Also, depending on the rate at which it updates, a batch or streaming data pipeline can be used to generate and update the data used in an analytics dashboard for stakeholders.

Source de l’article sur DZONE

/par

Generating Simulated Streaming Data

,

For demos, system tests, and other purposes, it is good to have a way to easily produce realistic data at scale utilizing a schema of our own choice.

Fortunately, there is a great library for Python called Faker that lets us build synthetic data for tests. With a simple loop and a Pulsar produce call, we can send messages to topics at scale.

Source de l’article sur DZONE

/par

Transformations of Varying JSON Payloads Using Spark-Streaming

,

Spark-streaming can be used to read the data from a source in a streaming fashion. We just have to create a read-stream from the data source and then we can create the write-stream to load the data into a target datasource. 

For this demo, I will assume that we have different JSON payloads coming into a kafka topic that we need to transform and write it to another kafka topic.

Source de l’article sur DZONE

/par

Streaming Data From Files Into Multi-Broker Kafka Clusters

,

There are multiple ways to ingest data streams into the Apache Kafka topic and subsequently deliver to various types of consumers who are hooked to the topic. The stream of data that collects continuously from the topic by consumers, passes through multiple data pipelines and then stream processing engines like Apache Spark, Apache Flink, Amazon Kinesis, etc and eventually landed upon the real-time applications to deliver a final data-driven decision. From finances, manufacturing, insurance, telecom, healthcare, commerce, and more, real-time applications are becoming the best solution for organizations to take immediate action, gain insights from the updated data. In the present day, Apache Kafka shapes the central nervous system that brings data from all aspects of the business to the large information operational hubs where choices are made.

The text files contain unformatted ASCII text and are commonly used for the storage of information. Each line of the file represents a data record and can be updated continuously to store. Every insert of a new line or lines on the text file can be considered as new data insertion on the file. Henceforth, every addition of a new line or lines on the text file continuously either by humans or applications (no modification on the already inserted line)and subsequently moves or sends to a different location can be considered as data streaming from the file. Every addition of a new line or row in the text file can be analyzed continuously by exporting the new line/lines to the Kafka topic and importing them by consumers that hooks up with the topic.

Source de l’article sur DZONE

/par

Big / Bug Data: Analyzing the Apache Flink Source Code

,

Applications used in the field of Big Data process huge amounts of information, and this often happens in real time. Naturally, such applications must be highly reliable so that no error in the code can interfere with data processing. To achieve high reliability, one needs to keep a wary eye on the code quality of projects developed for this area. The PVS-Studio static analyzer is one of the solutions to this problem. Today, the Apache Flink project developed by the Apache Software Foundation, one of the leaders in the Big Data software market, was chosen as a test subject for the analyzer.

So, what is Apache Flink? It is an open-source framework for distributed processing of large amounts of data. It was developed as an alternative to Hadoop MapReduce in 2010 at the Technical University of Berlin. The framework is based on the distributed execution engine for batch and streaming data processing applications. This engine is written in Java and Scala. Today, Apache Flink can be used in projects written using Java, Scala, Python, and even SQL.

Source de l’article sur DZONE

/par

Streaming ETL With Apache Flink — Part 3

,

Previous posts – Part 1  |  Part 2.

Introduction

In this post, let us start exploring Flink to solve a real-world problem. This post from zalando.com shows how they are using Flink to perform a complex event correlation. I will take a simplified and practical event correlation problem and try to solve using Flink.

Source de l’article sur DZONE

/par

Big Data: Volume, Variety, and Velocity

,

Big data is the new competitive advantage and it is necessary for businesses. With the growing proliferation of data sources such as smart devices, vehicles, and applications, the need to process this data in real-time and to deliver relevant insights is more urgent than ever. The 2019 Guide to Big Data will explore tools and ecosystems for analyzing big data and relevant use cases ranging from sustainability science to autonomous vehicles.
Source de l’article sur DZONE

/0 Commentaires/par