Articles

Building a Data Pipeline Using QuestDB and Confluent Kafka

,

A data pipeline, at its base, is a series of data processing measures that are used to automate the transport and transformation of data between systems or data stores. Data pipelines can be used for a wide range of use cases in a business, including aggregating data on customers for recommendation purposes or customer relationship management, combining and transforming data from multiple sources, as well as collating/streaming real-time data from sensors or transactions.

For example, a company like Airbnb could have data pipelines that go back and forth between their application and their platform of choice to improve customer service. Netflix utilizes a recommendation data pipeline that automates the data science steps for generating movie and series recommendations. Also, depending on the rate at which it updates, a batch or streaming data pipeline can be used to generate and update the data used in an analytics dashboard for stakeholders.

Source de l’article sur DZONE

/par

Open API and Omnichannel with Apache Kafka in Healthcare

,

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Open API and Omnichannel. Examples include Care.com and Invitae.

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Source de l’article sur DZONE

/par

Machine Learning and Data Science With Kafka in Healthcare

,

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Machine Learning and Data Science. Examples include Recursion and Humana.

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Source de l’article sur DZONE

/par

Bring Streaming to Apache Cassandra with Apache Pulsar

,

Twitch, YouTube, Instagram, Facebook — virtually every major brand nowadays uses live streaming to connect and engage their audience. For enterprises and developers building cloud-native applications, this growing trend creates a need for streaming technologies that can reliably handle the rush of massive amounts of data, while also being flexible and easy to manage for developers.

One such technology is Apache Pulsar® — an open-source, distributed messaging and streaming platform that’s easy to deploy, simple to scale, and packed with developer-friendly APIs. So the next question is: how can you stream from Pulsar to Apache Cassandra®, the powerful NoSQL database designed to support data-heavy applications in the cloud?

Join our beginner-friendly Pulsar workshop on YouTube and learn how to connect Pulsar with Cassandra for streaming! In this post, we’ll set the scene with an introduction to Pulsar and guide you through four hands-on exercises where you’ll use these free, cloud-native technologies: Katacoda, Kesque, GitPod, and DataStax Astra DB. Each exercise will also be linked to the step-by-step instructions on the DataStax Developers GitHub wiki.

Source de l’article sur DZONE

/par

Kafka vs. JMS: Which One Should You Be Using?

,

Two of the most popular message brokers used today are Kafka and those based around JMS. JMS is a long-standing Java API used generally for developing messaging applications, with its primary function of being able to send messages between two or more clients. Kafka, on the other hand, is a distributed streaming platform that provides a lot of scalabilities and is useful for real-time data processing. 

While both offer their own advantages and are highly useful in their own right, which of the two should you be actually using?

Source de l’article sur DZONE

/par

The Definitive Guide to Building a Data Mesh With Event Streams

,

Data mesh. This oft-talked-about architecture has no shortage of blog posts, conference talks, podcasts, and discussions. One thing that you may have found lacking is a concrete guide on precisely how to get started building your own data mesh implementation. We have you covered. In this blog post, we’ll show you how to build a data mesh using event streams, highlighting our design decisions, and the key benefits and challenges you’ll need to consider along the way. In fact, we’ll go one better: we’ve built a data mesh prototype for you to check out on your own to see what this would look like in action, or fork to bootstrap a data mesh for your own organization. 

Data mesh is technology agnostic so there are a few different ways you can go about building one. The canonical approach is to build the mesh using event streaming technology that provides a secure, governed, real-time mechanism for moving data between different points in the mesh. 

Source de l’article sur DZONE

/par

Exciting New Tools for Designers, May 2022

,

The underlying theme of this month’s collection of new tools and resources is development. Almost every tool here makes dev a little easier, quicker, or plain fun. There are a few great tutorials in the mix to help you get into the spirit of trying new things and techniques.

Here’s what is new for designers this month…

Cryptofonts

Cryptofonts is a huge open-source library of icons that represent cryptocurrencies. There are more than 1,500 CSS and SVG elements in the collection. Cryptofonts includes all scalable vector icons that you can customize by size, color, shadow, or practically anything else. They work with Sketch, Photoshop, Illustrator, Adobe XD, Figma, and Invision Studio, and there’s no JavaScript.

 

Reasonable Colors

Reasonable Colors is an open-source color system for building accessible and beautiful color palettes. Colors are built using a coded chart. Each color comes in six numbered shades. The difference between their shade numbers can infer the contrast between any two shades. The differences correspond to WCAG contrast ratios to help you create an accessible palette. This is a smart project and a valuable tool if you work on projects where color contrast and accessibility are essential (which is all of them).

 

Chalk.ist

Chalk.ist is a fun tool to make your code snippets look amazing. Add your code (there’s a vast language selector), pick some colors and backgrounds, and then download it as a shareable image. Your code has never looked so beautiful!

 

WeekToDo

WeekToDo is a free minimalist weekly planner. Improve productivity by defining and managing your week and life easily and intuitively. Plus, this tool is focused on privacy with data that is stored on your computer (in your web browser or the application). The only person who has access to it is you.

 

Bio.Link

Bio.Link is a tool that collects all your links – from social media to blog posts to any other kind of link you want to share. It’s free to use, includes 15 design themes, visitor stats, and is super fast.

 

Spacers

Spacers are a set of three-dimensional space characters that you can use in projects. Characters are in multiple poses and ultra high-def formats to play with.

11ty

11ty is a super simple, static website generator. Try it for small projects and read the documentation to see everything you can do with this tool.

Scrollex

Scrollex is a react library that lets you build beautiful scroll experiences using minimal code. You can create scroll animations in all kinds of combinations – vertical, horizontal, almost anything you want to try. The documentation is fun and easy to understand if you’re going to see how it works.

GetCam

GetCam is an app that lets you turn your smartphone into a webcam for your computer. It works with any iPhone and a Mac or Windows computer. It works with most video conference and streaming tools as well as browser-based apps.

Flatfile

Flatfile is a data onboarding platform that intuitively makes sense of the jumbled data customers import and transforms it into the format you rely on. You won’t have any more messy spreadsheets or have to build a custom tool.

Loaders

Loaders is a collection of free loaders and spinners for web projects. They are built with HTML, CSS, and SVG and are available for React and copypasta.

Lexical

Lexical is an extensible JavaScript web text-editor framework emphasizing reliability, accessibility, and performance. It’s made for developers, so you can easily prototype and build features with confidence. Combined with a highly extensible architecture, Lexical allows developers to create unique text editing experiences that scale in size and functionality.

Picture Perfect Images with the Modern img Element

This tutorial is a primer on why the img element is such a powerful tool in your development box. Images are so prominent that they are part of the most important content in over 70% of pages on both mobile and desktop, according to the largest contentful paint metric. This post takes you through how to better optimize and improve core web vitals simultaneously.

Building a Combined CSS-Aspect-Ratio-Grid

Building a Combined CSS-Aspect-Ratio-Grid provides two solutions for creating the title effect. You can define an aspect ratio for the row or use Flexbox with a little flex grow magic. Learn how to try it both ways.

QIndR

QIndR is a QR code generator made for events and appointments. The form is designed to capture your event information so you can quickly build and use a QR code for listings and even allow users to add it to their calendars! It’s super quick and easy to use.

On-Scroll Text Repetition Animation

On-Scroll Text Repetition Animation shows you how to create an on-scroll animation that shows repeated fragments of a big text element. This is a fun and easy lesson that you can use right away.

Eight Colors

Eight Colors won’t do anything for your productivity, but it is a fun game that you may not be able to stop playing. It is a block-shifting game with the goal to shift circular blocks to reach the target given.

Creative Vintage

Creative Vintage is a pair of typefaces including a thin script and vintage slab serif (with rough and smooth styles). The pair is designed to work together for various uses or can be used independently.

Hardbop

Hardbop is a vintage-style typeface with a lot of personality. It would work great for display, and the family includes seven full-style character sets.

Kocha

Kocha is a funky ligature-style typeface perfect for lighter design elements, including logos or packaging. It includes clean and rough versions.

Magnify

Magnify is a large font family with 16 styles and plenty of fun alternates. You can use it straight or with the more funky styles that create less traditional character forms.

Stacker

Stacker is a fun and futuristic style font with a triple outline style. Use it for display when you really want to make an impression.

Source

The post Exciting New Tools for Designers, May 2022 first appeared on Webdesigner Depot.

Source de l’article sur Webdesignerdepot

/par

Next-Gen Data Pipes With Spark, Kafka, and K8s: Part 2

,

Introduction 

In our previous article, we discussed two emerging options for building new-age data pipes using stream processing. One option leverages Apache Spark for stream processing and the other makes use of a Kafka-Kubernetes combination of any cloud platform for distributed computing. The first approach is reasonably popular, and a lot has already been written about it. However, the second option is catching up in the market as that is far less complex to set up and easier to maintain. Also, data-on-the-cloud is a natural outcome of the technological drivers that are prevailing in the market. So, this article will focus on the second approach to see how it can be implemented in different cloud environments.

Kafka-K8s Streaming Approach in Cloud

In this approach, if the number of partitions in the Kafka topic matches with the replication factor of the pods in the Kubernetes cluster, then the pods together form a consumer group and ensure all the advantages of distributed computing. It can be well depicted through the below equation:

Source de l’article sur DZONE

/par

Generating Simulated Streaming Data

,

For demos, system tests, and other purposes, it is good to have a way to easily produce realistic data at scale utilizing a schema of our own choice.

Fortunately, there is a great library for Python called Faker that lets us build synthetic data for tests. With a simple loop and a Pulsar produce call, we can send messages to topics at scale.

Source de l’article sur DZONE

/par

Apache Kafka in Cybersecurity for SIEM/SOAR Modernization

,

Apache Kafka became the de facto standard for processing data in motion across enterprises and industries. Cybersecurity is a key success factor across all use cases. Kafka is not just used as a backbone and source of truth for data. It also monitors, correlates, and proactively acts on events from real-time and batch data sources to detect anomalies and respond to incidents. This blog series explores use cases and architectures for Kafka in the cybersecurity space, including situational awareness, threat intelligence, forensics, air-gapped and zero trust environments, and SIEM/SOAR modernization. This post is part six: SIEM/SOAR Modernization.

Blog Series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

Source de l’article sur DZONE

/par