This blog post demonstrates how you can use Change Data Capture to stream database modifications from PostgreSQL to Azure Data Explorer (Kusto) using Apache Kafka.
Change Data Capture (CDC) can be used to track row-level changes in database tables in response to create, update and delete operations. It is a powerful technique, but useful only when there is a way to leverage these events and make them available to other services.
In a previous post, I showed how to automate a PostgreSQL fault-tolerant cluster with Vagrant and Ansible.
This kind of setup makes our database cluster resilient to server failure and keeps the data available with no need for human interaction. But what about the apps using this database? Are they fault-tolerant too?
It’s fun to see new website design tools that reflect current times and the state of the world. That’s very true this month with new databases devoted to diversity and women in technology, as well and resources to make your design life easier.
Here’s what’s new for designers and developers this month:
Ztext.js is an easy to implement, three-dimensional typography tool for the web that works with any font you want to use. With the popularity of 3D effects and animation, this tool has a lot of practical applications. Everything you need, including documentation, is available from developer Bennett Feely on his website and GitHub. (It’s free but you can show appreciation with a donation if you like it.)
Gradient Magic is a free gallery of fun and interesting CSS gradients. You can sort through a random selection or by category of color to find just the right gradient for your project. Some of them would make really neat backgrounds or image overlays.
Impossible Checkbox is a fun little divot that you’ll want to play with and emulate. Click or tap the slider to activate and a nifty little friend pops up. Now here’s the fun part: You can’t leave it checked, and take note of the changing expression of the checkbox character.
Diversify Tech isn’t your average job board; it is a collection of resources – and opportunities – for underrepresented people in technology. It includes a weekly roundup and everything from scholarships, to events, to jobs, to speaking opportunities.
Women in Tech is a list of apps made by women. The apps are ranked and chosen based on upvotes and is a good resource if you want to help support women-owned projects. Search or submit an app for inclusion.
Devello Studio is a tool that allows you to write code in the cloud. You don’t have to install anything and no matter where you are, just can open a project in-browser, and continue development where you had left off last time. Plus, it works with GitHub support built right in.
Hustl is a premium Mac app that allows you to create time-lapse videos of your screen. Use it to show off work or projects or create a cool video for your portfolio. Plus you can use it to capture just one active app so you don’t have to do a lot of editing later.
FeedBaxley is a user feedback tool that helps you (and users) figure out what’s frustrating before it becomes a real issue. You can customize everything to match your brand and set it up with copy and paste tools. Feedback integrates with Slack, making it easy for you to analyze information with a team.
BestTime launched a major update with a new tool that makes it possible to analyze visitor peaks of public business (cafe, gym, etc) for whole areas. Using the heatmap API you can find businesses at popular times, locations, or by business type.
Pixeltrue is a new collection of free SVG illustrations and Lottie animations in a trendy style. They are available for commercial and personal use and add a bit of whimsical delight to website projects. (The error illustrations are particularly fun.)
Previewed has tons of cool and realistic mockups that you can use to create the perfect setting for digital projects. You can find mockups for a variety of devices and cool panoramas that work perfectly for elements such as app store previews.
The Alt Text Overlay Bookmarklet solves a common problem: It shows what images use alt text and what that text is. The tool was created by Christian Heilmann and he’s put it on GitHub for you to play with and test.
MergeURL allows you to merge and shorten up to five links. Enter the links and mergeurl.com/o/xxxxx, for example, will open all the URLs associated with that link. The tool is free to use and you don’t have to register to use the service.
Infinity Search is a new search engine that lets you look for things privately and efficiently. Search the web, images, or videos. Here’s a little about how it works: “While we retrieve results from other search engines like Bing and Wikipedia, we also have our own indexes of links that are displayed in our search results. We are actively working on improving these indexes and they will only get better.”
Blade UI Kit is a set of renderless components to use in Laravel Blade Views. It’s built for the tall stack and is completely open source. It includes 26 components and you can contribute as well.
Trusted News is a Google Chrome extension that uses AI to assist in evaluating the quality of the online content. In its first release, it scores the objectivity for a selected article, testing whether it is written from a neutral perspective as opposed to a subjective one.
BaseDash allows you to edit production data without coding. You can make changes to the database with the ease of a spreadsheet. This tool makes it easy to find and edit information in a hurry. It works with all major databases including MySQL, PostgreSQL, Amazon Redshirt, Microsoft SQL Server, and more.
Email2Go is a service that helps you create email templates and test them on dozens of physical devices and applications. It’s free right now while it is in early release.
The Iconscout Converter allows you to convert icons and images from one file format to another for free. Convert SVG, PNG, JPG, and PDF with a single click.
Shape 2 is a massive collection of 5,000+ unique icons and illustrations with a full-blown web editor. Customize colors, stroke width, size and full variations that can export to SVG, PDF, PNG, GIF, and React. This is a premium tool and includes a discounted release price for now.
Aestetico is a beautiful sans serif that includes a massive family with 54 styles. This premium typeface is highly readable and has modern lines and curves that make it a great option for a variety of uses.
Arcades is a modern display font with a retro, 1980s-style vibe. It includes regular and italic styles.
Brimington is a handwriting style typeface with rough strokes and smooth curves. It includes a set of 227 characters and 219 glyphs in a readable design.
California Signature is a typeface duo with a slab serif and handwriting style that are perfectly paired. The thick and thin options provide a yin and yang effect.
Eastblue is a script typeface with long swashes and interesting curves. It includes a solid character set and is free for personal use only.
xxxxxxxxxx
/ / MuSQL
$Database = new PDO ("mysql: host = $ host; dbname = $Databasename", $username, $password);
/ / PostgreSQL
$Database = new PDO ("pgsql: host = $ host; dbname = $Databasename", $username, $password);
/ / MS SQL
$Database = new PDO ("mssql: host = $ host; dbname = $Databasename", $username, $password);
/ / SQLite
$Database = new PDO ("sqlite: my / database / path / database.db");
xxxxxxxxxx
try {
$Database = new PDO ("mysql: host = $ host; dbname = $Databasename", $username, $password);
}
catch (PDOException $ e) {
echo "Unable to connect to database";
}
Change Data Capture Architecture Using Debezium, Postgres, and Kafka
was a tutorial on how to use Debezium for change data capture from Azure PostgreSQL and send them to Azure Event Hubs for Kafka – it used the wal2json output plugin.
pgoutput Plugin?This blog will provide a quick walk through of how to pgoutput plugin. I will not be repeating a lot of details and use containerized versions (using Docker Compose) of Kafka connect, Kafka (and Zookeeper) to keep things simple. So, the only thing you need is Azure PostgreSQL, which you can setup using a variety of options including, the Azure Portal, Azure CLI, Azure PowerShell, ARM template.
In a post published on our blog earlier this year, we described some of the decision-making that went into the design and architecture of Snuba, the primary storage and query service for Sentry’s event data. This project started out of necessity; months earlier, we discovered that the time and effort required to continuously scale our existing PostgreSQL-based solution for indexing event data was becoming an unsustainable burden.
Sentry’s growth led to increased write and read load on our databases, and, even after countless rounds of query and index optimizations, we felt that our databases were always a hair’s breadth from the next performance tipping point or query planner meltdown. Increased write load also led to increased storage requirements (if you’re doing more writes, you’re going to need more places to put them), and we were running what felt like an inordinate number of servers with a lot of disks for the data they were responsible for storing. We knew that something had to change.
The past several years have seen increasing adoption for PostgreSQL. PostgreSQL is an amazing relational database. Feature-wise, it is up there with the best, if not the best. There are many things I love about it — PL/ PG SQL, smart defaults, replication (that actually works out of the box), and an active and vibrant open source community. However, beyond just the features, there are other important aspects of a database that need to be considered.
If you are planning to build a large 24/7 operation, the ability to easily operate the database once it is in production becomes a very important factor. In this aspect, PostgreSQL does not hold up very well. In this blog post, we will detail some of these operational challenges with PostgreSQL. There is nothing fundamentally unfixable here, just a question of prioritization. Hopefully, we can generate enough interest in the community to prioritize these features.
In this post, we look at the long and storied history of the SQL vs NoSQL debate. The articles included stretch way back to the ancient days of the early 2010s and go through 2019 – each exploring a different aspect of the SQL vs NoSQL quandry.
After all these years duking it out, is either solution winning the battle for developers’ hearts and minds?
A critical PostgreSQL client contains valuable data, and PostgreSQL databases should be backed up regularly. Its process is quite simple, and it is important to have a clear understanding of the techniques and assumptions.
The idea behind this dump method is to generate a text file from DataCenter1 with SQL commands that, when fed back to the DataCenter2 server, will recreate the database in the same state as it was at the time of the dump. In this case, if the Client cannot access the primary server, they can have access to the BCP server. PostgreSQL provides the utility program pg_dump for this purpose. The basic usage of this command is: pg_dump dbname >backupoutputfile.db.
If you’re a database developer, you know that you need different SQL statements for creating schemas, ad-hoc querying, initiating backups, or troubleshooting. For these scenarios, finding the right graphical tool can speed up these tasks and make you more productive. Over the years, the PostgreSQL community has developed several open-sourced graphical tools for managing PostgreSQL databases, visualizing the datasets it contains, and running queries. You can find a comprehensive list of tools on the PostgreSQL wiki here.
This post is the first in a series of on-going articles that will show you how to work with Yugabyte DB using popular PostgreSQL GUI tools.
