Thanks to services provided by AWS, GCP, and Azure it’s become relatively easy to develop applications that span multiple regions. This is great because slow apps kill businesses. There is one common problem with these applications: they are not supported by multi-region database architecture.
In this blog, I will provide a solution for the problem of getting Kubernetes pods to talk to each other in multi-region deployments.
A combination of AWS Lambda and Amazon API Gateway is a widely-used architecture for serverless microservices and API-based solutions. They enable developers to focus on their applications, instead of spending time provisioning and managing servers.
API Gateway is a feature-rich offering that includes support for different API types (HTTP, REST, WebSocket), multiple authentication schemes, API versioning, canary deployments, and much more! However, if your requirements are simpler and all you need is an HTTP(S) endpoint for your Lambda function (for example, to serve as a webhook), you can use Lambda Function URLs! When you create a function URL, Lambda automatically generates a unique HTTP(S) endpoint that is dedicated to your Lambda function.
Step Functions, the serverless finite state machine service from AWS. With DynamoDB, Lambda, and API Gateway, it forms the core of serverless AWS services. If you have tasks with multiple steps and you want to ensure they will get executed in the proper order, Step Functions is your service of choice.
It offers direct integrations with many AWS services, so you don’t need to use Lambda Functions as glue. This can improve the performance of your state machine and lower its costs.
Amazon Web Services (AWS) is the biggest cloud platform in the world, with over 200 features. In this article, we break down 10 AWS services that support at least some SQL syntax, talk about their use cases, and give examples of how to write queries.
| Service | Description | SQL Support | Use Case |
|---|---|---|---|
| RDS | Postgres, MySQL, etc. | Full | Small-medium web apps |
| Aurora | Serverless databases | Full | Serverless apps |
| Redshift | Data warehouse | Full | OLAP, Petabytes of data, analytics |
| DynamoDB | NoSQL database | Some – PartiSQL | Ecommerce, building fast |
| Keyspaces | Managed Cassandra (key value) | Some – CQL | Messaging |
| Neptune | Graph database | Some – openCypher | Social networks |
| Timestream | Time series database | Partial | IOT, Logging |
| Quantum Ledger | Cryptographically verified transactions | Some – PartiSQL | Finance |
| Athena | Ad-hoc queries on S3 | Some – CTAS | Historical data |
| Babelfish | MSFT SQL Server on Aurora | Full | .NET |
The table above shows how SQL support varies between the services. A graph database cannot be queried in the same way as a classic relational database, and various subsets of SQL, like PartiQL, have emerged to fit these models. In fact, even within standard SQL, there are many SQL dialects for different companies like Oracle and Microsoft.
In our previous article, we discussed two emerging options for building new-age data pipes using stream processing. One option leverages Apache Spark for stream processing and the other makes use of a Kafka-Kubernetes combination of any cloud platform for distributed computing. The first approach is reasonably popular, and a lot has already been written about it. However, the second option is catching up in the market as that is far less complex to set up and easier to maintain. Also, data-on-the-cloud is a natural outcome of the technological drivers that are prevailing in the market. So, this article will focus on the second approach to see how it can be implemented in different cloud environments.
In this approach, if the number of partitions in the Kafka topic matches with the replication factor of the pods in the Kubernetes cluster, then the pods together form a consumer group and ensure all the advantages of distributed computing. It can be well depicted through the below equation:
After an application is deployed to production, developers should lock down its underlying infrastructure to prevent accidental changes. Some of the common accidents that can affect the availability of an application in production are: moving, renaming, or deleting the resource crucial to the function of the application. You can use locks that prevent anyone from performing a forbidden action to avoid such mishaps.
Almost every resource in Azure supports locks, so you will find the lock option in the settings section of nearly all resources in the portal. For example, the following screenshot illustrates locks on resource groups:
You might have noticed that resources comprising some Azure services such as Azure Kubernetes Service (AKS) span multiple resource groups by default. In some cases, you might intentionally want to segregate resources such as disks and network interfaces from VMs by placing them in different resource groups for better management. A common problem arising from the resource spread is that you might find it challenging to delete multiple resources and resource groups to entirely remove a service from a subscription.
We can solve the problem by using resource tags to associate resources and resource groups to a service. Tags are key-value pairs that can be applied to your Azure resources, resource groups, and subscriptions. Of course, you can use tags for many other purposes apart from resource management. The Azure docs website has a detailed guide on the various resource naming and tagging strategies and patterns.
Lambda functions are a fundamental component of the AWS serverless model. They provide a simple, cost-effective, and easily scalable programming model based on FaaS (functions as a service).
Lambda functions can be referenced by their ARN (Amazon Resource Name). For example, the ARN to reference a ‘helloworld‘ function in the ‘us-east-2‘ region in account ‘3445435‘ would be:
Today, more than 100 billion searches are conducted every month on the Google search engine alone. Search engine users conduct searches for several reasons including the foundational conversion of information into action. An action could be a decision to purchase, consume information for decision-making, or seek a better understanding of an issue or topic among others. Search engines make information available at our fingertips right whenever we need it.
In this era of big data, search solutions are useful not only for popular search engines like Google, Yahoo, and Bing but also for enterprises for monitoring and managing the growing volumes of data in their databases to enhance operational efficiency. The enterprise search industry has grown remarkably and is expected to be worth $8.90 billion by 2024.
At its core Snowflake is a data platform. It’s not specifically based on any cloud service which means it can run any of the major cloud providers like Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP). As a SaaS (Software-as-a-Service) solution, it helps organizations consolidate data from different sources into a central repository for analytics purposes to help solve Business Intelligence use cases.
Once data is loaded into Snowflake, data scientists, engineers, and analysts can use business logic to transform and model that data in a way that makes sense for their company. With Snowflake users can easily query data using simple SQL. This information is then used to power reports and dashboards so business stakeholders can make key decisions based on relevant insights.
