Event-Driven Architecture (EDA) vs Request/Response (RR)

In this video, we're going to look at event-driven architectures, or EDA, that are powered by Apache Kafka. This is where systems communicate with one another through exchanging events. We're going to compare and contrast EDA with request-response architecture, where the services communicate directly with one another. Examples of request-response include REST, as well as RPC, or remote procedure calls. We're going to look at six factors to see how these architectures vary, and to give you a better idea of the strengths and weaknesses of each. Let's start with an example. Imagine you have a storefront that shows customers products and lets them place an order, say via their cell phone. On this side, we're going to look at it through request-response, and on this side, we're going to look at it through event-driven architectures. So, let's say our customer here places an order. What happens next? Well, the order is going to be written into the storefront database. And now, for our first factor, we're going to look at reactivity. Event-driven architectures are reactive. Upon storing this order here, the storefront writes a record representing that order to this order's topic. The fulfillment service, in turn, will consume this record, possibly store it in its own data store, and then process it for fulfillment and shipping out to the customer. For a request-response service, the business logic of the storefront service needs to drive the next step. So, the storefront will either tell the fulfillment that there's an order to process, it'll request that it processes it, and get a response back indicating if it could or could not. Alternatively, the fulfillment service could ask the storefront, is there any work that I need to do? And could, of course, then be returned that work to do for itself. In either case, these two services are communicating directly with one another through their APIs. The second factor we're going to look at is coupling, both in space and in time. EDAs are inherently asynchronous. By writing this record into this topic, we've decoupled the storefront from the fulfillment. If the fulfillment service is offline, it can come back online and resume processing orders. If the storefront service is offline, fulfillment can keep processing all the until it runs out. The consumer doesn't know the identity of the producer service, nor does it have any direct coupling on it. They're only coupled via this Kafka topic. For request-response, the services, these two services here, they need to know the specific details of each other's API, as well as their location at runtime. These services are tightly coupled to each other. And furthermore, both of these systems need to be online and available at the same time. If either one goes offline, then the service will have to try again to re-communicate to get the work that it needs to do completely. They're tightly coupled in both space and time. The third factor we'll look at is consistency. This asynchronous temporal decoupling of event-driven architectures gives us a restriction known as eventual consistency. So, this fulfillment service will be eventually consistent with the orders that were placed in the storefront service, but that might take some time. How much time it takes will depend heavily on the volume of events and the consumer's processing rate. In request-response services, you often have an easier time with consistency. If you keep all of your data inside of a single service, by definition, it's consistent because there's only one copy. However, if this fulfillment service, let's say, were to request work to do from the storefront and the storefront were to respond with a list of orders to fulfill, the order data that ends up in here will be eventually consistent with the data that's coming from the storefront. This is true anytime you make a copy of data. The fourth point we're going to look at is historical state. So, let's consider that this customer who placed an initial order wants to modify it and they want to get the all-in-one shipping, everything shipped in one box. So, they say they log in, they edit their order, it hasn't shipped yet, and they submit that into the storefront. The storefront will update that data in place and then write a new record into this topic with that information available. The fulfillment service, upon consuming that, will see the modifications and if it's already shipped the package, it may just have to ship out a second one. If it hasn't shipped it yet, it can amalgamate the two together and ensure that they get shipped out at the same time. This Kafka topic contains a complete history of state for as long as we want to keep it. A new producer, let's say an inventory service, could spin up and consume this data, these orders, from the beginning of time. It could consume that from the middle or it might just consume it from the latest. It's entirely up to this service. It can re-read it as many times as it wants in case there is a failure or a bug and you want to reprocess it. You have that flexibility and that option. For request-response systems, typically, we overwrite this data in place. So, when we've modified that order, we simply overwrote the old one. The storefront service, though, upon noticing that it's overwriting existing orders, would need to communicate to fulfillment and say, listen, this order's been modified and you need to account for it right now. Typically, we don't store historical state alongside the current state in a lot of common database patterns. You can do it, but it is fairly uncommon to do so. This brings us to our fifth factor, which we will call architectural flexibility. Event-driven architectures, the streams, make it very easy to build new services, as we just showed here. But it also makes it easy to take data from across the company from different services. So, here's a different topic. Perhaps this is incoming inventory, stuff that's being received at the warehouses, being created by some producer somewhere else. And we can add that into our inventory service to help get a complete picture, not only of inventory going out, but also inventory coming in to create a complete computation of the two. We're able to reuse order data for multiple use cases. In a request response system, we are relying on the service APIs. These service APIs are the building blocks for our applications. So, API blocked, and here, it would be data are the building blocks. So, let's say we create a new inventory service over here in the fulfillment world. This new service also has its own API, and we'll need to communicate with the fulfillment service. We'll need to communicate with the storefront service, perhaps. And, of course, if we have this shipping service over here that's measuring, tracking the incoming deliveries, we would also need to communicate with this API here to see, hey, what products do we have coming in? So, this new service is relying on all these other existing APIs to solve its business functionality. If your boundaries align well between your services, this can work. But often, the service boundaries don't necessarily align how you like, and you might have to call multiple services to get your work done, and those services, in turn, might have to call other services. So, a fairly trivial task might actually be pretty challenging to implement, given how your service boundaries have evolved over time. This problem is the most noteworthy when the data your service needs to do its work is spread across multiple systems, and you're forced to make many different service calls. And this brings us to our final point, our final factor, data access and data reuse. Event-driven architectures are extremely powerful for data reuse, right? We've already shown that you can power these different applications, but you can also power things such as your data lakes, warehouses, you can train machine learning models, you can pump this into your generative AI vector databases. All of these existing event streams are very easily routed into these data-intensive applications. You can use something like Kafka Connect to easily create these connectors to get that data in there. You can use native consumers, you have a lot of flexibility. But all of the work that you did already to get these events defined and populated into these streams can be reused again and again and again. In contrast, request-response architectures are primarily about serving real-time operational data. So, if you want to populate your own data lakes over here, let's say, and you have a lot of these individual request-response services, you're going to have to set up ETL jobs or extract, transform, and load. And these can be their own challenge to set up. And so, a lot of the work that you've already done in curating these APIs for your business use cases don't necessarily transfer over when you're trying to get your data into a format to do all sorts of other important data work like your data lake analytics, machine learning, and generative AI. So, let's wrap this up. Event-driven architectures such as those powered by Apache Kafka provide a loosely coupled and asynchronous framework for building reactive applications. These event streams act as reusable building blocks for powering applications as well as data-intensive use cases and for connecting everything all together. Request-response services provide business functions for other services to call. Uptime and liveness are critical as a failure in one service can instantly affect all of the other dependent services. While request-response services are indeed capable of powering distributed applications, it still remains complex and challenging to simply access important business data for your own use cases. If you would like to learn more, please check the video details for related content. And if you've enjoyed this video, please like and subscribe. Stay tuned for future episodes where we'll talk more about event streams, event-driven architectures, and how to build a healthy data ecosystem.