MicroServices Pattern: Event Sourcing - A Light Introduction

A basic introduction on designing mircoservices using event sourcing

Overview

The Event Sourcing pattern is gaining significant popularity, building a microservices architecture. As many service designs adopt real-time asynchronous communication, they rely on events from related services to update the state of the current business process transaction. In this blog post, I'd like to explain what event sourcing is and how it could be used to build enterprise products, using some real-world examples.

What is Event Sourcing

Event Sourcing is a design pattern where, instead of just storing the current state of an application, you store the sequence of events that led to that state. Traditional microservices often use a database to hold the current state of a service; for example, an online store maintains the product details and its stock availability in the database. It updates the product availability/quantity with each new order the product was part of, which means the database would have the current state of product availability. With Event Sourcing, instead of updating that number directly, it could record events like "ItemAddedToStock" or "ItemOrdered." The current stock level is then a projection of all those events. The sequence of events is often stored in an event store, which is a specialized database that can be queried to reconstruct the state at any point in time. In the context of microservices, this pattern offers a powerful way to handle data, communication, and system evolution.

Core Concepts

• Events: These are immutable facts that represent something that has happened in the system. They are written in the past tense, in the form of events as they occur.

• Event Store: This acts as the database for the events. It's a single source of truth for the system's history. It allows you to append new events but not modify or delete old ones.

• Projections: Since the event store only holds events, you need a way to get the current state. A projection is a read-only representation of the current state, built by replaying the events. You can have multiple projections for different purposes, each optimized for specific queries.

• Commands: These are requests to do something. A command like "PlaceOrder" leads to the creation of an event, such as "OrderCreated."

Event Sourcing Building Blocks

Let’s start with an example. Consider that we are trying to build a developer portal for an organisation that offers Saas products. One of the main components of the developer portal is the Application, which enables developers to access the API products by creating access credentials and selecting the API products to be used.

Typically, when tasked with building a Developer Portal Application, we end up with a simple model similar to the one depicted in the following figure.

A developer portal Application could involve many different business processes, which could generate events at each stage of the business process transaction. Let’s list a few of them below, with some examples of the business process flow for some of them.

Business Processes Developer Portal Application

• Create Application

• Add the API product to the application

• Remove the API product from the application

• Change Application Owner

• Generate credentials for the application

• Application credentials expire

• Delete Application

Let’s try to analyse some of these business processes with the process flow. For the business process flow, let's consider the data store is a NoSQL document database - MongoDB, Kafka as a message broker for events, and Neo4j as an events store and a backend application serving API requests and publishing business events as required.

Create Application

The flow mentioned above generates some events in the process of creating the developer portal Application. The system would want to record all these events and would also run other business process flows based on the event. Based on what we have considered in the process flow, let's say the create application process generates the following events.

• APPLICATION_INSERT - Domain Model Event

• CREDENTIAL_INSERT - Domain Model Event

The two events mentioned above appear to be domain model events triggered when records or documents for those are added to the datastore. In this case, MongoDB generates the change stream event and publishes it to the message broker. Here, the event handler(as mentioned above) would consume these events, process and then add to the event store in this case, Neo4j GraphDB.

Add API Product

Considering the flow mentioned above, similarly, the Add API product business process would generate a few events. Let's consider the most common events

• ADD_API_PRODUCT_TO_APPLICATION - Business Event

• APPLICATION_UPDATE - Domain Model Event

Remove API Product

Let’s consider the developer portal Application added earlier, which included the three API products. Now, the organization or developer using the products chooses not to use one of the SaaS products from this particular software vendor, but rather chooses to use it from another software vendor. In this case, the developer removes the specific API product from the developer portal Application. This process of removing an API product from the Application could generate business and domain events. Below is the list of the most common events generated.

• REMOVE_API_PRODUCT_FROM_APPLICATION - Business Event

• APPLICATION_UPDATE - Domain Model Event

Significance of events to the business

Let’s elaborate with an example on what business value the above-mentioned events have. Would consider a few of the above-mentioned events just to help understand the value of storing events as part of event sourcing.

APPLICATION INSERT

The APPLICATION_INSERT event is the database change stream event, which is triggered when a new application is added to the developer portal. Storing the event details in the events store when an application is added brings real business value.

Consider below business value stream the organization is looking at for this particular event.

1. The organization would like to target organizations or developers who have added the new application but are using the free subscription model for more than 30 days.

2. The application is using a paid subscription; the organization offering the API products would want the feedback on the current API product offerings and recommendations, which could lead to new product offerings.

Process Flow

As in the process flow, the create application backend triggers a survey business process; here, the trigger is “paid subscription model for more than 3 months”. The survey app creates a Google Form survey and stores the survey details in the datastore. Further in the process, the survey link is sent to the respective application owner, providing an opportunity to provide feedback. This process flow generates a few events, which are valuable to the business in closing the feedback loop by incorporating feedback into change requests for improvement (if required) on the current API products or pushing for new features.

Below are the possible events that could be generated in the process flow.

1. SURVEY_ADDED(Generated by Datastore change stream)

2. SURVEY_SENT(Generated by communication app)

3. SURVEY_FEEDBACK_RECORDED(Generated by survey app on receiving Google Forms response)

Business Owners can now use the data from these events and act on it as and when required.

REMOVE API PRODUCT FROM APPLICATION

The REMOVE_API_PRODUCT_FROM_APPLICATION event is a business event generated when an API product is removed by the application owner from the application(s) on the developer portal. Now, when an API product is removed from the application, the business would want to understand why the application owner has removed it or has stopped using it. Here, event sourcing is the best way for businesses to get the details on what API products were removed by the application owners who are using the organization’s SaaS platform. Using the event data, the organization’s API platform could trigger a business process flow to get feedback from the application owners on the API product(s) that were removed from the application(s).

As mentioned above, for the APPLICATION INSERT event, the survey process flow generates a few events, which are valuable to the business in closing the feedback loop by incorporating feedback into change requests for improvement (if required) on the current API products or pushing for new features, or in certain cases running an offer campaign on certain API products to be competitive with product offerings.

Handling and Processing Events

As mentioned in the preceding sections, all events generated on the SaaS platform have specific event handlers and processors that handle and store events in the event store. The diagram below gives an understanding of how the events for different domain objects on the SaaS platform could be handled and processed.

The above diagram describes how the events are handled and processed. Each application generates events related to the domain object and publishes them to the message broker. The events could be business or domain events. There are also certain business processes, like feedback surveys(as mentioned in the above sections), that generate and store the events in the event store. Any application or business process generating events would have associated event handlers to listen to all the events and process further. The event handlers call the specific event processor based on the event type to process and store event data in the event store.

Conclusion

Event sourcing is a powerful design pattern that offers a fresh perspective on managing application state, particularly within the context of microservices. By treating every state change as an immutable event, it provides a comprehensive and verifiable history of all invaluable actions.