Event Sourcing, Hacker News

May the source be with you.

I recently became intrigued by the concept ofevent sourcingas applied to back-end architecture, specifically a microservice-oriented approach. I have spent the last few years working predominantly on the front-end, and became enamored by the simplicity and elegance of this pattern as the backbone of front-end architecture, popularized byRedux.

To better understand the trade-offs between a traditional, monolithic back-end, and a microservice-oriented, event sourced approach, I began sketching a toy architecture for the initial user flow of seemingly every web application: signing up for a user account, and receiving an activation email. Easy enough, right?

Little did I realize just how alien the event sourcing pattern would feel. I quickly developed more questions than answers. I spent the next several days reading everything I could about the subject, desperately begging Google to show me the way. I learned an incredible amount during that time, and in the spirit of the greatJulia EvansI felt compelled to distill and summarize what I have learned for you, my fellow traveler.

This is by no means a guide indicating the “correct” way to do anything. My hope is that, if you’re new to event sourcing, this summary might help you to start reasoning about how such a system could work.

Things We Will Talk About

**************** (May the source be with you.   

(what is the hell) is event sourcing?  

(Failure Modes)       

Validation       

Maybe You Don’t Need Immediate Consistency

      

Locks

Temporal Anomalies       

Fencing Tokens       

(Filtering Duplicates)       

Uniq Service       

Relational Database Envy       

(The Read Model) ******************************       

Sending Mail***********************       

Dependency Woes       

(That’s All, Folks*********************     

  

Where do we go from here?

What the hell is event sourcing?

Good question! Martin Fowlercan tell you:

Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.

But – and this is true of most definitions of most things – this will just leave you with even more questions. So I’ll try to explain event sourcing instead of defining it.

(Your application produces a

********** log of events. For example, you might log a (UserAccountCreated) event for each user account that is created. The log might be split into smaller, independent logs calledtopics, to help organize your events.   

The events are thesource of truthor system of recordfor your application. It is common for applications to write to a database and treat it as the source of truth, but when event sourcing we write to the event log instead.  

Other parts of the application can read from the event log. This allows for a pub / sub style of communication, where multiple listeners can react to events they are interested in.

  

Listeners can reconstruct their own application state by reading from the event log and applying the events to their own, private data store, such as a database. They might apply some of the events, or all of them, depending on their use case. Events are always applied in the total order that they appear in the log.

*************************************************************** What can event sourcing do for me?

I can recommend two really good sales pitches for event sourced architectures (sometimes calledlog-orientedarchitectures), and a more pragmatic overview. I recommend that you read and watch these in the following order:

(Martin Kleppmann has an) excellent write-upto whet your appetite.

  

Greg Young gave

********************************** a great talk

which really helped me to understand how event sourcing can be useful

even at small traffic scales.

**************************************************************

But it’s not fair for me to dump two hours of educational materials into your lap, so I’ll do my best to summarize the observations of the great masters.

Historical Queries (**********************************************************************

A typical database can answer questions about your data as it exists right now, but it struggles to answer time-series queries about the historical context and evolution of your data.

For example, you can query your database to determine the number of user accounts that exist. But what if your business stakeholder wanted to know how many users create an account, delete it, and then change their mind and create it again? Your database typically will not capture this data, since it only stores thecurrentstate – it only stores the user account, and not thestepsthat were taken to create that account. Writing events to a lognaturally makes these kinds of queries possible, because the historical data is never deleted. The ability to be able to answeranyquestion that the business asks about the history of the application is incredibly valuable.

)

Historical queries can ask, “How did we arrive at this state?”, Instead of, “What does the current state look like?”

(Immutable Data)

Modeling your data as an immutable, append-only log of events greatly simplifies reasoning about how the application works. It is harder to get yourself into a confusing situation by accidentally mutating state. This is easier to understand when we consider the utility of time-traveling debugging.

Time-Traveling Debugging

Dan Abramov (creator of Redux) has sung the praises

of time-traveling debugging from a front-end perspective, and the same principle applies from a back-end one.

Given that the event log is immutable, all changes to the application’s state must be driven byappendingto the event log instead of changing it. This means that when our application behaves in a confusing way, we can simply start from the “beginning of time” and replay events one by one until we isolate the event that is triggering the confusing behavior. This is a powerful and incredibly simple tool for debugging our application.

But that’s not all!Just as our version control system can “check out ”code at a particular point in the project’s history, our event log can“ check out ”a particular point in time so that we can inspect how the state looked at that moment.

As Martin Fowlerpointed out, instead of exclusively writing end-to- end tests we can explore a complementary approach: store and replay a sequence of events into the log, and then inspect the application’s state to ensure that it matches what we expect.

These are just some examples. Retaining the time-series data in our event log opens up numerous opportunities for building technical wealth

(Easily Connect Data Consumers)

An event sourced architecture features an event log as the central hub to which data producers write, and from which data consumers read. This pub / sub architecture minimizes or eliminates the need to write custom adapters to get data out of one system and into another. All data is published in a standardized message format (JSON, or whatever you enjoy). Writing a new consumer becomes easier and more predictable, since systems share data in a consistent way. Multiple listeners can subscribe to an event log without a problem.

(*******

Systems often mutate into Frankenstein architectures as new features and use cases are (bolted on) accommodated. Martin Kleppmann does a great job of describing this phenomenon. Modeling data consumption as a log of events can mitigate this unsatisfactory result.

Reasonable Scaling Defaults

An event sourced architecture provides reasonable defaults for common scalability challenges that applications face as load increases, and after exhausting vertical scaling strategies. It isn’t a silver bullet (nothing is), but we can take comfort in the fact that we are probably not painting ourselves into a corner.

If writing to the event log is the bottleneck, we can split a single log intopartitionsspread over multiple servers, each responsible for handling writes to its fair share of the partitions. This is how Apache Kafka works.

If reading from the event log is the bottleneck, we can introduce log replication and have consumers read from the replicas.

If consumers cannot keep up with the volume of events, we can add more consumers and parallelize the work of processing the events.

If we run out of disk space to store the log, we can explore options for long-term storage. We could write a service to read older log messages and push them into a some kind of data warehouse. Consumers which only need to keep up with processing new-ish messages read directly from the primary log. Consumers which wish to rebuild their local state by processing all log messages from the beginning of time may do so by reading from the data warehouse until they reach the most recent warehoused message, and then switch to reading from the primary log.

()

Fault Tolerance and Resiliency (**********************************************************************

This is my favorite feature of a log-oriented architecture, and the one that attracted me to event sourcing.

Often, one portion of an application will need to react to a change in a different subsystem. For example, when a user account is created, we might want to send an account activation email to that user.

In a traditional monolithic system, the controller which handles this logic might look something like this pseudocode:

(********************************************************************************************** (******************************************************** (user=new User) ‘ [email protected] ‘) user.save () MailService.sendAccountActivationEmail (user)

The above logic will work 100% of the time. But every now and then the  (MailService) ************************************************************ will go offline. The new account will be created but the user will not receive their activation email. The user cannot activate their account!

This is a tricky situation to recover from, and an example of the problem of (dual writes) . It would be much better if we could build an application which simplypauseswhen a subsystem goes down, and resumes from where it left off when that subsystem comes back online. This would provide tremendous peace of mind, save countless users from headache, and prevent us from wasting many days recovering from, debugging, and prematurely optimizing the availability problems of our MailService. ********

Remember: we can often substitute rapid recovery for high availability. Instead of investing significant sums to achieve high availability, we canPareto-optimizeby investing a smaller amount into rapid recovery. For example, instead of buying the world’s most reliable hard drive, we could simply make frequent backups. Our system can go down frequently, but the user will never notice as long as we can recover in a reasonable amount of time. As Gary Bernhardtastutely points out, TCP is so good at this that we take it for granted!

TCP is so successful at its job [packet retransmission, rapid recovery] that we don’t even think of networks as being unreliable in our daily use, even though they fail routinely under normal conditions.

This is a great example of the unreasonable effectiveness of defense in depthstrategies. The first layer of defense is designing for availability, and the second layer is designing for recovery.

A log-oriented architecture can give us these benefits! Let’s rewrite our pseudocode controller:

(********************************************************************************************** (******************************************************** (user=new User) ‘ [email protected] ‘) event=new AccountCreatedEvent (user) EventLog.append (event)

Notice how we are no longer performing dual writes. Instead, we perform a single append to the event log. This is equivalent to saving the user account in the first example. If we model our writes as single log appends, they become inherently atomic.
The email service would be monitoring the log for new account creation events, and would send emails in response to those events. If the email service were to go offline, it could simply pick up from where it left off. In fact, the email service could go offline fordays, and catch up on unsent emails when it comes back online. It could also contain a memory leak which causes the system to crash every hour, but as long as the email service restarts automatically, your users will not likely perceive a service interruption.
Mitigation of Data Inconsistencies
Kleppmann  points outthat systems which employ dual writes pretty much guarantee data consistency problems.
For example, let’s say you update a user account record in the database, and then update a cache containing the now stale data. Let’s further say that the cache update operation fails. Your cache is now out of sync with your database. Have fun debugging the consequences!
A read-through cache can exhibit a similar problem. Updates to a user account in the database will not be immediately reflected in its corresponding cache entry until that entry expires. Stale cache data can be very confusing to both users and developers.
But what if we perform all writes to an event log? The cache can read and apply the events in order. The cache is always in sync with its source of truth, with the standard disclaimers about eventual consistency applying. But under normal circumstances, your cache could be quite consistent with its data source. Should anything go wrong, the cache can be rebuilt by simply starting from scratch and re-consuming the event log.
 (Simplicity) 
Nothing about software architecture is truly simple. But anyone who has been burned by the legacy of a bad decision will intuitively understand that simpler solutions are generally preferable. Simple solutions reduce cognitive overload, maximizing the chances that you will correctly predict the system's behavior.
An event sourced architecture really shines as a simplifying abstraction when compared to the Frankenstein architecture which tends to evolve from modest monolithic beginnings. Producers write to a log, consumers read from the log. This simple, unifying principle allows us to reason about data flow between subsystems without becoming bogged down in their idiosyncrasies.
Kleppmann  describedthe event sourcing approach as Unix philosophy (clearly pipes) for distributed systems. The simplicity of Unix pipes is precisely what makes them so composable and powerful.
 (Forgiving Of Mistakes) 
We all love that feeling when we write a piece of code and it works on the first try. That feeling is so wonderful because it is so rare. It is more common to spend as much time debugging our code as we did writing it. Mistakes are by far the normal mode of software development. Anything our architecture can do to help us recover from mistakes will have a dramatic impact on our iteration speed.
The traditional, stateful model of data persistence is very unforgiving in this regard. A bug in your code which mutates state in the wrong way will often require a one-off, compensating transaction to correct. And it's a race against time to make the correction, since subsequent operations based on bad data will only  (compound the error

).

But what if we can fix the bug and simply rebuild the state by re-consuming events via the patched system? We wouldn’t need to duct tape our state. When the application is corrected, so is the state. We can reduce instances of fixing the application and thenalsofixing the state.

Of course, there will always be exceptions.All abstractions leak.But in general we prefer boats with fewer leaks.

Ends Normalization Debate **********************************************************************

Kleppmann makes an Excellent observationregarding the best practice of database normalization. There is a tension between read- and write-optimized schemas. At a certain point, in order to boost read performance, we are tempted to denormalize our database. We might also attempt to cache query results from the normalized database, usually employing some kind of error-prone, dual write strategy.

********

A log-oriented system breaks the tension byderivingone or more read models from the log. We accept from the outset that one model cannot be great at ever ything. The log is write-optimized, and the derived read models can be denormalized to suit their specific usage pattern.

(Audit Trail) ********************************************************************

Greg Young recallsthat he was initially attracted to event sourcing because he needed to implement auditing. Storing a log of every event that has occurred in the history of the application provides a natural audit trail.

If we aren’t working on, say, a financial application, we tend to think that auditing will not be an important use case for our software. Then an incident occurs in production, and what do we do? We check the logs!

(Better Business Agility)

Kleppmann seesagility enhancingbenefits in this approach to building software, and I think he’s on to something.

Monolithic, stateful systems are optimized for consistency, not for change. At a certain point it becomes difficult to make changes, because those changes must render the system consistent when completed. Within a large system, that is no small feat! The result is that the rate of change decreases, because it becomes a huge pain to run even a small experiment.

The ability to connect new consumers to the log stream opens up the possibility ofbypassingexisting systems to build one-off experiments. There is no need to run a migration to modify a database schema – simply deploy a new service with a different database, and store the additional data there for the duration of the experiment. The same goes for new read models, which can provide denormalized views for experimental new queries.

Have you ever noticed thatchangingan existing system tends to trigger a bikeshedding process?Addinga new system, in my experience, does not produce the same strong political reaction. My hunch is that this is because changing an existing system might break something which a colleague considers incredibly valuable, even sacrosanct. So by architecting our application to allow for the easy introduction of new subsystems, it seems reasonable to expect that we could actually reduce the amount of political debate associated with the running of experiments.

What might event sourcing look like in practice?

Recall the initial user flow I described earlier: the user signs up for an account, and receives an activation email. Thinking about how to implement these features in an event sourced architecture provides a surprising amount of insight into the pattern and its subtleties. Let’s work through it!

Signing Up For An Account (********************************************************************

Our user lands on the account sign up page and fills out the form, providing their (username**********************************************, (email) , and (password) ************************************************************.

The user submits the form and an HTTP request is sent to our API Gateway service, which is the public-facing portion of our system. It might implement server-side rendered views, or it might expose an API or Backend For Frontend (BFF)

for a single-page or mobile application to consume. Immediate Feedback For The User

We want to build this application in a microservice style, and so we have decided to delegate ownership of all write-related user logic to a User Command service.

We might be tempted to have the API Gateway publish an (AccountSignUp) ********************************************************** event, which our user command service would listen for and process. After all, this is how a lot of event-driven architectures behave – the userdid a thingthat the system can react to. Unfortunately this creates a huge UX problem: we lose the ability to provide immediate feedback to the user. There is no guarantee that the User Command service is currently available – it could be overloaded, or it might have crashed. If we publish an (AccountSignUp) ************************************************************ event and some of the form data is invalid, we have no way of informing the user. The best we can do would be to display an optimistic “success” message,hopethat the form data is valid,hopethat the user account is persisted, andhopethat, should any errors occur, the user would return to our website to try again.

The breakthrough approach here, for me, was when I understood that in an event sourced system, all of thewritesmust occur to the event log. It could be interesting or useful to log some of the antecedent details (such as requests), but the only thing we reallymust

do is ensure that all writes are modeled as log appends.

If the user interface requires a synchronous response for immediate feedback, so be it. We can achieve this in the traditional, RESTful way, by having the API Gateway issue an HTTP request to the User Command service. Perhaps this would be modeled as a (POST / users) endpoint. The User Command service would perform any validations, write an (AccountCreated) ********************************************************** (event to the event log, and return a(Created) Response to the API Gateway. The API Gateway would then render a success message for the user. The user account creation is considered to be a success - a historical fact - at the moment the log append occurs. (Greg Young******************************************************************************** emphasizes the importanceof storing only facts in our event log.)

(Failure Modes)

Let’s think about how this part of the system would handle various failures:

If the API Gateway’s HTTP request to the User Command service fails, the API Gateway can immediately render an error message for the user. The user is then able to retry their request.

If any of the form validations fail, the User Command service can return a (*********************************************************************************************************************************************************************************** Bad Baderror to the API Gateway, which in turn can render field errors for the user.

************************************ (response, and wouldnotforward the event to the log. Use cases involving changing or deleting an existing email address are easily supported by the HTTP (PATCH) ******************************************************** and DELETEsemantics. Unlike the Unix (uniq) , our Uniq requires these semantics because it is stateful. It isn’t simply counting unique items, but rather allowing that set of unique items to be maintained in the face of change.

Of course, given that Uniq will be maintaining a set in memory, we must consider what will happen in the event that it crashes. If this were to occur, Uniq can rebuild its set by re-consuming log messages. Wh en Uniq has caught up with where it left off, it can accept write traffic again. Because our log is totally ordered, and because Uniq processes write requests serially, it should never commit a duplicate write to the log, even when recovering from an outage.

If availability is a concern, a second instance of Uniq can operate in follower mode, consuming from the event log to maintain a replica of the leader’s state. When the leader dies, the follower can be promoted to leader.

Relational Database Envy (********************************************************************

So we have our strategy for handling concurrent requests for user accounts with the same email address: we will pipe all writes through a Uniq service, which enforces the uniqueness constraint.

Couldn’t we have used a relational database to enforce this constraint instead? Well, yes, we could. Michael Ploed makes thewise recommendationthat we implement the level of consistency that our business domain requires. The ideal level of consistency for one subsystem may not be needed across the entire system. For example, it might be considered unacceptable for two user accounts toeveraccidentally share the same email address, since this would prevent users from logging in. Therefore, we can enforce a uniqueness constraint only for email addresses, paying the complexity cost because we see it as justifiable. But for other subsystems where we might traditionally enforce a uniqueness constraint, we could employ more creative solutions. We are dialing in the amount of consistency that each part of the system requires.

So it would make sense for the User Command service to write to a relational database implementing a uniqueness constraint. But this approach creates one very unfortunate side-effect: we lose atomicity. Writing a new user to the database, andthenwriting an event to the log, opens the possibility that the first write will succeed but the second one will fail. This situation would render our data inconsistent, and a compensating transaction would be required to reconcile the two. Dual writes strike again!

It might be possible to create a second table in our relational database, and write our events to that table. We could then wrap both our user write and the event write in a transaction, regaining atomicity. But now we need a way to get those eventsoutof the database and into the event log, and ideally without implementing a polling strategy, which would increase replication lag. The amount of plumbing required to make this happen is excessive, in my opinion. And the solution wouldn’t be reusable across services.

The Read Model (**********************************************************************

So our User Command service is able to write an AccountCreated

event to the log via the Uniq service. But how would we handle reads? One cannot simply perform queries against an event log, since query performance would decrease as the log grows in size! To support reads, we will need to implement a

read model.

The read model consists a service wrapping a persistence mechanism. Most likely we would choose a database which provides a good fit for the types of queries we will be performing. For our User Query service we will assume a document database which stores JSON-like documents.

The read model will consume relevant events from the event log, and update its database accordingly. In the case of the User Query service, every (AccountCreated) ********************************************************** (event consumed would trigger the insertion of a new user document into the database.)

Where this pattern can become very powerful is in improving highly optimized, incrementally computed query results. One could imagine introducing a Friends service which maintains a list of frequently contacted friends for each user, entirely derived from log messages. Batch computing this contact frequency information could take a long time, and the results would quickly become stale. Incrementally computing with each new piece of information can provide a more consistent view, while maintaining fast query response times.

Another interesting property is the potential for the elimination of schema migrations. Denormalizing the read model brings the possibility of introducing NoSQL stores. The addition of a new field would be handled in application code, and a schema “rollback”, if one were required, could be accomplished by reverting the application code and replaying events from the affected period.

(Sending Mail) ********************************************************************

After our user has signed up, we want to send them an account activation email. To accomplish this, we will create a Mail service which monitors the event log for (AccountCreated) ********************************************************* events, and sends activation emails to those users, probably by calling the RESTful API of a third-party email provider.

But what happens if the Mail service crashes after reading a message from the log? As it turns out, this is not a problem, provided that the mail service persists the ID of the last message it consumed – acheckpoint. And where better to persist this ID than to a topic within the event log!

How frequently should we store checkpoints? If we store a checkpoint for every message consumed, log consumption speed will be limited by the need to write a checkpoint in between every read. If this was inadequate for our purposes, we could have the Mail service store a checkpoint every hundred writes, or every thousand, or every 99 seconds. But then wouldn’t we be at risk of sending a hundred, or a thousand, or 90 seconds worth of duplicate emails if the Mail service crashes?

As it turns out, the Mail service cannot guarantee exactly-once delivery of emails to users. Let’s say that the Mail service has already consumed event 1, and is now consuming event 2 from the log. If an email is sent and the service crashes before checkpoint 2 can be written, when the service restarts it will begin working from the next event after its last checkpoint. The last checkpoint was 1, so event 2 will be processed again. A duplicate email will be sent!

Writing the checkpoint before sending the email will only make things worse. If we store checkpoint 2 and then crash before sending email 2, when the service restarts it will begin working from the next event after its last checkpoint. The last checkpoint was 2, so event 3 will be processed. In this case, event 2 will be skipped!

Since we cannot prevent the Mail service from sending duplicate emails, and since it would be a Very Bad Thing ™ to fail to send any account activation emails, we can feel a bit better about setting a less frequent checkpoint rate.

(Dependency Woes)

It is a prudent exercise to think about what would happen if our third-party email provider were to suffer various failures.

If for any reason our Mail service does not receive a response from the provider, we can retry the request. In fact, weneedto retry the request, because the log-oriented nature of our system can only guarantee that all events are processed if they are handled sequentially. If we start skipping events, we would need to enqueue those skipped events into – you guessed it – another log for later processing. It’s logs all the way down.

So the system will guarantee delivery bypausingwhen an error occurs, polling for success, and resuming when the error condition has passed. This entails installing a circuit breakerto gate calls to the email provider. If the provider becomes unresponsive, the Mail service will retry the request repeatedly until the circuit breaker trips, at which point it will retry the request at a slower rate. When the provider comes back online, a request will eventually be successful and the Mail service can catch up with its backlog.

That’s All, Folks (********************************************************************

I am reminded of Spolsky’s Law of Leaky Abstractions. Event sourcing is just another abstraction which seeks to simplify the complexity of the software we write. Naturally, this abstraction will leak, creating problems for us. But it is important to keep in mind that the monolithic, relational, strongly consistent style of architecture isalsoan abstraction. Our comfort with the abstraction we know too often spares us the terrifying advantages of new ways of doing things!

One thing I have learned over the years is that I can never predict the practical consequences of introducing an unfamiliar technique into an organization. Because something always goes wrong, we actuallyneedto implement the technique to discover where it breaks down. Only after introduction can we identify the concrete problems, and begin to devise solutions.

The knowledge that somethingwillgo wrong is often used as a justification for not experimenting with new techniques at all. “We’ll revisit this discussion later.” Later effectively means never. Interestingly, this seems to be the wrong conclusion to draw from the mere possibility of risk. Problems may be unavoidable, but we have the power to control the scope of the introduction, and thus shape the size of the problems encountered. Given that we possess a “risk knob” which we can dial down to comfortable levels, what justification remains for failing to experiment with new techniques?

I think the most sensible course of action is to treat event sourcing as an evolutionary pattern. Incorporate this pattern into a portion of your project – get your feet wet. But don’t dive in, because you’ll probably drown. But don’t stay out of the pool, because then you’ll never learn to swim! Learning takes time, so the sooner you can get started, the sooner you can determine how to incorporate the benefits while mitigating the pitfalls. As the famous Chinese proverbsays:

“The best time to [begin reinforcing event sourcing patterns] was

.

********        (****************************************************************************************************************************************************** (******************************************************************************************************************************************************************** (Read More) (********************************************************************************************************************************************************************** ()