What is Idempotence - Using Idempotence for System Design (With Examples)

You might hear the term “idempotence” used a lot in the context of APIs and system design. But what does it mean? And why is it important?

In this article, we’ll explore the concept of idempotence and its significance in building reliable and fault-tolerant applications.

For the impatient: An idempotent operation is one that can be repeated multiple times without causing any side effects.

What is Idempotence?

Idempotence is a property of an operation that describes the effect of applying it multiple times. In other words, an idempotent operation is one that can be repeated multiple times without causing any side effects.

Let’s try to explain it with a non-technical example: Suppose you have a button that can only turn on a light. If you press the button once, the light turns on. If you press it again, the light stays on. And if you press it a third time, the light stays on. In this case, the button is idempotent because pressing it multiple times does not change the state of the light.

Why is Idempotence Relevant to System Design?

Whenever we’re building a system, we need to consider how it will behave when things go wrong.

For example, what happens if a network failure occurs? Or if a request times out? Or if a server crashes? These are all common scenarios that can happen in any distributed system.

In most cases, the most common solution is to retry the operation. But what happens if we try it again? In that case, retrying it could lead to unintended consequences or data corruption.

For example, suppose we’re trying to create a user’s profile. If we retry the operation, we might end up with duplicate records or inconsistent data. Another example is if we’re trying to make an online payment. If we retry the operation, we might end up with double charges or inconsistent order statuses.

This is where idempotence comes into play: it ensures that executing an operation multiple times does not produce different or unintended results.

In the following sections, we’ll learn how we can implement idempotence in practice.

Using an Idempotence Key

One way to ensure idempotence is by using an idempotence key. An idempotence key is a unique identifier that is generated for each request. It can be used to identify duplicate requests and prevent them from being processed more than once.

Consider the previous example where we’re trying to create a user’s profile. On the client side, we can generate a random string of text for each request, which we will use as an idempotence key, and send it along with each request.

When we receive a new request on the server, we can check if the idempotence key already exists in the database. If it does, we can reject the request and return an error message. Otherwise, we can process the request and store the idempotence key in the database.

This mechanism will make sure that even if the client retries the same update multiple times, it will only be processed once. For our example, this means that the users profile would not be created multiple times.

Implementing Idempotence in REST APIs

In this section, we’ll discuss what idempotence means when implementing REST APIs.

According to IETF RFC 7231, the following methods should be implemented as idempotent:

1. GET
2. PUT
3. DELETE
4. HEAD
5. OPTIONS
6. TRACE

Since GET, HEAD, OPTIONS, and TRACE are all read-only operations, they are already idempotent by definition.

PUT operations are used to update an existing resource, and DELETE is used to delete an existing resource. Both of these operations should be idempotent because they can be repeated multiple times without causing any side effects.

For example, if you try to delete an entry that’s already deleted, or try to update the same property of an entry that’s already updated, it shouldn’t cause any change of state.

POST Requests

The only method that is not expected to be idempotent is POST.

This is because POST requests are used to create new resources, and creating a new resource multiple times will result in multiple resources being created.

However, there are some cases where POST requests can (and should) be made idempotent. Taking the example we discussed in the previous sections: if we’re trying to create a user’s profile, we would be using a POST operation to do so. However, we don’t want to create multiple profiles for the same user. In this case, we can make the POST request idempotent by using an idempotence key.

For many systems, operations defined by REST APIs result in changes to the underlying database. So, in addition to implementing idempotence at the API layer, we also need to ensure that the database operations are idempotent.

Idempotence in Database Operations

When working with databases, we want to make sure that our data is consistent and accurate. So, if we insert or update some data, we don’t want to end up with duplicate or inconsistent records.

Idempotence can be used to ensure that executing the same operation multiple times has the same effect as executing it once. This property is valuable for maintaining data integrity, consistency, and avoiding unintended side effects.

We can implement idempotence in different ways depending on the type of operation we’re performing.

1. INSERT operations: When inserting data into a database, idempotence means that executing the same INSERT statement multiple times will result in a consistent state without duplicating or modifying existing data. This can be achieved by using techniques such as primary keys or unique constraints to prevent duplicate entries.
2. UPDATE and DELETE operations: This can be achieved by using fields that have unique constraints in the update (or delete) statement to ensure that executing the statement multiple times doesn’t update (or delete) additional records.
   For example, consider the SQL statement DELETE FROM my_table WHERE id=123 - if we execute this statement multiple times, it will only delete one record with the ID 123 (assuming id is a unique field). If we execute it again, it will not delete any records because there are no records with the ID 123 left in the table.

By leveraging idempotence in database operations, you can design systems that are more robust, resilient, and less prone to data corruption or inconsistencies. It allows for safe retries, handles network failures gracefully, and helps maintain data integrity in the face of concurrency and distributed system challenges.

Conclusion

Remember, idempotence is a property, and not a feature. It’s not something that you can turn on or off. It’s something that you need to design for and implement in your system.

When implemented correctly, idempotence can be a powerful tool for building reliable and fault-tolerant systems. It allows you to handle network failures gracefully, avoid unintended side effects, and maintain data integrity.

It also makes your system design process much simpler. If you can just retry an operation without worrying about unintended consequences, that makes it a good solution for cases when you need to handle failures gracefully.