Gracefully handling failures in sequence of requests

Question

I am designing architecture for service like pastebin.com. I plan to store user data on S3 storage and metadata with my own server. So whenever a user requests for some paste creation there are two steps to this:

Store content in S3
Store metadata with local metadata db.

I am thinking what would happen if step 1 succeeds and step 2 fails. When it happens, I have data paste in S3 which I don't have a way to access it, and user will get a failure response back, and he/she can retry and then may get succeeded. But the initial failure has created some data in my storage, and I don't have any metadata and any way to access it.
My question
When we have to perform 2-3 activities where one happens after another, and some activity in between fails, how can we rollback?

Ralf Kleberhoff · Answer

There are three aspects of failure handling applying to your case:

Informing the service user (important). Giving a failure response is the correct way to go. Normally, your user isn't interested in the details why the call failed, just the fact that it failed. That aspect is important and you already do it.
Leaving the service in a consistent state (important). It's important to leave your service in a state where it can accept new requests (as far as that's possible, depending on the type of failure). Having additional, inaccesible data in some storage typically won't affect future requests. That aspect is important, and I bet your code behaves well in that respect.
Cleaning up unused resources (optional). Filling storage with inaccessible data is undesirable. You can either try to remove the data in your error handler or create something like a nightly garbage-collection job that scans your storage for inaccessible data. But if resource cost isn't extraordinarily high, you can as well simply live with a small percentage of garbage. It's up to your decision if you want to spend time on this aspect.

Gracefully handling failures in sequence of requests

One Answer

Add your own answers!

Ask a Question