At 16:05 on 15 September 2017 I added a loop that calls the post route ten times.

The route belongs to a cricket scores tracker of mine. The first time anyone opens a match page it tweets the match, once, and a Redis key remembers that it has. The loop came with the fifth version of that dedupe in one day. Before it, the file ran itself with one call. I replayed all five. With one call, the first four each tweet once. With ten calls at once, they each tweet ten times, and the fifth tweets once.

Five versions

At 11:55 the route asks whether the key exists, tweets, and sets the key with SETEX after the tweet succeeds. At 12:22 the check moved inside a MULTI. It is a transaction around one read. The reply still comes back to a callback, and the callback still decides. At 12:33 the check came back out, with a WATCH in front of it. The write after the tweet went into a MULTI of its own.

At 12:41 the write moved ahead of the tweet:

redis.multi().setex(redisKey, 24 * 60 * 60, 1).exec(function(err, replies) {
  if (!err && replies.length) {
    tweets.tweet(req, message);

It looks like a claim. The key is written before anything is posted. But SETEX sets the key whether or not it was there, and answers OK either way.

At 16:05:

redis.multi()
  .setnx(redisKey, 1)
  .expire(redisKey, 1 * 24 * 60 * 60)
  .exec(function(err, replies) {
    if (!err && replies && replies.length && replies[0] === 1) {
      tweets.tweet(req, message);
    } else {
      util.log('[tweets][post] request skipped: ' + (err ? err.message : redisKey))
    }
  });

SETNX sets the key only if it is absent, and its reply says which happened.

I ran all five against Redis 7.4 with the client the lockfile pins, redis 2.8.0, on one connection, with the tweet and the link shortener stubbed. All ten calls leave before the first reply is read, so in every version the exists check answers 0 ten times. The check never has a chance. What differs is the write’s reply. Until 12:41 there is no write before the tweet. At 12:41 it answers OK, ten times, and ten tweets go out. At 16:05 it answers 1 once and 0 nine times. One tweet, nine lines in the log. Five runs of each gave the same counts.

The WATCH is still in the final file, and an UNWATCH with it. I deleted both and ran it again: one tweet, nine skips. On one connection it changes nothing.

The claim is taken before the tweet, and nothing gives it back. I made the stubbed tweet fail. The key stayed, with a day on its clock, and the next call was told the match had already been posted. At 11:55, with the key written after the tweet, the same failure left no key, and the next call tried again. At 16:05 a failed tweet is a match that goes unannounced until tomorrow. For a cricket score that is the right way round. Nobody notices the tweet that never came. Everybody notices two.

The row is the claim

Designing a wallet keeps the same kind of claim: an idempotency key in Redis with a 24-hour TTL, and a sentence saying that claiming it with SETNX before the work starts closes the gap between a retry and the request still running. It does. The sentence stops before the two things the scraper settles by accident: what the loser is told, and what a failed run leaves behind. The scraper’s answers are “skipped” and “a day”. A top-up can take neither.

One top-up can arrive three times. The app retries, two workers take the same job, the bank’s callback fires twice. The app’s request and the bank’s callback carry different ids for the same top-up, so a key on either misses the other. The key has to be the top-up.

The wallet already writes a row for it before any money moves: a ledger entry with a RefID, status initiated. That row can be the claim.

create unique index on ledger (user_id, type, ref_id);  -- all three not null

insert into ledger (user_id, type, ref_id, amount)
values ('u1', 'credit', 'topup-42', 500000)
on conflict do nothing
returning id;

I released three sessions at the same instant, on PostgreSQL 16 at read committed. One insert returned a row. Two returned nothing. All three then read the same row and saw initiated. With a plain insert, the two losers got a duplicate-key error, 23505.

Then the check-first shape, in SQL this time: if not exists, then insert, no index. Ten trials wrote three rows nine times and two rows once. With the index back and a 0.3 s pause staged between check and write, all three checks said absent, and the write refused two of them.

Last, the winner inserts and holds its transaction open for three seconds. The other two sessions stop. Postgres shows both waiting on the winner’s transaction. When the winner rolls back, one of them gets the row and the other gets nothing. When it commits, both get nothing. A rollback takes the claim with it. The key at 16:05 had no such tie. It outlived its failed tweet by a day.

Where it stops

A rollback cannot take back the call to the bank. So the row commits as initiated first and the call goes out after. If the worker dies between the two, the claim is committed and nobody is working it. A retry inserts, gets nothing back, and reads initiated.

That is the scraper’s key again, except that it is a row with a status, and it does not expire tomorrow. The status check in the wallet post walks rows stuck in processing, and this row never got there.

Nine of the ten calls ended as a line in the log. In the ledger, the loser reads initiated. Someone claimed it. Whether anyone is still working on it, the row does not say.