Idempotent POST

1

I noticed a very old thread about idempotent POST that went dead #764.

I think it is a common use case to have natural keys (uniqueness constraints that identify unique resources), but to allow the system to provide a generated id. I POST the resource and get a 201 Created with Location. Great.

Its also very common that this could happen many times concurrently or in quick succession. In this semantic I don’t want to return a 409 conflict (I may in other cases), I want the client logic to be race agnostic -or at least ambivalent. The semantic of course is data model dependent.

I see 2 choices (you may suggest others):

  1. Lie and say I created it the second/third/fourth time, even though I didn’t and return 201 Created + Location (same location each time)
  2. Be honest, return 201 Created+ Location for the first POST and 200 OK + Location (same location each time) for subsequent ones.

I think option (1) violates the spirit of the jsonapi spec, but make client handling very slightly easier, whereas option (2) appears to be acceptable, and the client handling is still pretty clean.

I’m pressing forward with (2) but very keen to hear thoughts and ideas from the community experienced in using jsonapi.

2

I have been in a similar situation, and have chosen (and been able) to return 409.

Neither of the other options are optimal. Both break the spec and therefore have the possibility of breaking clients, though IMHO they’d have to be fairly strict or make very specific assumptions in order for that to happen. In any case, make sure you clearly document any way your API deviates from the JSON:API spec.

A drawback of both approaches is that if someone POSTs with attribute A set to “foo” and the resource already exists with A set to “bar”, what do you return? Do you also update the resource and set A to “bar”, or do you skip updating the value and return the resource with A = “foo”?

Aside from these two options, my only suggestion is to reconsider if you really cannot return 409. Since you haven’t given concrete information about your use-case, only you can answer that. Could you change anything else to make it less likely that multiple clients are attempting to create the same resource?

1 Like
3

Hi, thanks for taking the time to reply.

Regarding considering 409.

In the case of a POST with a natural unique key and content that does not exactly replicate that which already exists - then it is not a question of idempotency, so i’d return 409. I think this is fine with the specification and do this already in many places when uniqueness constraints are violated.

If the intention is to PATCH atrributes to different values, the client should the retry with a PATCH. Hopefully the 409 carried enough information to allow the client to do this. I don’t think there is any built in syntax to support this, but this is not my problem today.

The special case that concerns me is that a second POST that is logically identical (modulo internal caches or aspects not consideed important for the semantics, such as a “created” timestamp), I want to return saying OK, you got what you wanted, but by the way it wasn’t you that did it (400 instead of 402).

If I instead returned a 409, there is no standard way to encourage the client not to give up, and most solutions would involve a lot of non-standard design of the error message, or a number of additional calls from the client. This, to me, feels like a lot of non-standard work to express something that is easily acheived with a 400 (and an explanation in my OPENAPI document of how to interpret the 400)

I’m not keen on the 402 because it is factually incorrect.

In this case I have automated processing of incoming messages that is splitting information across microservices - there are many-to-one relationships where the “one” is learned through presence in the incoming messages. Even if each thread or application queries first to see if it exists, it would be racing many other procesing workers. I feel the “create if required” semantic is pretty common.

4

Not a bad idea!

JSON:API supports code in the error, which is useful to govern client behavior. You can return a specific documented code with the 409 error when the resource exists (e.g. set code to resourceAlreadyExists), and the client can then use that to inform its next steps (either ignore it if they just wanted to make sure it exists, or do a GET request for the resource if they need it, or do a PATCH if they wanted to change some fields).

5

I’d be keen to understand if there are sets of error codes and object structures that have been standardised in some way. I’ve not seen resourceAlreadyExists in the jsonAPI standard.

Of course I’m aware I can make error codes and structures up. In the bad old days that was all we ever did, but then every single application behaved in its own special way all the time requiring a lot of careful investment in understanding the dark corners of the API spec.

Other than that, I have satisfied myself that my solution works for me, so thanks again for sharing your thoughts.

6

AFAIK this is not standardized in any way. You’d have to document this for your API(s).

Just in case, note that I’m not talking about HTTP response codes. You should stick to the standard ones. As for “structures”, I’m not sure what you mean, but as you likely know, JSON:API has a specific error format it uses. For telling the API clients how to behave, use the error object’s code property, which is the only one intended for programmatic parsing to differentiate various error conditions. If you need to supply any needed additional data, do that in the error’s meta.

7

Isn’t option (2) just PRG, Post/Redirect/Get, aka, GET-After-POST or Redirect-After-POST, by another form, i.e., Post/Create/Get. On returning 201 Created + Location, the client implementation performs a GET with a 200 OK on the resource. There could be an argument sending a proper Redirect code, 3xx, with 303 See Other being preferred. The HTTP spec states clients MAY fetch the resource in the Location header in Redirects (3xx), where as it is not elaborated on what a client MAY or MAY NOT do with the Location header in 201 Created.

If so, is the PRG pattern all we are really discussing when we talk about “Idempotent POST”? @PeaDubYa’s OP and the discussion may have made that clear to me for the first time.

8

Good question and thanks for your comment.

Option 2 does have strong similarities with PRG (I was not aware of the term, thank you!) - but in that approach (according to google), the POST response is ALWAYS a location and no body - forcing the GET.

I understand it is really a pattern to deal with browser behaviour, which is not my current concern in this case, but it also violates the jsonapi spec when a new resource (with new ID) is created:

“

If a POST request did not include a Client-Generated ID and the requested resource has been created successfully, the server MUST return a 201 Created status code.

The response SHOULD include a Location header identifying the location of the newly created resource.

The response MUST also include a document that contains the primary resource created.

“

Instead I was suggesting to include the Location header and the primary resource document in the response body irrespective of whether this was the first POST or not, but to return 200 OK when this didn’t happen to be the first attempt to create the resource.

I think this is OK with JSONAPI – when a “MUST” condition is not met:

“

A server MAY respond with other HTTP status codes.

“

In this case any POST which was not first for that natural key did not actually create a resource, so is not required to return a 201, and in my view, should not.

I guess I was seeking to develop an expectation that this should be a “normal” approach for this semantic. I did not want to use a redirect, because it does create a very different pattern to the “first time response” of 201.

The value of conforming to a standard such as jsonapi over something completely home-rolled is to reduce the cognitive load of the client developer. In this case I had to home roll a little, but I am sleeping easy :slight_smile:

1 Like
9

I see PRG walking up to the line, but not crossing it. It becomes a client implementation detail on what to do with the response body. In 3xx redirects it is established user agent behavior to perform the redirect. In 201, the presence of a Location header is expected in per RFC7231, and more explicit notes are present in RFC2616, i.e., SHOULD include a representation, MUST create before responding, SHOULD respond with 202 if it can’t, MAY include ETag, etc. RFC2616 includes a note on Content-Type too. (One is reading several RFCs to get to all of this. :slight_smile: )

A client implementation could use the Location header to perform the GET. The challenge, as you point out, is that a browser could render the JSON response, especially with a Content-Type: application/json; this would be undesirable :frowning:, but not typical behavior for most Tier 1, Class A, “modern” browsers. These would parse the response and likely populate the global namespace with the objects, besides performing any caching that might be relevant. And do we start a conversion here about namespacing JSON:api responses (or has that already happened … :thinking:

I liked your notes and thinking, I’m just wondering: as PRG was not a pattern before 2006—it feels stumbled upon—in trying to solve a common problem with double POSTs from the client. Did you stumble on something without a name yet? I’m not partial to “Idempotent POST” as it is an oxymoron per the specification. :stuck_out_tongue: