[openstreetmap/openstreetmap-website] Idempotency for API 0.6 (#2201)

mmd notifications at github.com
Sun Apr 7 11:35:37 UTC 2019


A minimum scope approach for JOSM could be as follows: it generates a new Idempotency-Key outside the retry look, and passes it as HTTP header for POST requests.

I also added some log file excerpts below to demo a chunked upload. Every single upload has its own Idempotency-Key. Towards the end, I shut down the Rails server, and JOSM did one unsuccessful post, followed by another 5 unsuccessful retries, each time sending the same Key.

Assuming we had some corresponding logic on the backend as well, this  would already improve today's situation. Of course there's always room for improvement, but I don't think this will add a massive burden to clients willing to implement this approach.

```
Index: src/org/openstreetmap/josm/io/OsmApi.java
===================================================================

 import java.io.StringReader;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.UUID;
 import java.util.function.Consumer;

... 

     /**
      * Generic method for sending requests to the OSM API.
      *
     protected final String sendRequest(String requestMethod, String urlSuffix, String requestBody, ProgressMonitor monitor,
             boolean doAuthenticate, boolean fastFail) throws OsmTransferException {
         int retries = fastFail ? 0 : getMaxRetries();
+   
+        // Generate new Idempotency-Key
+        String idempotencyKey = UUID.randomUUID().toString();
 
         while (true) { // the retry loop
             try {
                     addAuth(client);
                 }
 
+                if ("POST".equals(requestMethod)) {
+                    client.setHeader("Idempotency-Key", idempotencyKey);
+                    Logging.info("Using Idempotency-Key " + idempotencyKey);
+                }
+


```


```
2019-04-07 13:13:36.106 INFORMATION: OK
2019-04-07 13:13:36.208 INFORMATION: Using Idempotency-Key e8f69ee7-b958-4d95-a7bd-b0d964947cf5
changeset/1303/upload
2019-04-07 13:13:36.208 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (159 B) ...
2019-04-07 13:13:36.253 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload -> 200
2019-04-07 13:13:36.253 INFORMATION: OK
2019-04-07 13:13:36.355 INFORMATION: Using Idempotency-Key e8f40f0b-27c1-4934-93e3-3646f29dc245
changeset/1303/upload
2019-04-07 13:13:36.355 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.405 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload -> 200
2019-04-07 13:13:36.405 INFORMATION: OK
2019-04-07 13:13:36.507 INFORMATION: Using Idempotency-Key 973c9c41-fd18-4346-8536-00b55ab4e6e2
changeset/1303/upload
2019-04-07 13:13:36.507 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.553 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload -> 200
2019-04-07 13:13:36.553 INFORMATION: OK
2019-04-07 13:13:36.655 INFORMATION: Using Idempotency-Key 7f4c3f5b-3180-4dd9-b13b-f7e8d55246db
changeset/1303/upload
2019-04-07 13:13:36.655 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (159 B) ...
2019-04-07 13:13:36.701 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload -> 200
2019-04-07 13:13:36.701 INFORMATION: OK
2019-04-07 13:13:36.803 INFORMATION: Using Idempotency-Key ee95ef0f-e760-42fb-ac29-f25dcfeaff90
changeset/1303/upload
2019-04-07 13:13:36.803 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.804 INFORMATION: Using Idempotency-Key ee95ef0f-e760-42fb-ac29-f25dcfeaff90
changeset/1303/upload
2019-04-07 13:13:36.805 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.805 INFORMATION: Using Idempotency-Key ee95ef0f-e760-42fb-ac29-f25dcfeaff90
changeset/1303/upload
2019-04-07 13:13:36.806 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.806 INFORMATION: Using Idempotency-Key ee95ef0f-e760-42fb-ac29-f25dcfeaff90
changeset/1303/upload
2019-04-07 13:13:36.807 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.807 INFORMATION: Using Idempotency-Key ee95ef0f-e760-42fb-ac29-f25dcfeaff90
changeset/1303/upload
2019-04-07 13:13:36.807 INFORMATION: POST http://localhost:3000/api/0.6/changeset/1303/upload (160 B) ...
2019-04-07 13:13:36.808 INFORMATION: Using Idempotency-Key ee95ef0f-e760-42fb-ac29-f25dcfeaff90
changeset/1303/upload
```



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/issues/2201#issuecomment-480582206
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20190407/065b834f/attachment-0001.html>


More information about the rails-dev mailing list