[OSM-dev] Overpass API delivers randomly invalid data

Gubler, Ruediger rgubler at init-ka.de
Mon Oct 31 08:03:14 GMT 2011


Hi,

I wrote a small test script to give you more help. 
The following results are reproducible: 

'<way id="39990889">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=8.509576000000006,49.34158799999985,8.559576000000007,49.39158799999985][@meta]'
'<way id="92738103">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=8.509576000000006,49.34158799999985,8.559576000000007,49.39158799999985][@meta]'
'<way id="24685758">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=8.559576000000007,49.34158799999985,8.609576000000008,49.39158799999985][@meta]'
'<way id="99176827">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=8.75957600000001,47.44158799999996,8.80957600000001,47.49158799999996][@meta]'
'<way id="11261866">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=8.959576000000013,48.74158799999989,9.009576000000013,48.791587999999884][@meta]'
'<way id="22781417">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.009576000000013,48.74158799999989,9.059576000000014,48.791587999999884][@meta]'
'<way id="10597166">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.059576000000014,48.4915879999999,9.109576000000015,48.5415879999999][@meta]'
'<way id="29363076">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.059576000000014,48.4915879999999,9.109576000000015,48.5415879999999][@meta]'
'<way id="8092073">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.109576000000015,48.84158799999988,9.159576000000015,48.89158799999988][@meta]'
'<way id="24059413">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.159576000000015,48.4415879999999,9.209576000000016,48.4915879999999][@meta]'
'<way id="62473930">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.159576000000015,48.4415879999999,9.209576000000016,48.4915879999999][@meta]'
'<way id="65385528">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.159576000000015,48.4415879999999,9.209576000000016,48.4915879999999][@meta]'
'<way id="32895245">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.159576000000015,48.4915879999999,9.209576000000016,48.5415879999999][@meta]'
'<way id="52035311">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=9.209576000000016,48.4415879999999,9.259576000000017,48.4915879999999][@meta]'
'<way id="134372194">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=10.809576000000039,48.391587999999906,10.85957600000004,48.4415879999999][@meta]'
'<way id="38245700">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=10.85957600000004,49.841587999999824,10.90957600000004,49.89158799999982][@meta]'
'<way id="38245704">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=10.85957600000004,49.841587999999824,10.90957600000004,49.89158799999982][@meta]'
'<way id="27287573">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=11.009576000000042,49.54158799999984,11.059576000000042,49.59158799999984][@meta]'
'<way id="52640474">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=11.359576000000047,48.4415879999999,11.409576000000047,48.4915879999999][@meta]'
'<way id="28996953">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=11.55957600000005,49.89158799999982,11.60957600000005,49.94158799999982][@meta]'
'<way id="5024389">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=13.059576000000071,50.64158799999978,13.109576000000072,50.691587999999776][@meta]'
'<way id="25762694">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=13.059576000000071,50.64158799999978,13.109576000000072,50.691587999999776][@meta]'
'<way id="30844055">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=13.509576000000077,48.29158799999991,13.559576000000078,48.34158799999991][@meta]'
'<way id="84378596">' looks wrong at query 'http://www.overpass-api.de/api/xapi?*[bbox=14.309576000000089,51.34158799999974,14.35957600000009,51.391587999999736][@meta]' 



Yours Rüdiger

-----Ursprüngliche Nachricht-----
Von: Gubler, Ruediger [mailto:rgubler at init-ka.de] 
Gesendet: Donnerstag, 27. Oktober 2011 08:20
An: 'Roland Olbricht'
Cc: dev at openstreetmap.org
Betreff: Re: [OSM-dev] Overpass API delivers randomly invalid data

Hi,

we tried to download a huge area which is done with a groovy script. 
The download starts at 25.10.11 15:09:37 GMT+2 and the error was detected at 26.10.11 01:50:28 GMT+2
I can't say which paricular query is failed but i have a node ID wich was wrong: 679111231 
The error is reproducable but it will cause at changing elements (nodes, ways, relations) and different ID's. 
There are only a few elements broken. The surrounding elements are OK. 

Here is the download part of our sript:

MIN_X_Property = 8.059576
MIN_Y_Property = 46.741588
MAX_X_Property = 14.670413
MAX_Y_Property = 51.430882
CHUNK_SIZE = 0.05
PrintWriter pw = new PrintWriter(raw_unsortedFile, "UTF-8")
pw.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
pw.println("<osm version=\"0.6\">")
processErrorStatus = createJobsAndDownloadData(MIN_X_Property, MAX_X_Property, MIN_Y_Property, MAX_Y_Property, CHUNK_SIZE, pw, Master_Logfile)
pw.println("</osm>")
pw.close()

def int createJobsAndDownloadData(BigDecimal min_x, BigDecimal max_x, BigDecimal min_y, BigDecimal max_y, BigDecimal chunk_size, PrintWriter destinationPrintWriter, File masterLogFile)
{
  // Define a closure to log to the console as well as in our logfile
  def logALine = { String logMessage ->
    println(logMessage)
    masterLogFile.append(logMessage + "\r\n")
  }
  
  def int processErrorStatus = 0;

  // let's start
  double job_min_x = min_x
  double job_max_x = min_x + chunk_size
  boolean break_x = false
  //long count_x = 0L
  while(job_max_x <= max_x + chunk_size && !break_x)
  {
    if(job_max_x > max_x)
    {
      job_max_x = max_x
      break_x = true
    }
    double job_min_y = min_y
    double job_max_y = min_y + chunk_size
    boolean break_y = false
    //long count_y = 0L
    while(job_max_y <= max_y + chunk_size && !break_y)
    {
      if(job_max_y > max_y)
      {
        job_max_y = max_y
        break_y = true
      }

      URL api_url = null
      try {
        api_url = new URL("http://www.overpass-api.de/api/xapi?*[bbox=" + job_min_x + "," + job_min_y + "," + job_max_x + "," + job_max_y + "][@meta]");
        logALine("Calling: " + api_url.toString())
        URLConnection yc = api_url.openConnection()
        BufferedReader br = new BufferedReader(new InputStreamReader(yc.getInputStream(), "UTF-8"))
        String inputLine;
        while ((inputLine = br.readLine()) != null)
          if(!inputLine.contains("<osm"))
            if(!inputLine.contains("</osm"))
              if(!inputLine.contains("<?xml"))
                if(!inputLine.contains("<bound"))
                  if(!inputLine.contains("<note"))
                    if(!inputLine.contains("<meta"))
                      destinationPrintWriter.println(inputLine)
        br.close()

        // wait to jolly OSM server along
        Thread.sleep(1000)
      }
      catch(e) {
        processErrorStatus = -1
        logALine ("Error at URL: " + api_url.toString())
      }
      if(processErrorStatus != 0)
      {
        return processErrorStatus
      }

      //count_y++
      job_min_y = job_max_y
      job_max_y = job_max_y + chunk_size
    }
    //count_x++
    job_min_x = job_max_x
    job_max_x = job_max_x + chunk_size
  }
  
  return processErrorStatus
}


Yours Rüdiger



-----Ursprüngliche Nachricht-----
Von: Roland Olbricht [mailto:roland.olbricht at gmx.de] 
Gesendet: Mittwoch, 26. Oktober 2011 19:20
An: dev at openstreetmap.org
Cc: Gubler, Ruediger
Betreff: Re: [OSM-dev] Overpass API delivers randomly invalid data

Hi,

> We are querying data for a bounding box and randomly entries are delivered
>  as 
>           <way id="30134048">
> or
>           <node id="31881194" lat="47.5137295" lon="11.3250178"/>
>  
> The entries are cut before the version attribute. If we query the entries
>  with the id all attributes are delivered. 

I've tried to reproduce this. But all changesets I have receieved have 
complete meta data.

So could you please give more detailed information, in particular

- At what time as exactly as possible have you tried to download? There have 
been an outage on Sunday including a backlog for meta data until Monday 
evening, and I would need to figure out whether it is related to this or not.

- Which query exactly failed?
- Does is fail sometimes or reproducably?
- Are in a damaged response no meta data at all or is it partly missing?

Best regards,

Roland

_______________________________________________
dev mailing list
dev at openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev



More information about the dev mailing list