[OSM-dev] Overpass API delivers randomly invalid data

[Gubler, Ruediger]

Hi,

we tried to download a huge area which is done with a groovy script. 
The download starts at 25.10.11 15:09:37 GMT+2 and the error was detected at 26.10.11 01:50:28 GMT+2
I can't say which paricular query is failed but i have a node ID wich was wrong: 679111231 
The error is reproducable but it will cause at changing elements (nodes, ways, relations) and different ID's. 
There are only a few elements broken. The surrounding elements are OK. 

Here is the download part of our sript:

MIN_X_Property = 8.059576
MIN_Y_Property = 46.741588
MAX_X_Property = 14.670413
MAX_Y_Property = 51.430882
CHUNK_SIZE = 0.05
PrintWriter pw = new PrintWriter(raw_unsortedFile, "UTF-8")
pw.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
pw.println("<osm version=\"0.6\">")
processErrorStatus = createJobsAndDownloadData(MIN_X_Property, MAX_X_Property, MIN_Y_Property, MAX_Y_Property, CHUNK_SIZE, pw, Master_Logfile)
pw.println("</osm>")
pw.close()

def int createJobsAndDownloadData(BigDecimal min_x, BigDecimal max_x, BigDecimal min_y, BigDecimal max_y, BigDecimal chunk_size, PrintWriter destinationPrintWriter, File masterLogFile)
{
  // Define a closure to log to the console as well as in our logfile
  def logALine = { String logMessage ->
    println(logMessage)
    masterLogFile.append(logMessage + "\r\n")
  }
  
  def int processErrorStatus = 0;

  // let's start
  double job_min_x = min_x
  double job_max_x = min_x + chunk_size
  boolean break_x = false
  //long count_x = 0L
  while(job_max_x <= max_x + chunk_size && !break_x)
  {
    if(job_max_x > max_x)
    {
      job_max_x = max_x
      break_x = true
    }
    double job_min_y = min_y
    double job_max_y = min_y + chunk_size
    boolean break_y = false
    //long count_y = 0L
    while(job_max_y <= max_y + chunk_size && !break_y)
    {
      if(job_max_y > max_y)
      {
        job_max_y = max_y
        break_y = true
      }

      URL api_url = null
      try {
        api_url = new URL("http://www.overpass-api.de/api/xapi?*[bbox=" + job_min_x + "," + job_min_y + "," + job_max_x + "," + job_max_y + "][@meta]");
        logALine("Calling: " + api_url.toString())
        URLConnection yc = api_url.openConnection()
        BufferedReader br = new BufferedReader(new InputStreamReader(yc.getInputStream(), "UTF-8"))
        String inputLine;
        while ((inputLine = br.readLine()) != null)
          if(!inputLine.contains("<osm"))
            if(!inputLine.contains("</osm"))
              if(!inputLine.contains("<?xml"))
                if(!inputLine.contains("<bound"))
                  if(!inputLine.contains("<note"))
                    if(!inputLine.contains("<meta"))
                      destinationPrintWriter.println(inputLine)
        br.close()

        // wait to jolly OSM server along
        Thread.sleep(1000)
      }
      catch(e) {
        processErrorStatus = -1
        logALine ("Error at URL: " + api_url.toString())
      }
      if(processErrorStatus != 0)
      {
        return processErrorStatus
      }

      //count_y++
      job_min_y = job_max_y
      job_max_y = job_max_y + chunk_size
    }
    //count_x++
    job_min_x = job_max_x
    job_max_x = job_max_x + chunk_size
  }
  
  return processErrorStatus
}


Yours Rüdiger



-----Ursprüngliche Nachricht-----
Von: Roland Olbricht [mailto:roland.olbricht at gmx.de] 
Gesendet: Mittwoch, 26. Oktober 2011 19:20
An: dev at openstreetmap.org
Cc: Gubler, Ruediger
Betreff: Re: [OSM-dev] Overpass API delivers randomly invalid data

Hi,

> We are querying data for a bounding box and randomly entries are delivered
>  as 
>           <way id="30134048">
> or
>           <node id="31881194" lat="47.5137295" lon="11.3250178"/>
>  
> The entries are cut before the version attribute. If we query the entries
>  with the id all attributes are delivered. 

I've tried to reproduce this. But all changesets I have receieved have 
complete meta data.

So could you please give more detailed information, in particular

- At what time as exactly as possible have you tried to download? There have 
been an outage on Sunday including a backlog for meta data until Monday 
evening, and I would need to figure out whether it is related to this or not.

- Which query exactly failed?
- Does is fail sometimes or reproducably?
- Are in a damaged response no meta data at all or is it partly missing?

Best regards,

Roland