[OSM-talk] OSM Wikidata SPARQL service updated

[Yuri Astrakhan]

Sarah, how would I set the node cache file to the repserv.apply_diffs()?
The idx param is passed to the apply_file() - for the initial PBF dump
parsing, but I don't see any place to pass it for the subsequent diff
processing.  I assume there must be a way to run .apply_diff() that will
download the minute diff file, update node cache file with the changed
nodes, and afterwards call my way handler with the updated way geometries.

Also, I assume you meant dense_file_array, not dense_file_cache. So in my
case I would use one of these idx values when calculating way centroid, and
None otherwise:
sparse_mem_array
dense_mmap_array
sparse_file_array,my_cache_file
dense_file_array,my_cache_file

Thanks!


On Mon, Aug 14, 2017 at 4:31 PM, Sarah Hoffmann <lonvia at denofr.de> wrote:

> On Mon, Aug 14, 2017 at 11:10:39AM -0400, Yuri Astrakhan wrote:
> > mmd, the centroids are calculated with this code, let me know if there
> is a
> > better way, I wasn't aware of any issues with the minute data updates.
> >       wkb = wkbfab.create_linestring(obj)
> >       point = loads(wkb, hex=True).representative_point()
> > https://github.com/nyurik/osm2rdf/blob/master/osm2rdf.py#L250
>
> It doesn't look like you have any location cache included when
> processing updates, so that's unlikely to work.
>
> Minutely updates don't have the full node location information.
> If a way gets updated, you only get the new list of node ids.
> If the nodes have not changed themselves, they are not available
> with the update.
>
> If you need location information, you need to keep a persistent
> node cache in a file (idx=dense_file_cache,file.nodecache)
> and use that in your updates as well. It needs to be updated
> with the fresh node locations from the minutely change files
> and it is used to fill the coordinates for the ways.
>
> Once you have the node cache, you can get the geometries for
> updates ways. This is still only half the truth. If a node in
> a way is moved around, then this will naturally change the
> geometry of the way, but the minutely change file will have
> no indication that the way changed. Normally, these changes are
> relatively small and for some applications it is good enough
> to ignore them (Nominatim, the search engine, does so, for example).
> If you need to catch that case, then you also need to keep a
> persistent reverse index of which node is part of which way
> and for each changed node, update the ways it belongs to.
> There is currently no support for this in libosmium/pyosmium.
> So you would need to implement this yourself somehow.
>
> Kind regards
>
> Sarah
>
> >
> > Your query is correct, and you are right that (in theory) there shouldn't
> > be any ways without the center point. But there has been a number of ways
> > with only 1 point, causing a parsing error "need at least two points for
> > linestring". I will need to add some special handling for that
> > (suggestions?).
> >
> > You can see the error by adding this line:
> >    OPTIONAL { ?osmId osmm:loc:error ?err . }
> > The whole query --  http://tinyurl.com/ydf4qd62  (you can create short
> urls
> > with a button on the left side)
> >
> > On Mon, Aug 14, 2017 at 5:18 AM, mmd <mmd.osm at gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Am 13.08.2017 um 19:49 schrieb Yuri Astrakhan:
> > >
> > > > * all ways now store "osmm:loc" with centroid coordinates, making it
> > > > possible to crudely filter ways by location
> > >
> > > out of curiosity, can you say a few words on how your overall approach
> > > to calculate centroids for ways? As we all know it's an endless pain to
> > > get that information out of minutely diffs :)
> > >
> > > I have to say that I'm pretty much unfamiliar with SPARQL and just
> tried
> > > the following query. My expectation was that I won't get any results,
> > > making me wonder if my query has some issue?
> > >
> > > SELECT * WHERE {
> > >   ?osmId osmm:type 'w' .
> > >   FILTER NOT EXISTS { ?osmId osmm:loc ?osmLoc }.
> > > } LIMIT 100
> > >
> > >
> > > BTW: A quick search on Github yielded the following:
> > > https://github.com/nyurik/osm2rdf. Would that be the right place to
> look
> > > for more details?
> > >
> > > Best,
> > > mmd
> > >
> > >
> > > --
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > talk mailing list
> > > talk at openstreetmap.org
> > > https://lists.openstreetmap.org/listinfo/talk
> > >
>
> > _______________________________________________
> > talk mailing list
> > talk at openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20170820/38358a1a/attachment.html>