<div dir="ltr">Hi <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">François,</span><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">  Ah.  Well, the step of deduplicating and finding connected nodes will not go away - OSRM requires a *graph*, not disconnected geometry, so you won't be able to get away from the problem of turning your LineStrings into connected data.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">  OSM's data structure (ways & nodes) has this structure built in, so OSRM doesn't contain any of this kind of logic already.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">  You might want to look at how pgRouting does this bit (they call it "building the topology") - <a href="http://docs.pgrouting.org/latest/en/pgRouting-concepts.html#build-a-routing-topology">http://docs.pgrouting.org/latest/en/pgRouting-concepts.html#build-a-routing-topology</a>, there might be some performance tips you can get from their approach.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">daniel</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 2, 2018 at 10:13 AM, François Lacombe <span dir="ltr"><<a href="mailto:fl.infosreseaux@gmail.com" target="_blank">fl.infosreseaux@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Daniel,<br><br><div><div class="gmail_extra"><div class="gmail_quote"><span class="">2018-03-02 18:31 GMT+01:00 Daniel Patterson <span dir="ltr"><<a href="mailto:daniel@mapbox.com" target="_blank">daniel@mapbox.com</a>></span>:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div>Well, it *could* be done.  It would all boil down to providing an alternative for `ParseOSMData` here:</div></span><span class=""><div>That function is responsible for parsing the OSM file, and copying/converting the OSM fields into a memory structure called `ExtractionContainers`:</div></span></div></blockquote><div><br></div><div>Thank you, this is really useful information<br> <br></div><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div>    0) You'll have to implement it - the core team have other priorities.</div></div></blockquote><div><br></div></span><div>That's fair<br></div><div>I could eventually propose a complete alternative to osrm-extract as to clearly separate responsibilities.<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>    1) I suspect you wouldn't see a huge performance improvement - I suspect the overhead of querying postgis would dominate the extractor time.</div></div></blockquote><div><br></div></span><div>I have about 5M ways and 40M nodes.<br></div><div>Producing xml or pbf takes hours, while querying postgis takes about 3min.<br></div><div>I would probably agree that the process of postgis output isn't well optimized<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>    2) You'll be on the hook for maintaining this code - the core team haven't built this into the core tool because we don't need it, and it's a big ask for us to maintain something we don't use.</div><div><br></div><div>  I'd strongly consider trying to optimize your Postgres->OSM extraction process.  Consider using `osmium` libraries to write out the data in PBF form directly instead of XML - it's significantly smaller, which makes it faster to move around and write to disk, and OSRM will import it more quickly.</div></div></blockquote><div><br></div></span><div>Problem isn't to get data from postgis, but to organize them as to fit in the xml :<br></div><div>- Creating nodes records, out of Linestrings / polygon geometries<br></div><div>- Search and deduplicate them, especially when 2 or more ways have nodes located on the same lat/lon point. Currently done with a GROUP BY on nodes geometry.<br></div><div>- Create numeric and auto incremented ids since we use uuid in postgis (the easy part)<br></div><div>- Iterate over all of this to produce a xml with Python. Didn't try c++ libosmium for now but I know i should. That's the longer part in the current process.<br></div><div><br></div><div>This takes hours, and I'll be really happy if I find a way to directly feed osrm graph without recreating such things.<br></div><div><br></div><div>Simple suppositions:<br></div><div>It would be so nice to not have to produce nodes out of geometries. It's the key point.<br></div><div>I guess you don't have proper records for each nodes in .osrm files don't you ?<br></div><div>Once they gone through profile's node_process, we only need their coordinates and not their tags any more.<br></div><div>Then it would be great to only send tagged nodes (coming from dedicated postgis tables) to osrm-extract.<br><br><br></div><div>Enjoy your weekend,<br><br></div><div>François<br></div></div></div></div></div>
<br>______________________________<wbr>_________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.<wbr>org/listinfo/osrm-talk</a><br>
<br></blockquote></div><br></div>