[OSRM-talk] map matching service for a huge number of requests

[Patrick Agin]

Great, thanks again for your help.
Have a good week-end,
Patrick

2018-05-11 14:46 GMT-04:00 Sasha Khapyorsky <sashakh at gmail.com>:

> Technically it is possible (for example
> https://stackoverflow.com/questions/145270/calling-c-c-from-python),
> but I'm pretty sure that you will need to create sort of c++ envelop,
> shared lib, etc.. It would be easier just to do what you need in c++
> instead of python.
>
> Sasha
>
> On Fri, May 11, 2018 at 9:37 PM, Patrick Agin <agin.patrick at gmail.com>
> wrote:
> > thanks Sasha. And do you know if it's possible to do calls to libosrm
> > functions directly from Python?
> >
> > 2018-05-11 14:33 GMT-04:00 Sasha Khapyorsky <sashakh at gmail.com>:
> >>
> >> Hi again, Patrick,
> >>
> >> On Fri, May 11, 2018 at 9:22 PM, Patrick Agin <agin.patrick at gmail.com>
> >> wrote:
> >> >
> >> > And are you aware of some python code that would do the calls to
> >> > osrm-routed
> >> > in parallel threads?
> >>
> >> There are lot of examples of how to make python things in parallel
> >> threads. For example:
> >> https://stackoverflow.com/questions/2846653/how-to-use-
> threading-in-python
> >>
> >> Sasha
> >>
> >> > Regards,
> >> > Patrick
> >> >
> >> > 2018-05-11 14:07 GMT-04:00 Daniel Patterson <daniel at mapbox.com>:
> >> >>
> >> >> Patrick,
> >> >>
> >> >>   There are about a million possible paths you could take here, a lot
> >> >> of
> >> >> it will depend on what skills you have available.  Off the top of my
> >> >> head:
> >> >>
> >> >>     1) Speed things up by avoiding HTTP overhead and calling the
> >> >> libosrm.a
> >> >> functions directly instead of hitting `osrm-routed` over HTTP
> >> >>     2) Modify the OSRM C++ source code and strip out the parts of the
> >> >> map-matching response you don't need
> >> >>     3) Simplify your trace geometries to speed up map-matching
> >> >>     4) Break your trace list into sets and run these on multiple
> >> >> machines
> >> >> in parallel (make copies of the OSRM data onto multiple machines)
> >> >>     5) Just wait 10 hours, and get a good nights sleep
> >> >>
> >> >>   libosrm.a is thread-safe, so if you're calling functions from
> threads
> >> >> you can do many at once.
> >> >>
> >> >>   osrm-routed is multi-threaded, so you can run many queries in
> >> >> parallel -
> >> >> how many will depend on how many CPUs your machine has.  Profiling
> >> >> multi-threaded server performance is kind of beyond the scope of OSRM
> >> >> itself, there is lots of literature on it.
> >> >>
> >> >> daniel
> >> >>
> >> >> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin <
> agin.patrick at gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Sorry for the newbie question but what's the difference between
> >> >>> osrm-routed and libosrm? Is it mandatory to use the latter for a
> >> >>> parallel
> >> >>> usage? And do you have an example of code that does the calls in
> >> >>> parallel
> >> >>> threads? Thanks Sasha for your help.
> >> >>> Patrick
> >> >>>
> >> >>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky <sashakh at gmail.com>:
> >> >>>>
> >> >>>> Hi Patrick,
> >> >>>>
> >> >>>> If you are using libosrm (which reported to be thread safe:
> >> >>>> https://github.com/Project-OSRM/osrm-backend/issues/4966) you can
> >> >>>> just
> >> >>>> split your list and run its parts in different parallel threads.
> >> >>>>
> >> >>>> Sasha
> >> >>>>
> >> >>>> On Fri, May 11, 2018 at 8:14 PM, Patrick Agin
> >> >>>> <agin.patrick at gmail.com>
> >> >>>> wrote:
> >> >>>> > Hi,
> >> >>>> > I want to call the map matching service for nearly a million of
> >> >>>> > routes. I
> >> >>>> > calculated that the task could take 10 hours on my laptop to
> >> >>>> > complete.
> >> >>>> > I was
> >> >>>> > wondering if there's someone who faced the same problem and who
> >> >>>> > addressed it
> >> >>>> > in parallel, maybe with Spark? Another tool or way?
> >> >>>> > Any hint is greatly appreciated!
> >> >>>> > Patrick
> >> >>>> >
> >> >>>> > _______________________________________________
> >> >>>> > OSRM-talk mailing list
> >> >>>> > OSRM-talk at openstreetmap.org
> >> >>>> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>> >
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> OSRM-talk mailing list
> >> >>>> OSRM-talk at openstreetmap.org
> >> >>>> https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> OSRM-talk mailing list
> >> >>> OSRM-talk at openstreetmap.org
> >> >>> https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> OSRM-talk mailing list
> >> >> OSRM-talk at openstreetmap.org
> >> >> https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > OSRM-talk mailing list
> >> > OSRM-talk at openstreetmap.org
> >> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >> >
> >>
> >> _______________________________________________
> >> OSRM-talk mailing list
> >> OSRM-talk at openstreetmap.org
> >> https://lists.openstreetmap.org/listinfo/osrm-talk
> >
> >
> >
> > _______________________________________________
> > OSRM-talk mailing list
> > OSRM-talk at openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osrm-talk/attachments/20180511/db4b386e/attachment.html>