[OSRM-talk] map matching service for a huge number of requests

[Sasha Khapyorsky]

Technically it is possible (for example
https://stackoverflow.com/questions/145270/calling-c-c-from-python),
but I'm pretty sure that you will need to create sort of c++ envelop,
shared lib, etc.. It would be easier just to do what you need in c++
instead of python.

Sasha

On Fri, May 11, 2018 at 9:37 PM, Patrick Agin <agin.patrick at gmail.com> wrote:
> thanks Sasha. And do you know if it's possible to do calls to libosrm
> functions directly from Python?
>
> 2018-05-11 14:33 GMT-04:00 Sasha Khapyorsky <sashakh at gmail.com>:
>>
>> Hi again, Patrick,
>>
>> On Fri, May 11, 2018 at 9:22 PM, Patrick Agin <agin.patrick at gmail.com>
>> wrote:
>> >
>> > And are you aware of some python code that would do the calls to
>> > osrm-routed
>> > in parallel threads?
>>
>> There are lot of examples of how to make python things in parallel
>> threads. For example:
>> https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python
>>
>> Sasha
>>
>> > Regards,
>> > Patrick
>> >
>> > 2018-05-11 14:07 GMT-04:00 Daniel Patterson <daniel at mapbox.com>:
>> >>
>> >> Patrick,
>> >>
>> >>   There are about a million possible paths you could take here, a lot
>> >> of
>> >> it will depend on what skills you have available.  Off the top of my
>> >> head:
>> >>
>> >>     1) Speed things up by avoiding HTTP overhead and calling the
>> >> libosrm.a
>> >> functions directly instead of hitting `osrm-routed` over HTTP
>> >>     2) Modify the OSRM C++ source code and strip out the parts of the
>> >> map-matching response you don't need
>> >>     3) Simplify your trace geometries to speed up map-matching
>> >>     4) Break your trace list into sets and run these on multiple
>> >> machines
>> >> in parallel (make copies of the OSRM data onto multiple machines)
>> >>     5) Just wait 10 hours, and get a good nights sleep
>> >>
>> >>   libosrm.a is thread-safe, so if you're calling functions from threads
>> >> you can do many at once.
>> >>
>> >>   osrm-routed is multi-threaded, so you can run many queries in
>> >> parallel -
>> >> how many will depend on how many CPUs your machine has.  Profiling
>> >> multi-threaded server performance is kind of beyond the scope of OSRM
>> >> itself, there is lots of literature on it.
>> >>
>> >> daniel
>> >>
>> >> On Fri, May 11, 2018 at 10:57 AM, Patrick Agin <agin.patrick at gmail.com>
>> >> wrote:
>> >>>
>> >>> Sorry for the newbie question but what's the difference between
>> >>> osrm-routed and libosrm? Is it mandatory to use the latter for a
>> >>> parallel
>> >>> usage? And do you have an example of code that does the calls in
>> >>> parallel
>> >>> threads? Thanks Sasha for your help.
>> >>> Patrick
>> >>>
>> >>> 2018-05-11 13:50 GMT-04:00 Sasha Khapyorsky <sashakh at gmail.com>:
>> >>>>
>> >>>> Hi Patrick,
>> >>>>
>> >>>> If you are using libosrm (which reported to be thread safe:
>> >>>> https://github.com/Project-OSRM/osrm-backend/issues/4966) you can
>> >>>> just
>> >>>> split your list and run its parts in different parallel threads.
>> >>>>
>> >>>> Sasha
>> >>>>
>> >>>> On Fri, May 11, 2018 at 8:14 PM, Patrick Agin
>> >>>> <agin.patrick at gmail.com>
>> >>>> wrote:
>> >>>> > Hi,
>> >>>> > I want to call the map matching service for nearly a million of
>> >>>> > routes. I
>> >>>> > calculated that the task could take 10 hours on my laptop to
>> >>>> > complete.
>> >>>> > I was
>> >>>> > wondering if there's someone who faced the same problem and who
>> >>>> > addressed it
>> >>>> > in parallel, maybe with Spark? Another tool or way?
>> >>>> > Any hint is greatly appreciated!
>> >>>> > Patrick
>> >>>> >
>> >>>> > _______________________________________________
>> >>>> > OSRM-talk mailing list
>> >>>> > OSRM-talk at openstreetmap.org
>> >>>> > https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>>> >
>> >>>>
>> >>>> _______________________________________________
>> >>>> OSRM-talk mailing list
>> >>>> OSRM-talk at openstreetmap.org
>> >>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> OSRM-talk mailing list
>> >>> OSRM-talk at openstreetmap.org
>> >>> https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> OSRM-talk mailing list
>> >> OSRM-talk at openstreetmap.org
>> >> https://lists.openstreetmap.org/listinfo/osrm-talk
>> >>
>> >
>> >
>> > _______________________________________________
>> > OSRM-talk mailing list
>> > OSRM-talk at openstreetmap.org
>> > https://lists.openstreetmap.org/listinfo/osrm-talk
>> >
>>
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>