[OSM-dev] osmjs says 'Killed'

[Jochen Topf]

Hi!

I thought I had found all memory leaks but it seems I haven't. Unfortunately I
don't have time at the moment to hunt it down. I'll put it on my todo list, but
can't say when I'll get to it.

Jochen

On Wed, Nov 09, 2011 at 08:23:54AM -0700, Martijn van Exel wrote:
> Date: Wed, 9 Nov 2011 08:23:54 -0700
> From: Martijn van Exel <m at rtijn.org>
> To: Jochen Topf <jochen at remote.org>
> Cc: dev Openstreetmap <dev at openstreetmap.org>
> Subject: Re: [OSM-dev] osmjs says 'Killed'
> 
> Jochen,
> 
> I compiled with the GC flag and can confirm that osmjs's memory usage
> is definitely still growing - much slower, but that is probably due to
> the garbage collection processes making the whole thing slow. I ran
> that US pbf again and it croaked when its memory footprint was about
> 50% in top.
> 
> As you said without any further output it's hard to do any further
> useful debugging.
> 
> Martijn
> 
> On Wed, Nov 9, 2011 at 1:37 AM, Jochen Topf <jochen at remote.org> wrote:
> > On Wed, Nov 09, 2011 at 12:24:55AM -0700, Martijn van Exel wrote:
> >> I'm testing my osmium setup, osmjs specifically. I'm feeding osmjs
> >> small pbf planet extracts (city scale) and run the time.js script,
> >> that goes OK. Then I feed it a US pbf planet extract, and osmjs says
> >> 'Killed'  after about a minute or so. I tried -l none but that makes
> >> no difference. -d gives me nothing.
> >>
> >> Osmosis says the pbf is OK (I ran $OSMOSIS --rb ../us.pbf --wn and it
> >> did not give me any errors, I assume that means the file is OK).
> >>
> >> osmjs does not seem to have run out of memory. I don't have a lot
> >> (2GB) but that should not prevent osmjs from running this, should it?
> >
> > If you don't use -l or -m osmjs should not use a lot of memory. Maybe there
> > is a memory leak somewhere. Unfortunately thats hard to see, because the
> > Javascript engine will take all the memory it gets (up to some limit) and
> > only do garbage collection once that runs out. So in the "ps" output the
> > process will always use lots of memory and suddenly crash, when it can't
> > garbage collect any more. I don't remember what the message was in that
> > case but thats probably the "Killed" you see.
> >
> > You can compile with -DOSMIUM_V8_FORCE_GC to force a garbage collection after
> > each callback. This will make osmjs really slow, but you'll see the memory
> > for the process growing over time, which it shouldn't do.
> >
> > I run osmjs regularly on a germany extract and sometimes others and don't have
> > any problem like that. But it could be that the problem only shows up with, say
> > very long ways, or objects with many tags or so. There could be something
> > different about the US extract that triggers this. (Or maybe its just much
> > bigger...)
> >
> > Jochen
> > --
> > Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298
> >
> >
> 
> 
> 
> -- 
> martijn van exel
> geospatial omnivore
> 1109 1st ave #2
> salt lake city, ut 84103
> 801-550-5815
> http://oegeo.wordpress.com
> 

-- 
Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298