[OSM-talk] planet2mysql.rb

[Jon Bright]

Jon Bright wrote:
> 
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
>  
>  28322 sircus    35  19 1570m 1.5g 1284 R   64 77.3 665:29.71 ruby
> 
> Is it normal for planet2mysql to take 1.5GB memory?  

I killed it at 1.7GB.  Thinking that this might be somehow related to
UTF-8 problems, I built and ran UTF8sanitizer over the file.  That
doesn't seem to be it, though:

$ ./UTF8sanitizer <~/planet-070718.osm >~/planet.sane 2>~/planet.errors

$ ls -l planet*
-rw-rw-r-- 1 sircus sircus 4977112403 2007-07-18 09:02 planet-070718.osm
-rw-rw-r-- 1 sircus sircus 4977112403 2007-07-29 14:07 planet.sane
-rw-rw-r-- 1 sircus sircus        104 2007-07-29 14:07 planet.errors

$ sha1sum planet*
4ec3786a631e56c1104965bcc6a74ab30be4068b  planet-070718.osm
4ec3786a631e56c1104965bcc6a74ab30be4068b  planet.sane
0733e63075c6c7d19acf9b3bcf2b8bde7581579a  planet.errors

$ cat planet.errors
Summary:
chars1: 4868864068
chars2: 214851
chars3: 2618
chars4: 0
chars5: 0
chars6: 0
lines : 107810779

IOW, UTF8sanitizer hasn't changed anything.  I assume this to mean that
the original isn't broken.  So I'm back to wondering why planet2mysql
was taking so much memory...

--
Jon