[Tile-serving] [osm2pgsql-dev/osm2pgsql] free(): invalid size Aborted Core dumped (Issue #2117)

Tue Dec 26 19:41:13 UTC 2023

**SUMMARY**
Im trying to get an all English titles map.  This is a newly built system, that has imported the whole planet quite easily with a standard command:  it resulted in a fully functional map, with non english titles on asia / north africa etc.. totally normal.  I tried using [Localization functions for Openstreetmap (osml10n)](https://github.com/giggls/osml10n) to get english titles, and it always core dumps.  In examining 
in a bdebuuger and valgrind it looks like it could be in osm2pgsql, and not the other project.  So im submitting here to see.

**DETAILS**

when i try to use the [Localization functions for Openstreetmap (osml10n)](https://github.com/giggls/osml10n) it fails with "free(): invalid size Aborted (core dumped)". (see below)  Ive tried this on a number of different extracts from geofabrik and did check the md5 on all.

```
postgres at pb5:/osm2/osm2pgsql/build$ ./osm2pgsql -G -O flex -d gis -S openstreetmap-carto-hstore-only-l10n.lua ./japan-latest.osm.pbf
2023-12-26 15:14:51  osm2pgsql version 1.10.0 (1.10.0-9-g4a8f42b5)
2023-12-26 15:14:51  Database version: 16.1 (Ubuntu 16.1-1.pgdg22.04+1)
2023-12-26 15:14:51  PostGIS version: 3.4
2023-12-26 15:14:51  Storing properties to table '"public"."osm2pgsql_properties"'.
2023-12-26 15:14:52  WARNING: The 'area' column type is deprecated. Please read
2023-12-26 15:14:52  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
2023-12-26 15:14:52  WARNING: The add_row() function is deprecated. Please read
2023-12-26 15:14:52  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
Processing: Node(11840k 538.2k/s) Way(0k 0.00k/s) Relation(0 0.0/s)free(): invalid size
Aborted (core dumped)
```

I recognize that its possible that the issue is with the other project, so i ran this in gdb, trying to narrow it down

```
Processing: Node(11420k 571.0k/s) Way(0k 0.00k/s) Relation(0 0.0/s)free(): invalid size
Thread 1 "osm2pgsql" received signal SIGABRT, Aborted.
```

This didnt show me alot, so i tried valgrind.  The error is in a lua utility (lua-utils.cpp:219) and it could be something on the osm2pgsql side. Valgrind details below..  

(ill acknowledge that some of these tests are with different PBF files, i lost some of my testing sessions, but all sessions resulted in the same regardless of the PBF file used)

```

`postgres at pb5:/osm2/osm2pgsql/build$ valgrind ./osm2pgsql -G -O flex -d gis -S /home/osm/openstreetmap-carto/openstreetmap-carto-hstore-only-l10n.lua ./israel-and-palestine-latest.osm.pbf
==61966== Memcheck, a memory error detector
==61966== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==61966== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==61966== Command: ./osm2pgsql -G -O flex -d gis -S /home/osm/openstreetmap-carto/openstreetmap-carto-hstore-only-l10n.lua ./israel-and-palestine-latest.osm.pbf
==61966== 
2023-12-26 15:41:31  osm2pgsql version 1.10.0 (1.10.0-9-g4a8f42b5)
2023-12-26 15:41:32  Database version: 16.1 (Ubuntu 16.1-1.pgdg22.04+1)
2023-12-26 15:41:32  PostGIS version: 3.4
2023-12-26 15:41:32  Storing properties to table '"public"."osm2pgsql_properties"'.
2023-12-26 15:41:33  WARNING: The 'area' column type is deprecated. Please read
2023-12-26 15:41:33  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
2023-12-26 15:41:36  WARNING: The add_row() function is deprecated. Please read
2023-12-26 15:41:36  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
Processing: Node(11870k 6.3k/s) Way(0k 0.00k/s) Relation(0 0.0/s)==61966== Invalid free() / delete / delete[] / realloc()
==61966==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==61966==    by 0x601E1A9: ssh_buffer_free (in /usr/lib/x86_64-linux-gnu/libssh.so.4.8.7)
==61966==    by 0x485EE5B: freelist_free (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x486165E: algf_gsub (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x4CC1AD5: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CCBF83: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1FC7: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBA746: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBCFDA: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1E9D: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x339BD9: luaX_pcall(lua_State*, int, int) (lua-utils.cpp:219)
==61966==    by 0x24FB09: output_flex_t::call_lua_function(prepared_lua_function_t, osmium::OSMObject const&) (output-flex.cpp:916)
==61966==  Address 0x1ffefff300 is on thread 1's stack
==61966==  in frame #3, created by algf_gsub (???:)
==61966== 
==61966== Invalid free() / delete / delete[] / realloc()
==61966==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==61966==    by 0x485EE5B: freelist_free (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x486165E: algf_gsub (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x4CC1AD5: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CCBF83: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1FC7: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBA746: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBCFDA: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1E9D: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x339BD9: luaX_pcall(lua_State*, int, int) (lua-utils.cpp:219)
==61966==    by 0x24FB09: output_flex_t::call_lua_function(prepared_lua_function_t, osmium::OSMObject const&) (output-flex.cpp:916)
==61966==    by 0x24FC51: output_flex_t::get_mutex_and_call_lua_function(prepared_lua_function_t, osmium::OSMObject const&) (output-flex.cpp:928)
==61966==  Address 0x1ffefff1e0 is on thread 1's stack
==61966==  in frame #2, created by algf_gsub (???:)
==61966== 
Processing: Node(11876k 6.3k/s) Way(80k 0.16k/s) Relation(0 0.0/s)^C==61966== 
==61966== Process terminating with default action of signal 2 (SIGINT)
==61966==    at 0x50B3117: __futex_abstimed_wait_common64 (futex-internal.c:57)
==61966==    by 0x50B3117: __futex_abstimed_wait_common (futex-internal.c:87)
==61966==    by 0x50B3117: __futex_abstimed_wait_cancelable64 (futex-internal.c:139)
==61966==    by 0x50B5A40: __pthread_cond_wait_common (pthread_cond_wait.c:503)
==61966==    by 0x50B5A40: pthread_cond_wait@@GLIBC_2.3.2 (pthread_cond_wait.c:627)
==61966==    by 0x197610: void std::condition_variable::wait<osmium::thread::Queue<osmium::thread::function_wrapper>::wait_and_pop(osmium::thread::function_wrapper&)::{lambda()#1}>(std::unique_lock<std::mutex>&, osmium::thread::Queue<osmium::thread::function_wrapper>::wait_and_pop(osmium::thread::function_wrapper&)::{lambda()#1}) (condition_variable:103)
==61966==    by 0x18DA92: osmium::thread::Queue<osmium::thread::function_wrapper>::wait_and_pop(osmium::thread::function_wrapper&) (queue.hpp:188)
==61966==    by 0x2496DA: thread_pool_t::worker_thread(unsigned int) (thread-pool.cpp:57)
==61966==    by 0x24A66D: void std::__invoke_impl<void, void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int>(std::__invoke_memfun_deref, void (thread_pool_t::*&&)(unsigned int), thread_pool_t*&&, unsigned int&&) (invoke.h:74)
==61966==    by 0x24A587: std::__invoke_result<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int>::type std::__invoke<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int>(void (thread_pool_t::*&&)(unsigned int), thread_pool_t*&&, unsigned int&&) (invoke.h:96)
==61966==    by 0x24A4A6: void std::thread::_Invoker<std::tuple<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) (std_thread.h:259)
==61966==    by 0x24A43F: std::thread::_Invoker<std::tuple<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int> >::operator()() (std_thread.h:266)
==61966==    by 0x24A41F: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int> > >::_M_run() (std_thread.h:211)
==61966==    by 0x4DC9252: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==61966==    by 0x50B6AC2: start_thread (pthread_create.c:442)
==61966== 
==61966== HEAP SUMMARY:
==61966==     in use at exit: 305,255,741 bytes in 8,837 blocks
==61966==   total heap usage: 15,449,926 allocs, 15,441,453 frees, 9,041,668,476 bytes allocated
==61966== 
==61966== LEAK SUMMARY:
==61966==    definitely lost: 115,456 bytes in 181 blocks
==61966==    indirectly lost: 0 bytes in 0 blocks
==61966==      possibly lost: 207,856 bytes in 1,599 blocks
==61966==    still reachable: 304,932,429 bytes in 7,057 blocks
==61966==         suppressed: 0 bytes in 0 blocks
==61966== Rerun with --leak-check=full to see details of leaked memory
==61966== 
==61966== For lists of detected and suppressed errors, rerun with: -s
==61966== ERROR SUMMARY: 364 errors from 2 contexts (suppressed: 0 from 0)

`
```

## What version of osm2pgsql are you using?

osm2pgsql version 1.10.0 (1.10.0-9-g4a8f42b5)

## What operating system and PostgreSQL/PostGIS version are you using?

<!-- Also what Linux distribution if applicable, OS version? -->
Ubuntu 22
Linux HOSTNAME 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

## Tell us something about your system
2 CPU / 10 Cores each / 40 Threads Total
bare metal
386 GB Ram.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/issues/2117
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/issues/2117 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231226/5a04e0d0/attachment.htm>