<div dir="ltr">Hi Deepak,<div><br></div><div>Thanks for the suggestion.</div><div><br></div><div>I came across tippecanoe again : <a href="https://github.com/mapbox/tippecanoe">https://github.com/mapbox/tippecanoe</a></div><div>and finally seeing a use case for it. So, while the DB is still being loaded, started off tippecanoe program on the original .geojsonl.</div><div><br></div><div>The program was quite fast - in about 6 hrs it had generated a 1.4GB .mbtiles file and it didn't even take too much RAM.</div><div><br></div><div>Then, with the help of <a href="https://github.com/maptiler/tileserver-php">https://github.com/maptiler/tileserver-php</a>:</div><div>This vector tiles URL is working at my end in QGIS:<br></div><div><a href="https://server.nikhilvj.co.in/buildings/tileserver-php/tileserver.php?/index.json?/buildings-z13/{z}/{x}/{y}.pbf">https://server.nikhilvj.co.in/buildings/tileserver-php/tileserver.php?/index.json?/buildings-z13/{z}/{x}/{y}.pbf</a><br></div><div><br></div><div>I was looking for options to generate raster .png etc tiles, but haven't found one yet.</div><div><br></div><div>I also saw an option in tippecanoe to output .pbf tiles to a folder instead of a single .mbtiles file. Did a couple of runs, but that hasn't worked out.</div><div><br></div><div>Meanwhile, Sreeram-K posted in OSM-India telegram group: folks at <a href="http://mapwith.ai">mapwith.ai</a> have incorporated the data and are now showing it in their rapID editor alongwith the AI roads data, as suggestions for openstreetmap editing. One link:</div><div><a href="https://mapwith.ai/rapid#background=none&datasets=fbRoads,msBuildings&disable_features=boundaries&map=16.17/13.09066/80.27361">https://mapwith.ai/rapid#background=none&datasets=fbRoads,msBuildings&disable_features=boundaries&map=16.17/13.09066/80.27361</a></div><div><br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>--<br>Cheers,<br>Nikhil VJ<br><a href="https://nikhilvj.co.in" target="_blank">https://nikhilvj.co.in</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 24, 2022 at 1:48 PM Deepak Sharda <<a href="mailto:deepsharda@gmail.com">deepsharda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hello Nikhil,<div dir="auto"><br></div><div dir="auto">I would definitely love to collobrate. The best i would suggest is to make Mbtiles and host them as data is not going to change a lot so most of the time static. </div><div dir="auto"><br></div><div dir="auto">Rendering from hosted Mbtiles will not only fast but easy to handle on small web server. </div><div dir="auto"><br></div><div dir="auto">How to proceed.</div><div dir="auto"><br></div><div dir="auto">First creation of Mbtiles</div><div dir="auto">second Hosting of Mbtiles</div><div dir="auto">Third Styling at client side.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Thanks</div><div dir="auto"><br></div><div dir="auto"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 24 May, 2022, 1:03 pm Nikhil VJ, <<a href="mailto:nikhil.js@gmail.com" target="_blank">nikhil.js@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Bhibhash,<div><br></div><div>Short answer: No.</div><div>There is no metadata. I took the top 1000 lines and visualized it : the shapes were spread all over the country and there wasn't any order in them.</div><div><br></div><div>With an intern's help I've started import of them into a portable postgresql DB (dockerized with a persistent volume) on my webserver. Started yesterday and it's 10 million shapes in; from size comparisons it'll probably be around 79 million shapes, so should be imported in a week's time. </div><div><br></div><div>I've set an auto-increment primary key column in the DB table, so once imported at least the data can be pulled out in batches.</div><div><br></div><div>There are 2 paths ahead once I've imported it:</div><div>-------</div><div><br></div><div><b>1. Serving vector tiles</b></div><div>I want to serve MVT vector tiles from this DB - and then have a map display the same. That way one can directly have this stuff render on a map as they browse it - fetching only one small region at a time and hopefully not being too slow.</div><div><br></div><div>Finding some references here:</div><div><a href="https://blog.jawg.io/how-to-make-mvt-with-postgis/" rel="noreferrer" target="_blank">https://blog.jawg.io/how-to-make-mvt-with-postgis/</a><br></div><div><a href="https://www.crunchydata.com/blog/dynamic-vector-tiles-from-postgis" rel="noreferrer" target="_blank">https://www.crunchydata.com/blog/dynamic-vector-tiles-from-postgis</a><br></div><div><a href="https://medium.com/@frederic.rodrigo/web-mapping-comparing-vector-tile-servers-from-postgres-postgis-405055e69084" rel="noreferrer" target="_blank">https://medium.com/@frederic.rodrigo/web-mapping-comparing-vector-tile-servers-from-postgres-postgis-405055e69084</a></div><div> </div><div>I'm new to this MVT business, so if anybody wants to collaborate pls feel free to jump in.</div><div><br></div><div>------</div><div><br></div><div><b>2. Tag with region ids</b></div><div>Add columns to the DB table and tag each shape by state, district ids etc using Within operations either in the DB or in python-geopandas.</div><div><br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>--<br>Cheers,<br>Nikhil VJ<br><a href="https://nikhilvj.co.in" rel="noreferrer" target="_blank">https://nikhilvj.co.in</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 24, 2022 at 10:14 AM Bibhash Nath <<a href="mailto:bibhash12@gmail.com" rel="noreferrer" target="_blank">bibhash12@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks for sharing.<div>Do we have state-wise datasets? It seems quite large to process on my computer.</div><div><br></div><div>Best,</div><div>Bibhash</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 18, 2022 at 4:58 AM Nikhil VJ <<a href="mailto:nikhil.js@gmail.com" rel="noreferrer" target="_blank">nikhil.js@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks Justin for sharing!<div><br></div><div>I've downloaded and extracted the India.geojsonl on a webserver.</div><div><br></div><div><div>ref about .geojsonl : <a href="https://www.interline.io/blog/geojsonl-extracts/" rel="noreferrer" target="_blank">https://www.interline.io/blog/geojsonl-extracts/</a> </div><div>-> pretty useful! One can loop through a huge file without having to load it all into RAM.</div><div></div></div><div><br></div><div>top lines look like:</div><div>{"type": "Feature", "properties": {},"geometry": {"type": "Polygon","coordinates": [[[83.06380515611697, 25.34167404697847], [83.06380909901775, 25.341635591519122], [83.06386494585949, 25.341640268589657], [83.06386100295869, 25.341678724047526], [83.06380515611697, 25.34167404697847]]]}}<br>{"type": "Feature", "properties": {},"geometry": {"type": "Polygon","coordinates": [[[87.87555977691633, 22.397660095199], [87.8754256865811, 22.397658576750985], [87.87542690191553, 22.397566835386712], [87.87556099225075, 22.39756835383578], [87.87555977691633, 22.397660095199]]]}}<br></div><div>...</div><div>So we have just basic polygons, one for each building, no properties or categorization.<br></div><div><br></div><div>Here are some ideas on what to do with this:</div><div><br></div><div>1. Postgresql DB:</div><div>- Load all of these into a PostGreSQL DB</div><div>- Setup an api that will take a lat/lon and give all shapes within 1km radius</div><div>- Next possible api: send a bounding polygon and get all buildings in that</div><div><br></div><div>2. Split up by district or lower level admin boundaries:</div><div>- Load the admin boundaries into a Postgresql DB</div><div>- Loop through each line (ie each building)</div><div>- Find out which place it's in by ST_Within query</div><div>- Dump it into a separate .geojsonl for that place</div><div>- We've now split it into multiple smaller files that are more usable, can be loaded up by OSM mappers etc.</div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>--<br>Cheers,<br>Nikhil VJ<br><a href="https://nikhilvj.co.in" rel="noreferrer" target="_blank">https://nikhilvj.co.in</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 18, 2022 at 3:51 AM Justin <<a href="mailto:justinelliotmeyers@gmail.com" rel="noreferrer" target="_blank">justinelliotmeyers@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><img alt="lg.png" width="550px" height="96px" src="cid:1bebcf64-5dc3-46f5-9854-f8cbb9b110bf"><br><div><br></div><div><a href="https://github.com/microsoft/GlobalMLBuildingFootprints" rel="noreferrer" target="_blank">https://github.com/microsoft/GlobalMLBuildingFootprints</a><br></div><div><br></div><div><br></div>
<p></p>
-- <br>
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting <a href="http://datameet.org" rel="noreferrer" target="_blank">http://datameet.org</a><br>
--- <br>
You received this message because you are subscribed to the Google Groups "datameet" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:datameet+unsubscribe@googlegroups.com" rel="noreferrer" target="_blank">datameet+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com?utm_medium=email&utm_source=footer" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com</a>.<br>
</blockquote></div>
<p></p>
-- <br>
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting <a href="http://datameet.org" rel="noreferrer" target="_blank">http://datameet.org</a><br>
--- <br>
You received this message because you are subscribed to the Google Groups "datameet" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:datameet+unsubscribe@googlegroups.com" rel="noreferrer" target="_blank">datameet+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com?utm_medium=email&utm_source=footer" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com</a>.<br>
</blockquote></div>
<p></p>
-- <br>
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting <a href="http://datameet.org" rel="noreferrer" target="_blank">http://datameet.org</a><br>
--- <br>
You received this message because you are subscribed to the Google Groups "datameet" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:datameet+unsubscribe@googlegroups.com" rel="noreferrer" target="_blank">datameet+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/datameet/CAHRkDO5qPr_E68nzFiTh%3DxSPd1myz0_WMbJi83mdQMDZ53pEkQ%40mail.gmail.com?utm_medium=email&utm_source=footer" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/datameet/CAHRkDO5qPr_E68nzFiTh%3DxSPd1myz0_WMbJi83mdQMDZ53pEkQ%40mail.gmail.com</a>.<br>
</blockquote></div>
<p></p>
-- <br>
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting <a href="http://datameet.org" rel="noreferrer" target="_blank">http://datameet.org</a><br>
--- <br>
You received this message because you are subscribed to the Google Groups "datameet" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:datameet+unsubscribe@googlegroups.com" rel="noreferrer" target="_blank">datameet+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/datameet/CAH7jeuOo0vfbxnsv%2B-V7%2Bt7zgkNGruyaT0xzULECGc-e8h3Qcw%40mail.gmail.com?utm_medium=email&utm_source=footer" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/datameet/CAH7jeuOo0vfbxnsv%2B-V7%2Bt7zgkNGruyaT0xzULECGc-e8h3Qcw%40mail.gmail.com</a>.<br>
</blockquote></div>
<p></p>
-- <br>
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting <a href="http://datameet.org" target="_blank">http://datameet.org</a><br>
--- <br>
You received this message because you are subscribed to the Google Groups "datameet" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:datameet+unsubscribe@googlegroups.com" target="_blank">datameet+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/datameet/CA%2B4SsQCfA5dxiWne%3Dsghbn8OH_FoBgKUDtVeF9hMs%2BxU6he_JQ%40mail.gmail.com?utm_medium=email&utm_source=footer" target="_blank">https://groups.google.com/d/msgid/datameet/CA%2B4SsQCfA5dxiWne%3Dsghbn8OH_FoBgKUDtVeF9hMs%2BxU6he_JQ%40mail.gmail.com</a>.<br>
</blockquote></div>