Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby montagdude » Wed Feb 05, 2020 6:39 pm

Hopefully this is the right place for this question. If I should make a new thread instead, let me know.

I have started using osm2city during the last week or so to generate scenery for different locations. It is an amazing tool, by the way, so thank you! I have a question, though. I am currently trying to run it for a relatively large area (about 4x3 degrees). I ran the script with -p3, but now after having run for a couple hours, only one process is active. I ran it with the --log-to-file option, and only one of the osm2city_process_SpawnPoolWorker*.log files has been written to recently. Any ideas why that would happen? This is obviously going to make it take forever to finish the entire area.

Also, in the event that the building fails or gets killed, is there a way to make it pick up again from where it left off?
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Wed Feb 05, 2020 7:55 pm

montagdude: Continuing an interrupted build is not really supported. I've written some scripts that let you do it. https://gitlab.com/merspieler/easy-osm2city
Note that they are a wraper around osm2city so you'd have to change your directory structure to the one it uses or you just look at it, how I did it and come up with something your self.
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 532
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby montagdude » Thu Feb 06, 2020 12:02 am

merspieler wrote in Wed Feb 05, 2020 7:55 pm:montagdude: Continuing an interrupted build is not really supported. I've written some scripts that let you do it. https://gitlab.com/merspieler/easy-osm2city
Note that they are a wraper around osm2city so you'd have to change your directory structure to the one it uses or you just look at it, how I did it and come up with something your self.

Thanks, that looks helpful! I started over with a couple settings changed back to default (PROBE_FOR_WATER and OWBB_USE_BTG_LANDUSE), and that seems to have prevented the parallel processes from dying, at least for now. It's still got a long way to go, though. If it fails, I'll definitely study your restarting solution and see if I can do something similar.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby vanosten » Wed Feb 19, 2020 8:50 pm

wlbragg wrote in Wed Feb 05, 2020 6:34 pm:One other item, I thought there were built in checks for existing models in the osm2city code generation. Is this no longer the case?
Image


Can I please get some coordinates? And what do you have in parameter PATH_TO_SCENERY_OPT - and where does that stuff come from?
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 444
Joined: Sat Sep 25, 2010 5:38 pm
Location: Denmark - but I am Swiss
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby montagdude » Tue Mar 10, 2020 5:11 pm

merspieler wrote in Wed Feb 05, 2020 7:55 pm:montagdude: Continuing an interrupted build is not really supported. I've written some scripts that let you do it. https://gitlab.com/merspieler/easy-osm2city
Note that they are a wraper around osm2city so you'd have to change your directory structure to the one it uses or you just look at it, how I did it and come up with something your self.

Here is a related question. I am currently trying to run osm2city for a relatively large area (Virginia, Maryland, and Delaware in the US). I know my computer can't handle that much (only 8 GB of RAM), so I've broken it into 17 smaller blocks, making sure larger urban areas like Washington, DC and Baltimore are in separate blocks. I successfully ran it for a couple of the blocks with less data, but now I'm working on Washington, DC. I think I am having problems due to not having a lot of RAM, so I killed it part way through. I had the idea to start it again with fewer processes, hopefully trading memory requirements for longer CPU time. Now for the question: can I just try the block again without deleting anything from the output directories? In other words, will it just overwrite anything that got written there during the first try? I'd rather not delete everything, because then I would have to re-do the other blocks too.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Tue Mar 10, 2020 5:34 pm

You are good to go ahead without removing anything...
Yes... RAM is an issue... I've got a single thread taking up over 80GB (Sao Paulo area).... it got killed by oom killer.... You don't really need to split it into block due to RAM as the will be build in sequence anyways. Where you've got to find the tradeoff is between parallel threads (-p option) and RAM usage.

Or you just download the scenery from here ;-)
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 532
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby montagdude » Tue Mar 10, 2020 5:43 pm

merspieler wrote in Tue Mar 10, 2020 5:34 pm:You are good to go ahead without removing anything...
Yes... RAM is an issue... I've got a single thread taking up over 80GB (Sao Paulo area).... it got killed by oom killer.... You don't really need to split it into block due to RAM as the will be build in sequence anyways. Where you've got to find the tradeoff is between parallel threads (-p option) and RAM usage.
Thanks for confirming.

merspieler wrote in Tue Mar 10, 2020 5:34 pm:Or you just download the scenery from here ;-)
Ooh, that's even better! Is the whole world completed already? Then I can just focus on the terrain.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Tue Mar 10, 2020 6:07 pm

Not the whole world... but USA is (except for boston... that's still running.)
You can check the progress here
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 532
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby montagdude » Tue Mar 10, 2020 6:41 pm

Great! Thanks so much for this effort.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Tue Mar 10, 2020 10:01 pm

You're very welcome

Oh... and boston just finished too.
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 532
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby montagdude » Wed Mar 11, 2020 3:05 am

Nice, I got the files. Now I just have to build FlightGear-nightly to use them.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby montagdude » Wed Mar 11, 2020 1:03 pm

I managed to get FlightGear nightly built and test out your data (I looked around Maryland and DC). I noticed that the OSM building density was much less than when I've generated it myself. I use more or less default parameters. Is that intentional to reduce the file size for the world build, or am I doing something wrong? Also, I hardly saw any roads, and the ones that I did see had lots of breaks in them, but I'm guessing that is because I am using custom terrain.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Wed Mar 11, 2020 6:13 pm

Try setting the Model shader to the lowest option... I've got the bug, that when I set it to anything higher, the shader based buildings disappear (and I guess, that you've build it without shaber buildings)

For the roads... most certainly caused by the terrain.
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 532
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby montagdude » Wed Mar 11, 2020 7:43 pm

merspieler wrote in Wed Mar 11, 2020 6:13 pm:Try setting the Model shader to the lowest option... I've got the bug, that when I set it to anything higher, the shader based buildings disappear (and I guess, that you've build it without shaber buildings)

For the roads... most certainly caused by the terrain.

That did it, thanks. Only problem is that those shader-based buildings flicker like crazy with my Intel integrated graphics. Oh well, one more reason to upgrade, I guess. Anyway, this will help a lot because now I just need to generate the roads with osm2city.
montagdude
 
Posts: 123
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Wed Mar 11, 2020 9:51 pm

I've reported the vanishing buildings as well as the flickering months ago... seems to be forgotten.... (like so many thing in fg) btw... if you upgrade to get rid of these bugs... don't go with amd... i've just got an amd card which work nicely but the bugs are there as well (not sure if it works on envidia tho)
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 532
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 0 guests