Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby Johan G » Sun Jul 19, 2020 8:20 pm

Some posts were split off to the new topic Typical specifications of computers running FlightGear?
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Some YouTube videos
Johan G
Moderator
 
Posts: 6625
Joined: Fri Aug 06, 2010 6:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 2020.3.4
OS: Windows 10, 64 bit

Re: osm2city.py development

Postby vanosten » Sat Jul 25, 2020 2:54 pm

I have pushed some updates to the logic for building exclusions/reduction. More importantly, there is now code to better detect and handle L-shaped roofs with neighbour buildings (in central Reykjavik these are around 4% of all buildings).
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby saul » Thu Jul 30, 2020 3:24 pm

Hi,

I just came back from a few months pause in FG and pulled the latest changes on the osm2city builder, but now I got some problems that I can not solve by myself. Can you please give me a hand to see what I am doing wrong?

I created a virtual environment for python 3.7 by using:

Code: Select all
python3.7 -m venv work3.7


and installed all the needed dependencies:
Code: Select all
(work3.7) saul@viking:~/flightgear/development/osm2city$ pip freeze
attrs==19.3.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
click-plugins==1.1.1
cligj==0.5.0
cycler==0.10.0
decorator==4.4.2
descartes==1.1.0
Fiona==1.8.13.post1
idna==2.10
kiwisolver==1.2.0
matplotlib==3.3.0
munch==2.5.0
networkx==2.4
numpy==1.19.1
Pillow==7.2.0
psycopg2-binary==2.8.5
pyparsing==2.4.7
pyproj==2.6.1.post1
python-dateutil==2.8.1
requests==2.24.0
scipy==1.5.2
Shapely==1.7.0
six==1.15.0
urllib3==1.25.10


When I run the builder from within the environment I have the following error:

Code: Select all
SpawnPoolWorker-2 root ERROR    Exception occurred while processing tile 1726160.
Traceback (most recent call last):
  File "/home/saul/flightgear/development/osm2city/build_tiles.py", line 131, in process_scenery_tile
    lit_areas, water_areas, osm_buildings = ol.process(the_coords_transform, my_airports)
  File "/home/saul/flightgear/development/osm2city/osm2city/owbb/landuse.py", line 783, in process
    btg_reader = btg.read_btg_file(transformer, None)
  File "/home/saul/flightgear/development/osm2city/osm2city/utils/btg_io.py", line 445, in read_btg_file
    out_proj = pyproj.Proj('epsg:4326', ellps='WGS84', datum='WGS84')
  File "/home/saul/work3.7/lib/python3.7/site-packages/pyproj/proj.py", line 107, in __init__
    self.crs = CRS.from_user_input(projparams, **kwargs)
  File "/home/saul/work3.7/lib/python3.7/site-packages/pyproj/crs/crs.py", line 440, in from_user_input
    return CRS(value, **kwargs)
  File "/home/saul/work3.7/lib/python3.7/site-packages/pyproj/crs/crs.py", line 296, in __init__
    super().__init__(projstring)
  File "pyproj/_crs.pyx", line 2338, in pyproj._crs._CRS.__init__
pyproj.exceptions.CRSError: Invalid projection: epsg:4326 +ellps=WGS84 +datum=WGS84 +type=crs: (Internal Proj Error: proj_create: crs not found)
SpawnPoolWorker-2 root ERROR    ******* Exception in tile 1726160 - to reprocess use boundaries: -75.0_1.25_-74.93_1.375 *******
Traceback (most recent call last):
  File "/home/saul/flightgear/development/osm2city/build_tiles.py", line 131, in process_scenery_tile
    lit_areas, water_areas, osm_buildings = ol.process(the_coords_transform, my_airports)
  File "/home/saul/flightgear/development/osm2city/osm2city/owbb/landuse.py", line 783, in process
    btg_reader = btg.read_btg_file(transformer, None)
  File "/home/saul/flightgear/development/osm2city/osm2city/utils/btg_io.py", line 445, in read_btg_file
    out_proj = pyproj.Proj('epsg:4326', ellps='WGS84', datum='WGS84')
  File "/home/saul/work3.7/lib/python3.7/site-packages/pyproj/proj.py", line 107, in __init__
    self.crs = CRS.from_user_input(projparams, **kwargs)
  File "/home/saul/work3.7/lib/python3.7/site-packages/pyproj/crs/crs.py", line 440, in from_user_input
    return CRS(value, **kwargs)
  File "/home/saul/work3.7/lib/python3.7/site-packages/pyproj/crs/crs.py", line 296, in __init__
    super().__init__(projstring)
  File "pyproj/_crs.pyx", line 2338, in pyproj._crs._CRS.__init__
pyproj.exceptions.CRSError: Invalid projection: epsg:4326 +ellps=WGS84 +datum=WGS84 +type=crs: (Internal Proj Error: proj_create: crs not found)
SpawnPoolWorker-2 root INFO     ******* Finished tile 1726160 - 2914/2915 ******



There are no output files created. My params.ini is very short:

Code: Select all
PREFIX = "ECUADOR"

MAX_SLOPE_ROAD = 0.20
MIN_ABOVE_GROUND_LEVEL = 0.02

PATH_TO_SCENERY = "/home/saul/flightgear/scenery"
PATH_TO_OUTPUT = "/home/saul/flightgear/fg_customscenery/ECUADOR"

PATH_TO_OSM2CITY_DATA = "/home/saul/flightgear/development/osm2city-data"

DB_HOST = "localhost"
DB_PORT = 5432
DB_NAME = "ecuador"
DB_USER = "giuser"
DB_USER_PASSWORD = "hola"

NO_ELEV = False
FG_ELEV = "/home/saul/flightgear/dnc-managed/install/flightgear/bin/fgelev"


Is there something that I am doing wrong?

Best regards,

Saul
Last edited by Johan G on Thu Jul 30, 2020 4:55 pm, edited 1 time in total.
Reason: Added [code] tags. They can be added manually or by clicking the [^_ ] button above the text editing box.
saul
 
Posts: 40
Joined: Tue Nov 26, 2013 10:57 pm

Re: osm2city.py development

Postby merspieler » Thu Jul 30, 2020 3:32 pm

Hey Saul,

known issue caused by pyproj 2.6.0.

There's a requirements.txt which you can use to create your venv. This will install all dependencies in working versions.

Code: Select all
$ python3.7 -m venv work3.7/
$ source work3.7/bin/activate
$ pip install -r osm2city/requirements.txt
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby saul » Fri Jul 31, 2020 11:46 pm

Many thanks Merspieler,

I got it working all again. I launched a build on a complete country, Ecuador, by using the latest osm.pbf file which is 67 MB. It took around a day to complete in a i7-4790k@4 GHz, 4 processors enabled. The osm2city scenery is a little more than 1 GB. The script actually did not exit. It just released all processors and stayed in the very last tile, but all the files are there.

Code: Select all
SpawnPoolWorker-3 root INFO     No platforms found -> aborting
SpawnPoolWorker-3 root INFO     Generating clusters -3895.1668000 -6910.9286484 3895.1668000 6910.9286484 for object_static
SpawnPoolWorker-3 root INFO     Reading OSM way data for ['man_made=>pier'] from db took 0.0076 seconds.
SpawnPoolWorker-3 root INFO     ways: 0
SpawnPoolWorker-3 root INFO     No piers found -> aborting
SpawnPoolWorker-3 root INFO     ******* Finished tile 1726160 - 2914/2915 *******


I have tested some cities and everything is there. Now I am thinking to build Sweden, but I am a bit skeptical since the osm.pbf is 484 MB and at the same speed than Ecuador it may take 1 week (and some good 7-8 GB, probably more). Are these numbers the same you are having or I am doing something wrong?

Saul
saul
 
Posts: 40
Joined: Tue Nov 26, 2013 10:57 pm

Re: osm2city.py development

Postby merspieler » Sat Aug 01, 2020 8:42 am

I didn't look at my exact numbers... but it sounds plausible...
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby merspieler » Sat Aug 29, 2020 2:09 pm

Rick: what do we really need from the fgdata directory?
I'm building a podman container for osm2city and I'd like to reduce the size (currently over 5GB) to which fgdata contributes a significant ammount so I'd like to remove all the unneeded stuff from it.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby vanosten » Sat Aug 29, 2020 2:16 pm

I do not know: what do you want to do? A container to generate osm2city sceneries? From the top of my had you need the apt-dat-file plus fgelev.exe. What fgelev need, I do not know.
Last edited by Johan G on Sat Aug 29, 2020 10:43 pm, edited 1 time in total.
Reason: Please do not quote the entire post directly aboove yours.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby merspieler » Sat Aug 29, 2020 3:36 pm

Yes, a container to create osm2city sceneries...

I guess, when it's uncertain, then I better keep everything.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby montagdude » Tue Sep 01, 2020 2:41 pm

I'm generating some osm2city scenery in small blocks for the current area I'm working on. Each block has ~40-100 tiles. I do it this way so that I can maximize CPU usage in less densely populated areas without running out of memory (use fewer cores for the more densely populated blocks). For 3 of the 6 blocks I've completed so far, I keep having one tile that never completes. The process just hangs and I never see "Finished tile ..." at the end of the log for that process. I thought I might be running into memory issues, but I'm pretty sure that was not the case for this last block. Any ideas why this might be happening and suggestions for how to locate and regenerate the problematic tiles?
montagdude
 
Posts: 266
Joined: Tue Dec 31, 2019 7:04 am

Re: osm2city.py development

Postby merspieler » Tue Sep 01, 2020 3:02 pm

There should be an exceptions.log which shows, what error occured.
If you've really got out of memory problems, it will look like this
Code: Select all
******* Exception in tile 5234554 - to reprocess use boundaries: 139.5_35.875_139.75_36.0 ******* at 2020-09-01_112520 -
Traceback (most recent call last):
  File "/storage/osm2city/easy-osm2city/osm2city/build_tiles.py", line 131, in process_scenery_tile
    lit_areas, water_areas, osm_buildings = ol.process(the_coords_transform, my_airports)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/owbb/landuse.py", line 906, in process
    bu.process_building_loose_parts(building_nodes_dict, osm_buildings)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/buildings.py", line 573, in process_building_loose_parts
    second_building.parent.transfer_children(first_building.parent)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/building_lib.py", line 1547, in transfer_children
    other_parent.add_child(child)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/building_lib.py", line 1465, in add_child
    self.children.append(child)
MemoryError

or this
Code: Select all
******* Exception in tile 3593417 - to reprocess use boundaries: 39.125_-6.875_39.25_-6.75 ******* at 2020-08-30_034854 -
Traceback (most recent call last):
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/utils/utilities.py", line 451, in probe
    elev_is_solid_tuple = self._cache[key]
KeyError: (39.24637649324977, -6.793663019266104)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/osm2city/easy-osm2city/osm2city/build_tiles.py", line 144, in process_scenery_tile
    osm_buildings, file_lock)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/buildings.py", line 1107, in process_buildings
    prepare_textures.facades, prepare_textures.roofs, stats)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/building_lib.py", line 1679, in analyse
    if not b.analyse_elev_and_water(fg_elev):
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/building_lib.py", line 660, in analyse_elev_and_water
    min_ground_elev, diff_elev = fg_elev.probe_list_of_points(self.pts_outer)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/utils/utilities.py", line 470, in probe_list_of_points
    elev_is_solid_tuple = self.probe(point)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/utils/utilities.py", line 454, in probe
    elev_is_solid_tuple = really_probe(position)
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/utils/utilities.py", line 400, in really_probe
    self._open_fgelev()
  File "/storage/osm2city/easy-osm2city/osm2city/osm2city/utils/utilities.py", line 369, in _open_fgelev
    bufsize=1, universal_newlines=True)
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1453, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory


The first line states: Exception in tile 5234554 - to reprocess use boundaries: 139.5_35.875_139.75_36.0 so you've got the tile name and the boundaries so you can rebuild that tile.

Else, just be patient... a few days are not really uncommon for some tiles.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby montagdude » Tue Sep 01, 2020 3:14 pm

merspieler wrote in Tue Sep 01, 2020 3:02 pm:There should be an exceptions.log which shows, what error occured.
If you've really got out of memory problems, it will look like this:

The first line states: Exception in tile 5234554 - to reprocess use boundaries: 139.5_35.875_139.75_36.0 so you've got the tile name and the boundaries so you can rebuild that tile.

Else, just be patient... a few days are not really uncommon for some tiles.

Okay, thanks. I haven't gotten that error, but the process is just not doing anything. No CPU usage at all. But I can probably just stick in some print statements to find out the tile boundaries and then reprocess.
montagdude
 
Posts: 266
Joined: Tue Dec 31, 2019 7:04 am

Re: osm2city.py development

Postby merspieler » Tue Sep 01, 2020 4:27 pm

oh... I remember that...
Never seen it in a long time tho.
The tile names are in the log... and from the tile name you can get the goundaries.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby montagdude » Tue Sep 01, 2020 9:14 pm

I had it happen again while I was paying more attention. It seems to happen when it starts using swap. The worker gets killed and a new one gets spawned, but there is no error log. I have --logtofile enabled. Swap hasn't been depleted, so I'm not sure why it is getting killed. Here is what's in the log. How do I translate this to tile boundaries?

Code: Select all
Processing tile 1712251 in prefix w080n30_w076n39_1712251 with process id = 4248 - 9/55
montagdude
 
Posts: 266
Joined: Tue Dec 31, 2019 7:04 am

Re: osm2city.py development

Postby merspieler » Tue Sep 01, 2020 10:24 pm

That's rather complicated, see http://wiki.flightgear.org/Tile_Index_Scheme
Here are the coords of your tile.
W -75.25
S 39.875
E -75.0
N 40.0

If you don't want to do the math (or maybe write a lil script) you can take the rough coords (in your case: w076n39) and manually look for that tile on this map https://scenery.flightgear.org/map/

There is already a "script" that I've written: https://gitlab.com/merspieler/osm2city- ... /common.py
It's not actually intended for manual use but will do the job (or you just write some sort of frontend for it.

To use it, go into that directory and run `python3`
It works like this:
Code: Select all
>>> import common
>>> tile = 1712251
>>> common.get_west(tile)
-75.25
>>> common.get_south(tile)
39.875
>>> common.get_east(tile)
-75.0
>>> common.get_north(tile)
40.0
>>>
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 3 guests