Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby miguel » Sat Jan 19, 2019 8:15 pm

what happens with the bridges of San francisco does not see them????????
miguel
 
Posts: 226
Joined: Wed Aug 19, 2015 5:05 pm

Re: osm2city.py development

Postby barta » Mon Feb 25, 2019 11:54 pm

Hi,
I'm using osm2city to build the area of Buenos Aires (Argentina), but there's a particular tile that makes osm2city crash. In fact, it's THE most important tile, where the airport SABE it's located.
The rest of the tiles builds with no errors.

Here is the error:
Code: Select all
SpawnPoolWorker-1 root ERROR    Exception occurred while processing tile 1986010.
Traceback (most recent call last):
  File "/home/julio/git/osm2city/build_tiles.py", line 145, in process_scenery_tile
    osm_buildings, file_lock)
  File "/home/julio/git/osm2city/buildings.py", line 752, in process_buildings
    prepare_textures.facades, prepare_textures.roofs, stats)
  File "/home/julio/git/osm2city/building_lib.py", line 1532, in analyse
    parent.align_textures_children()
  File "/home/julio/git/osm2city/building_lib.py", line 1314, in align_textures_children
    if child.longest_edge_length > default_child.longest_edge_length:
  File "/home/julio/git/osm2city/building_lib.py", line 507, in longest_edge_length
    return max(self.edge_length_pts)
TypeError: 'NoneType' object is not iterable
SpawnPoolWorker-1 root ERROR    ******* Exception in tile 1986010 - to reprocess use boundaries: -58.5_-34.583_-58.355_-34.534 *******
Traceback (most recent call last):
  File "/home/julio/git/osm2city/build_tiles.py", line 145, in process_scenery_tile
    osm_buildings, file_lock)
  File "/home/julio/git/osm2city/buildings.py", line 752, in process_buildings
    prepare_textures.facades, prepare_textures.roofs, stats)
  File "/home/julio/git/osm2city/building_lib.py", line 1532, in analyse
    parent.align_textures_children()
  File "/home/julio/git/osm2city/building_lib.py", line 1314, in align_textures_children
    if child.longest_edge_length > default_child.longest_edge_length:
  File "/home/julio/git/osm2city/building_lib.py", line 507, in longest_edge_length
    return max(self.edge_length_pts)
TypeError: 'NoneType' object is not iterable


The ini file contains:
Code: Select all
PREFIX="auto"
PATH_TO_SCENERY="/home/julio/src/scenery/osm-2018/auto/output"
PATH_TO_OUTPUT="/home/julio/src/scenery/osm-2018/auto/output"
PATH_TO_OSM2CITY_DATA = "/home/julio/git/osm2city-data"

NO_ELEV = False
FG_ELEV = '/usr/local/bin/fgelev'

DB_HOST="localhost"
DB_NAME="osm_auto"
DB_USER="postgres"
DB_USER_PASSWORD="xxxxxxxxx"


BOUNDARY_WEST=-58.512
BOUNDARY_EAST=-58.355
BOUNDARY_SOUTH=-34.583
BOUNDARY_NORTH=-34.534


Also, Airport terminals (airway=terminal) buildings are ignored. Any other type of building inside an airport is constructed ok.

¡Thanks in advance!
"When once you have tasted flight, you will forever walk the earth with your eyes turned skyward, for there you have been, and there you will always long to return."
- Leonardo da Vinci
barta
 
Posts: 36
Joined: Thu Mar 05, 2009 1:24 am
Location: On the air, preferrably

Re: osm2city.py development

Postby merspieler » Thu Feb 28, 2019 9:48 pm

I've got the following error:
Code: Select all
SpawnPoolWorker-2 root INFO     Spawning fgelev                                                                                                                                                   
/games/fg-linux/install/flightgear/bin/fgelev: error while loading shared libraries: libosgText.so.130: cannot open shared object file: No such file or directory                                 
SpawnPoolWorker-2 root CRITICAL Skipped 20 lines                                                                                                                                                 
SpawnPoolWorker-2 root CRITICAL 1 153.028 -27.4755                                                                                                                                               
SpawnPoolWorker-2 root CRITICAL fgelev returned <>, resulting in list index out of range. Did fgelev start OK (Record : 1)?                                                                       
SpawnPoolWorker-2 root ERROR    Exception occurred while processing tile 5459872.                                                                                                                 
Traceback (most recent call last):                                                                                                                                                               
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 447, in probe                                                                                                               
    elev_is_solid_tuple = self._cache[key]                                                                                                                                                       
KeyError: (153.0275893357453, -27.47550741757804)                                                                                                                                                 
                                                                                                                                                                                                 
During handling of the above exception, another exception occurred:                                                                                                                               
                                                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                               
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 415, in really_probe                                                                                                         
    elev = float(parts[1]) + self.h_offset                                                                                                                                                       
IndexError: list index out of range     

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/scenery/osm2city/osm2city/build_tiles.py", line 144, in process_scenery_tile
    osm_buildings, file_lock)
  File "/storage/scenery/osm2city/osm2city/buildings.py", line 752, in process_buildings
    prepare_textures.facades, prepare_textures.roofs, stats)
  File "/storage/scenery/osm2city/osm2city/building_lib.py", line 1491, in analyse
    if not b.analyse_elev_and_water(fg_elev):
  File "/storage/scenery/osm2city/osm2city/building_lib.py", line 652, in analyse_elev_and_water
    min_ground_elev, diff_elev = fg_elev.probe_list_of_points(self.pts_outer)
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 466, in probe_list_of_points
    elev_is_solid_tuple = self.probe(ve.Vec2d(point))
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 450, in probe
    elev_is_solid_tuple = really_probe(position)
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 430, in really_probe
    raise RuntimeError("fgelev errors are fatal.")
RuntimeError: fgelev errors are fatal.
SpawnPoolWorker-2 root ERROR    ******* Exception in tile 5459872 - to reprocess use boundaries: 153.0_-27.5_153.25_-27.375 *******
Traceback (most recent call last):
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 447, in probe
    elev_is_solid_tuple = self._cache[key]
KeyError: (153.0275893357453, -27.47550741757804)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 415, in really_probe
    elev = float(parts[1]) + self.h_offset
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/scenery/osm2city/osm2city/build_tiles.py", line 144, in process_scenery_tile
    osm_buildings, file_lock)
  File "/storage/scenery/osm2city/osm2city/buildings.py", line 752, in process_buildings
    prepare_textures.facades, prepare_textures.roofs, stats)
  File "/storage/scenery/osm2city/osm2city/building_lib.py", line 1491, in analyse
    if not b.analyse_elev_and_water(fg_elev):
  File "/storage/scenery/osm2city/osm2city/building_lib.py", line 652, in analyse_elev_and_water
    min_ground_elev, diff_elev = fg_elev.probe_list_of_points(self.pts_outer)
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 466, in probe_list_of_points
    elev_is_solid_tuple = self.probe(ve.Vec2d(point))
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 450, in probe
    elev_is_solid_tuple = really_probe(position)
  File "/storage/scenery/osm2city/osm2city/utils/utilities.py", line 430, in really_probe
    raise RuntimeError("fgelev errors are fatal.")
RuntimeError: fgelev errors are fatal.
SpawnPoolWorker-2 root INFO     ******* Finished tile 5459872 - 60901/63296 *******
MainProcess root INFO     Total time used: 35135.502209


What causes the problem?

I'm running:
osm2city 4b985c277035ff23a1923274a4d4d84378ea8228
osm2city-data fd8e6a0491cdb718fd4578b3c00c0ca533308ad3
fg build d92647b364b475424dcb5d126880f6923957a2c8 (on next)
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby radi » Fri Mar 01, 2019 8:51 am

Your problem is this:

merspieler wrote in Thu Feb 28, 2019 9:48 pm:/games/fg-linux/install/flightgear/bin/fgelev: error while loading shared libraries: libosgText.so.130: cannot open shared object file: No such file or directory


fgelev fails because your OS can't find the shared library libosgText.so.130.

Could be you updated OSG after you've compiled fgelev, in which case you should re-compile.

Could be libosgText.so.130 exists on your system, but for some reason not in a place where your OS typically looks for shared libs. Try searching for that lib
Code: Select all
find / -name libosgText.so.130

and if it exists, set (assuming your using bash)
Code: Select all
export LD_LIBRARY_PATH=/path/to/your/lib/

and then run
Code: Select all
fgelev

and see if the error goes away.
OSM buildings for LOWI, EDDC
Custom scenery for VHXX YMML
Edit .stg via the FG Object Placement Tool
radi
 
Posts: 659
Joined: Mon Aug 25, 2008 5:24 pm
Location: YMML, EDDC

Re: osm2city.py development

Postby merspieler » Sat Mar 02, 2019 1:03 am

Ok, i'll try that... i was only completely confused cause this happened after about 9:45h running without errors... i'll check the system...

UPDATE: I had to rebuild... seems to work for now... thanks for the help
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby merspieler » Sun Mar 03, 2019 12:54 am

Next problem... RAM got full... i'm not sure tho if this is a problem with osm2city or fgelev.... I'd like your oppinion on that before i report something to the ml...

I've been building quite a large area (australia, bounds: 112:-45_155_-10) with the following parameters (database info removed):
Code: Select all
PREFIX = "australia"
PATH_TO_SCENERY = "/games/.fgfs/TerraSync"
PATH_TO_OUTPUT = "/storage/scenery/osm2city/projects/australia/scenery"
PATH_TO_OSM2CITY_DATA = "/storage/scenery/osm2city/osm2city-data"

FG_ELEV = "/games/fg-linux/install/flightgear/bin/fgelev"

BUILDING_MIN_AREA = 30.0

HIGHWAY_TYPE_MIN = 3
POINTS_ON_LINE_DISTANCE_MAX = 20

C2P_PROCESS_OVERHEAD_LINES = True
C2P_PROCESS_STREETLAMPS = True


I had it running over night and somewhen it filled up the entire RAM and got stuck.
htop is showing me, when sorted by mem usage, a bunch of fgelev processes:
Image

I've started build_tiles.py with -p 4 so i'd expect 4 fgelev instances... one for every thread.

Is this a problem on the side of fgelev (eg memory leak) or in osm2city?
If the later... how can i help debuging?

I'll add the exceptions log in an update...
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby radi » Sun Mar 03, 2019 2:01 am

I don't think fgelev itself spawns subprocesses, so I reckon the problem is build_tiles.py. Mind you, I have never used, let alone looked at the code of build_tiles.py, so can't provide much help here.
OSM buildings for LOWI, EDDC
Custom scenery for VHXX YMML
Edit .stg via the FG Object Placement Tool
radi
 
Posts: 659
Joined: Mon Aug 25, 2008 5:24 pm
Location: YMML, EDDC

Re: osm2city.py development

Postby vanosten » Sun Mar 03, 2019 7:46 am

build_tiles.py is on me. I have never run with 4 parallel processes, only regularly with 2.

Could you please tell me how you get the output in your image (i.e. the exact command for your *nix tool, then I can compare with what happens in my default region on a little laptop.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby merspieler » Sun Mar 03, 2019 8:03 am

in the image i just run "htop -s PERCENT_MEM"
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby merspieler » Mon Mar 04, 2019 12:17 pm

I've been running it again... first with smaller scenery areas... I've noticed the following...
For the island of corse with 4 threads i had a total RAM usage of about 3.5GB (total including OS and other programs) over the entire time.
Building the more dense munich area it started again with about 3.5GB total use. I've let it run and when I've returned to look at it, the RAM usage has increased to > 6GB.

Currently I rebuild the australia scenery and I don't have too much problems with it... yet...
fgelev hasn't been called too often (I guess that is uses the cached <number>_elev.pkl files)
Tho I've noticed something interresting:
There are 2 fgelev processes, using about 630MB each, which have never exited tho they have been spawned hours ago. osm2city has doesn't seem to bother with these processes...
To me it looks like that's the same as happend last time but since there have been more processes got stuck the memory ended up being full.
I'm unable to kill -9 them (tho this frees at least the memory...

Update:
I've written to the mailing list... I'll see how that will work out...
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby vanosten » Mon Mar 04, 2019 6:52 pm

Memory usage between 500MB and 1 GB per process is around what I see, too.

Could you please have a look at osm2city-exceptions.log, whether you see a pattern? I could be that fgelev does not get closed correctly, when a process gets an exception - however I have never looked at it.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby merspieler » Tue Mar 05, 2019 1:11 am

there was only one thing (multiple times) in it... for which i've opend a gitlab issue: #107

I've just recorded htop showing how fgelev becomes a zombie even during normal? execution: https://asciinema.org/a/0ceI6aEu0NaF6eiLozeV5RAsx
Had again fgelev to fill the RAM... recording: https://asciinema.org/a/dCRXbA0cFbonp8stW5FUsRGV4
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby radi » Tue Mar 05, 2019 11:58 am

Can you please post the output of
Code: Select all
ps axf
when you find a zombie fgelev process? I bet it shows a stuck osm2city process.

I'm with vanosten -- I really doubt it's a problem with the fgelev code, but much rather the pipe to fgelev not getting closed properly.
OSM buildings for LOWI, EDDC
Custom scenery for VHXX YMML
Edit .stg via the FG Object Placement Tool
radi
 
Posts: 659
Joined: Mon Aug 25, 2008 5:24 pm
Location: YMML, EDDC

Re: osm2city.py development

Postby merspieler » Wed Mar 06, 2019 1:38 am

I will... on the next build
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby vanosten » Wed Mar 06, 2019 8:26 pm

barta wrote in Mon Feb 25, 2019 11:54 pm:Hi,
I'm using osm2city to build the area of Buenos Aires (Argentina), but there's a particular tile that makes osm2city crash. In fact, it's THE most important tile, where the airport SABE it's located.
The rest of the tiles builds with no errors.

Here is the error:
[code]
SpawnPoolWorker-1 root ERROR Exception occurred while processing tile 1986010.
Traceback (most recent call last):
File "/home/julio/git/osm2city/build_tiles.py", line 145, in process_scenery_tile
osm_buildings, file_lock)
File "/home/julio/git/osm2city/buildings.py", line 752, in process_buildings
prepare_textures.facades, prepare_textures.roofs, stats)
File "/home/julio/git/osm2city/building_lib.py", line 1532, in analyse
parent.align_textures_children()
File "/home/julio/git/osm2city/building_lib.py", line 1314, in align_textures_children
if child.longest_edge_length > default_child.longest_edge_length:
File "/home/julio/git/osm2city/building_lib.py", line 507, in longest_edge_length
return max(self.edge_length_pts)
TypeError: 'NoneType' object is not iterable


I have pushed a patch, which actually also should help with quite a few other problems, which just have been hidden. I have only tested at SABE due to lack of time - but I am quite confident. Please report if something else breaks.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 5 guests