Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby legoboyvdlp » Wed Apr 03, 2019 4:05 pm

Adding
Code: Select all
logging.debug('Running for %i out of %i', counter, len(my_buildings))


on line 321 of buildings.py might provide some more feedback as to the progress of checking building parts - currently my script has been sitting for over three hours processing the parts with no feedback as to progress.
User avatar
legoboyvdlp
 
Posts: 7981
Joined: Sat Jul 26, 2014 2:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Re: osm2city.py development

Postby merspieler » Thu Apr 04, 2019 12:12 am

sitting for over three hours without any progress feedback? sounds a bit like the problem I had.

Could you please check for me, if the processes cause CPU and IO load?
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby legoboyvdlp » Thu Apr 04, 2019 9:34 am

Yes, its causing CPU and IO and memory load.
Adding that statement shows incrementing counter:
Code: Select all
SpawnPoolWorker-1 root DEBUG    Running for 29312 out of 114243
SpawnPoolWorker-1 root DEBUG    Running for 29313 out of 114243


so hopefully its fine - just taking a very long time :)
We'll see what happens over the next few hours.
User avatar
legoboyvdlp
 
Posts: 7981
Joined: Sat Jul 26, 2014 2:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Re: osm2city.py development

Postby merspieler » Thu Apr 04, 2019 9:56 am

Ok... then it's different to my problem.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2228
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: osm2city.py development

Postby vanosten » Sun Apr 07, 2019 8:49 pm

legoboyvdlp wrote in Thu Mar 28, 2019 12:05 pm:Has anyone ever got this?

Code: Select all
 Unable to process building relation osm_id 1566802
Traceback (most recent call last):
  File "C:\Users\redpa\Documents\FlightGear\osm2city-work\osm2city\buildings.py", line 170, in _process_osm_relations
    my_buildings, coords_transform)
  File "C:\Users\redpa\Documents\FlightGear\osm2city-work\osm2city\buildings.py", line 248, in _process_multipolygon_buildings
    a_building = _make_building_from_way(nodes_dict, all_tags, outer_way, coords_transform, inner_rings)
  File "C:\Users\redpa\Documents\FlightGear\osm2city-work\osm2city\buildings.py", line 510, in _make_building_from_way
    refs=way.refs)
  File "C:\Users\redpa\Documents\FlightGear\osm2city-work\osm2city\building_lib.py", line 266, in __init__
    self.update_geometry(outer_ring, inner_rings_list, refs)
  File "C:\Users\redpa\Documents\FlightGear\osm2city-work\osm2city\building_lib.py", line 315, in update_geometry
    self.roll_inner_nodes()
  File "C:\Users\redpa\Documents\FlightGear\osm2city-work\osm2city\building_lib.py", line 381, in roll_inner_nodes
    outer_nodes_avail.remove(min_o)
ValueError: list.remove(x): x not in list



I pushed a fix. A bit ugly to remove some inner rings, when the number of inner rings is larger than the number of points in the outline. But this is very rare.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

ZeroDivisionError: float division by zero

Postby powoflight » Tue Apr 09, 2019 11:37 am

Hi Vanosten

After a longer time i am regenerating the bigger area of vienna.
For one tile (northpart of vienna 16.25_48.25_16.5_48.375 ) i get an error. The other tiles for vienna are ok. I use the lates Version of osm2city . Do i something wrong?
Other tiles are ok like 3320112, 3320114, 3320104 , 3320105, 3320106, etc

Code: Select all
Traceback (most recent call last):
  File "/home/bernhard/easy-osm2city/osm2city/build_tiles.py", line 150, in process_scenery_tile
    osm_buildings, file_lock)
  File "/home/bernhard/easy-osm2city/osm2city/buildings.py", line 771, in process_buildings
    write_buildings_in_meshes(coords_transform, buildings_in_meshes, stg_manager, stats)
  File "/home/bernhard/easy-osm2city/osm2city/buildings.py", line 693, in write_buildings_in_meshes
    cluster_elev, cluster_offset, prepare_textures.roofs, stats)
  File "/home/bernhard/easy-osm2city/osm2city/building_lib.py", line 1609, in write
    b.write_to_ac(ac_object, cluster_elev, cluster_offset, roof_mgr, face_mat_idx, roof_mat_idx, stats)
  File "/home/bernhard/easy-osm2city/osm2city/building_lib.py", line 1100, in write_to_ac
    face_mat_idx)
  File "/home/bernhard/easy-osm2city/osm2city/building_lib.py", line 1170, in _write_faces_for_ac
    self.body_height * tex_coord_top_input)
ZeroDivisionError: float division by zero
SpawnPoolWorker-1 root ERROR    ******* Exception in tile 3220113 - to reprocess use boundaries: 16.25_48.25_16.5_48.375 *******
Traceback (most recent call last):
  File "/home/bernhard/easy-osm2city/osm2city/build_tiles.py", line 150, in process_scenery_tile
    osm_buildings, file_lock)
  File "/home/bernhard/easy-osm2city/osm2city/buildings.py", line 771, in process_buildings
    write_buildings_in_meshes(coords_transform, buildings_in_meshes, stg_manager, stats)
  File "/home/bernhard/easy-osm2city/osm2city/buildings.py", line 693, in write_buildings_in_meshes
    cluster_elev, cluster_offset, prepare_textures.roofs, stats)
  File "/home/bernhard/easy-osm2city/osm2city/building_lib.py", line 1609, in write
    b.write_to_ac(ac_object, cluster_elev, cluster_offset, roof_mgr, face_mat_idx, roof_mat_idx, stats)
  File "/home/bernhard/easy-osm2city/osm2city/building_lib.py", line 1100, in write_to_ac
    face_mat_idx)
  File "/home/bernhard/easy-osm2city/osm2city/building_lib.py", line 1170, in _write_faces_for_ac
    self.body_height * tex_coord_top_input)
ZeroDivisionError: float division by zero
SpawnPoolWorker-1 root INFO     ******* Finished tile 3220113 - 1/1 *******
MainProcess root INFO     Total time used: 1805.099396


params.ini
Code: Select all
PREFIX = "donauturm"
PATH_TO_SCENERY = "/mnt/D_SSD_1TB/FG/SC/SC_HD"
PATH_TO_OUTPUT = "/mnt/D_HD/VMOrdner/OSM/A_NEW_TEST_2019/wien_nm"  #bounds=16.25_48.25_16.50_48.375
PATH_TO_OSM2CITY_DATA = "/home/bernhard/easy-osm2city/osm2city-data"

FG_ELEV = "/usr/games/fgelev"

DB_HOST = "localhost"
DB_NAME = "wien"
DB_USER = "gisuser"
DB_USER_PASSWORD = "xxxxxx"

BUILDING_FORCE_EUROPEAN_INNER_CITY_STYLE = True
HIGHWAY_TYPE_MIN = 3
POINTS_ON_LINE_DISTANCE_MAX = 20
TEXTURES_REGIONS_EXPLICIT = ["de", "gb", "us"]
OVERLAP_CHECK_CONVEX_HULL=True

SKIP_LIST = ["Donauturm",211614053, 211614052, 215821497, 215821496, 215821496, "Donau City Tower 1", 253582978, "Unfallkrankenhaus Lorenz Böhler", "Unfallkrankenhaus Meidling", "Rehabilitationszentrum Meidling", "Krankenanstalt Rudolfstiftung",9654966,  146919142, 146919143,  "Hanusch-Krankenhaus", "Rotes Bettenhaus", "Grünes Bettenhaus",7996051 , "Objekt 24 Richtfunkturm", "Heeresgeschichtliches Museum" , 28391000, 28415847,  "Müllverbrennungsanlage Spittelau", "Müllverbrennungsanlage Flötzersteig" , "Vienna Twin Tower" , 30586012, "Schloss Wilhelminenberg" , "NH Danube City", "Austria Center Vienna", "Kunsthistorisches Museum Wien", "Naturhistorisches Museum Wien","Ernst-Happel-Stadion", "Schloss Schönbrunn", "Gloriette" , "Hochhaus Neue Donau","Millennium Tower",  "AUVA Hauptstelle" ,  23681899 ,  23681898,  23681896, 27146873, 23681901, 23681900 ,"Ringturm" ,"Jubiläumswarte" ,"Rinterzelt" ,"Stift Klosterneuburg" ,"Stefaniewarte" ,26586105 , "Sender Kahlenberg" ,"St. Leopold" ,"Burg Greifenstein","Burg Kreuzenstein" ,"Aussichtswarte Oberleis" ,"ORS STM" ,"Buschberg", "Stephansdom","Rathaus","Pier Nord (Gates F, G)","Terminal 3 (check-in)","Pier Ost (Gates D)","Terminal 2 (check-in)","Terminal 1 (check-in)","Gates B", "T2",118931768,"Pier West","ehem ACC Ost Frachtterminal", "Hangar 3", "Hangar 4", 118931788, 658260437, 658260438, "Großhangar 2", "Großhangar 3", "Großhangar 1", "Feuerwehr 1", 118931796 ,118931785 ,118931785 ,118931754 ,103278755 ,  103278719 ,  103278733 , 444973468 ,   438544480 ,   23128561, 118931790 , 23128562 ,  "EDV 660", 438543637 ,138026612 ,23772451 ,118931787,444973467 ,"Parkhaus 8","Terminal 1A (check-in)","P 4 Nord", "P 4 Süd", "NH Vienna Airport", "Tower",  136987315 ,  136987314 ,"Office Park 3", "Office Park 1", 29877902,"Office Park 2", "Billa", "Feuerwehr 2","Burgtheater",  "Wiener Staatsoper", "Kirche Alt Ottakring" ,"Flakturm Augarten (Leitturm)","Flakturm Augarten (Gefechtsturm)", "Schloss Wilhelminenberg","Müllverbrennungsanlage Flötzersteig","Kirche am Steinhof","Kirche Alt Ottakring",354367724]



Bernhard
powoflight
 
Posts: 212
Joined: Fri Mar 25, 2016 11:04 am
Location: LOWW
Callsign: OE-POW
Version: 2020.4
OS: win7 Ubuntu 18.04

Re: osm2city.py development

Postby powoflight » Wed Apr 10, 2019 11:38 am

I have now narrowed the scope.

SpawnPoolWorker-1 root ERROR ******* Exception in tile 3220113 - to reprocess use boundaries: 16.35_48.25_16.43_48.275 *******
powoflight
 
Posts: 212
Joined: Fri Mar 25, 2016 11:04 am
Location: LOWW
Callsign: OE-POW
Version: 2020.4
OS: win7 Ubuntu 18.04

Re: osm2city.py development

Postby powoflight » Wed Apr 10, 2019 2:27 pm

In building_lib.py

def _write_faces_for_ac(self, ac_object: ac3d.Object, ring: shg.LinearRing,
.
.
In if statement :
if self.roof_shape is roofs.RoofShape.skillion:

Here the variablen are:
self.body_height 0.0
self.roof_height_pts[i] 0.8812032956446817
tex_coord_top_input 0.0

tex_y12 = self.facade_texture.y((self.body_height + self.roof_height_pts[i]) /
self.body_height * tex_coord_top_input)


self.body_height + self.roof_height_pts[i]) = 0.8812032956446817
self.body_height * tex_coord_top_input = 0

0.8812032956446817 / 0 => Division by zero
gives the ERROR
powoflight
 
Posts: 212
Joined: Fri Mar 25, 2016 11:04 am
Location: LOWW
Callsign: OE-POW
Version: 2020.4
OS: win7 Ubuntu 18.04

Re: osm2city.py development

Postby powoflight » Thu Apr 11, 2019 7:24 am

I have fixed this issue in: def _write_faces_for_ac
Roughly ... :?:
Code: Select all
    def _write_faces_for_ac(self, ac_object: ac3d.Object, ring: shg.LinearRing,
                            index_first_node_in_ac_obj: int, number_prev_ring_nodes: int, mat_idx: int) -> int:
        """Writes all the faces for one building's exterior or interior ring to an ac3d object."""
        tex_coord_bottom, tex_coord_top = _calculate_vertical_texture_coords(self.body_height, self.facade_texture)
        tex_coord_bottom = self.facade_texture.y(tex_coord_bottom)  # -- to atlas coordinates
        tex_coord_top_input = tex_coord_top
        tex_coord_top = self.facade_texture.y(tex_coord_top)

        number_ring_nodes = len(ring.coords) - 1

        for ioff in range(0, number_ring_nodes):
            i = number_prev_ring_nodes + ioff

            if ioff < number_ring_nodes - 1:
                ipp = i + 1
            else:
                ipp = number_prev_ring_nodes
            # FIXME: respect facade texture split_h
            # FIXME: there is a nan in textures.h_splits of tex/facade_modern36x36_12
            a = self.edge_length_pts[i] / self.facade_texture.h_size_meters
            ia = int(a)
            frac = a - ia
           
            tex_coord_right = self.facade_texture.x(self.facade_texture.closest_h_match(frac) + ia)
           
            if self.facade_texture.v_can_repeat:
                if not (tex_coord_right <= 1.):
                    logging.debug('FIXME: v_can_repeat: need to check in analyse')
                   
            if self.roof_shape is roofs.RoofShape.skillion:
 
                # =======================  Changed begin ======================
                if tex_coord_top_input == 0:
                    tex_y12 = tex_coord_top
                    tex_y11 = tex_coord_top
                    print("DEBUG if tex_coord_top_input == 0") # only for Debugging
                else:
                    tex_y12 = self.facade_texture.y((self.body_height + self.roof_height_pts[i]) /
                                                self.body_height * tex_coord_top_input)
               
                    print("DEBUG else") # only for Debugging
                    tex_y11 = self.facade_texture.y((self.body_height + self.roof_height_pts[ipp]) /
                                                self.body_height * tex_coord_top_input)
                # =======================  Changed end ======================
                 
            else:
                tex_y12 = tex_coord_top
                tex_y11 = tex_coord_top

            tex_coord_left = self.facade_texture.x(0)

            ac_object.face([(i + index_first_node_in_ac_obj, tex_coord_left, tex_coord_bottom),
                            (ipp + index_first_node_in_ac_obj, tex_coord_right, tex_coord_bottom),
                            (ipp + index_first_node_in_ac_obj + self.pts_all_count, tex_coord_right, tex_y11),
                            (i + index_first_node_in_ac_obj + self.pts_all_count, tex_coord_left, tex_y12)],
                           mat_idx=mat_idx,
                           swap_uv=self.facade_texture.v_can_repeat)
        return number_ring_nodes
powoflight
 
Posts: 212
Joined: Fri Mar 25, 2016 11:04 am
Location: LOWW
Callsign: OE-POW
Version: 2020.4
OS: win7 Ubuntu 18.04

Re: osm2city.py development

Postby pb321 » Thu Apr 11, 2019 3:49 pm

As a newcomer to osm2city, I want to thank the developers of some very fine scripts! The output is impressive, for sure!
So, as a newcomer, I risk asking questions that, to others, may have been answered long ago. But this thread is a long one, so I may have missed some of them. Here are some issues I’m having and would appreciate any thoughts/help:
1) Bridges: Some show up just fine, ramping nicely over a body of water, for example. But others show up as partially made? How can this be fixed?
2) Highways: Some highways seem to end abruptly when they shouldn't. How can these be fixed?
3) Buildings: A run of the State of Illinois (including Chicago) produced only three lat/long building files instead of the twenty or more that should be produced, yet there was no osm2city-exceptions.log produced. What should be the approach to fix this?
4) Should “.pkl” files be saved if the area will be re-run?
5) What are “roof_error” files? Should they be ignored or do they provide useful information?
6) How close to reality should be expected as output? Should only small areas be generated at a time with parameters tailored to that specific area? In other words, how do you decide when certain parameters are “right” and can’t be improved upon? Has anyone developed a checklist for developed scenery based on the capabilities of osm2city?

Thanks in advance!
pb321
 
Posts: 424
Joined: Sun Nov 27, 2016 5:08 pm
Version: 2020.4.0
OS: Windows 10 Pro

Re: osm2city.py development

Postby vanosten » Thu Apr 11, 2019 6:03 pm

powoflight wrote in Thu Apr 11, 2019 7:24 am:I have fixed this issue in: def _write_faces_for_ac
Roughly ... :?:

And the delta is? Also: are you sure you used the very latest version of osm2city?
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby vanosten » Thu Apr 11, 2019 6:09 pm

pb321 wrote in Thu Apr 11, 2019 3:49 pm:As a newcomer to osm2city, I want to thank the developers of some very fine scripts! The output is impressive, for sure!
So, as a newcomer, I risk asking questions that, to others, may have been answered long ago. But this thread is a long one, so I may have missed some of them. Here are some issues I’m having and would appreciate any thoughts/help:
1) Bridges: Some show up just fine, ramping nicely over a body of water, for example. But others show up as partially made? How can this be fixed?
2) Highways: Some highways seem to end abruptly when they shouldn't. How can these be fixed?
3) Buildings: A run of the State of Illinois (including Chicago) produced only three lat/long building files instead of the twenty or more that should be produced, yet there was no osm2city-exceptions.log produced. What should be the approach to fix this?
4) Should “.pkl” files be saved if the area will be re-run?
5) What are “roof_error” files? Should they be ignored or do they provide useful information?
6) How close to reality should be expected as output? Should only small areas be generated at a time with parameters tailored to that specific area? In other words, how do you decide when certain parameters are “right” and can’t be improved upon? Has anyone developed a checklist for developed scenery based on the capabilities of osm2city?

Thanks in advance!

1) Roads.py needs some love, but it is not easy
2) Same as 1)
3) not enough information
4) No. But they are handy if you regenerate, because it saves processing time. And: it depends on which pkl file.
5) Developer info for debugging
6) Parameters have to be tailored. That is basically why there are so many of them. The issue right now is that parameters are basically per tile. However they should be for "any" geometry. FG has the same issue with regions. Fixable, but difficult. In teh ideal world we would have a database of parameters per area, which people over time have contributed to. IMO the biggest problem by far right now is that the facades look terrible.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby powoflight » Fri Apr 12, 2019 7:03 am

Dear Vanosten

And the delta is? Also: are you sure you used the very latest version of osm2city?


I have pulled the last osm2city version on Tuesday 9. April.


And the delta is? I don't know exactly what do you mean.


But in the for loop:
for ioff in range(0, number_ring_nodes):
.
.
if self.roof_shape is roofs.RoofShape.skillion:

the variables are:
self.body_height: 0.0
self.roof_height_pts[i] : 0.8812032956446817
tex_coord_top_input : 0.0


And here we have a "ZeroDivisionError"

tex_y12 = self.facade_texture.y((self.body_height + self.roof_height_pts[i]) / 0.0 + 0.88 = 0.88 This is ok
self.body_height * tex_coord_top_input) 0.0 * 0.0 = 0

And a division 0.88/0 gives a ZeroDivisionError: float division by zero

Unfortunately, I do not know how to find out the corresponding OSM number.
powoflight
 
Posts: 212
Joined: Fri Mar 25, 2016 11:04 am
Location: LOWW
Callsign: OE-POW
Version: 2020.4
OS: win7 Ubuntu 18.04

Re: osm2city.py development

Postby vanosten » Sat Apr 13, 2019 3:02 pm

I have committed a simplified version of your fix. Please test.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby vanosten » Sat Apr 13, 2019 3:04 pm

If there is anyone out there, who has working knowledge of Cython, I would love to have an audio conversation. I consider enhancing the code with Cython to speed things up. However I do not want to make too many stupid beginner mistakes.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 4 guests