Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby vanosten » Sun Nov 25, 2018 7:44 am

stuart wrote in Sun Nov 18, 2018 10:08 pm:adam_one: I think you're just hitting the limitations of loading time for the large model files that osm2city is generating, unfortunately. There's also an interaction with OSG's PagedLOD which can result in models being unloaded earlier than you might want.

Rick and I have been discussing how to address this, and I've just pushed a change for 2018.4 which would allow using shaders for the OSM buildings: https://sourceforge.net/p/flightgear/ma ... /36470886/

-Stuart


The very first proof of concept is very promising. I have just run a first iteration over Edinburgh, which has about 72k buildings in tile 2892923. Writing all buildings either attached to another, having parent/child relations or being larger than 200 m2 the usual way and the rest with the new BUILDING_LIST STG-verb, results in ca. 25 MB disk used for meshes for ca. 13k buildings and 1.1 MB disk used for the test ca. 60k buildings in random buildings.

Not only is this a order of magnitude improvement in disk space, but my little old development notebook happily loads both - and the random buildings are loaded fast.

As you can tell from the pictures below (not mixed with mesh-based buildings) I will have to correct the corner and angle of buildings along streets and a lot of other stuff together with Stuart to get more variety etc. However the future looks very promising :-)

Given that Stuart's code is region based, there is regionalisation included from the start - so texture artist will be able to make sure that a house in Scandinavia looks different from one in Scotland or Gran Canaria or ...

Btw: the idea is that certain buildings in OSM (based on size and tags and geometry and ...) will continue to be generated in meshes, but the number might decrease over time.

Image

Image
Maintaining osm2city
vanosten
 
Posts: 385
Joined: Sat Sep 25, 2010 5:38 pm
Location: Denmark - but I am Swiss
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby legoboyvdlp » Sun Nov 25, 2018 10:01 am

Are these buildings built using heuristics or are the coordinates simply passed to the shader (i.e. each random building represents an actual building?)
User avatar
legoboyvdlp
 
Posts: 6416
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: osm2city.py development

Postby wlbragg » Sun Nov 25, 2018 10:56 am

Unrelated to @legoboyvdlp's question, regarding manufacturing buildings in an area that doesn't yet have any real buildings placed in OSM, I was wondering if you plan to provide any kind of check whereas if an actual building has been input in OSM it will override any random generated pseudo buildings.
What I am concerned with is, I like the idea of populating areas where OSM simply never had anybody input buildings yet but the track or street data is there so you can reasonably extrapolate and produce data for the area. But one thing that makes OSM so special is the realistic nature of the scenery where the data "has" been input into OSM. I hope that if an area is populated in OSM it would have priority over random generated pseudo buildings. Is that to be the case or are you diverging from using actual OSM data and opting more for data generated based on likely candidate based extrapolation logic?

What I am trying to say is I hope there will always remain an option in FG or osm2city, depending on where this eventually goes, to generate only the realistic "actual" data if one chooses, even if that means holes in the data due to lack of data. Maybe with the ideal situation being a combination of both, again as an option, where manufactured data will be generated in lieu of actual data if it doesn't exist yet.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
User avatar
wlbragg
 
Posts: 4495
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Debain/nVGT640

Re: osm2city.py development

Postby vanosten » Sun Nov 25, 2018 7:48 pm

wlbragg wrote in Sun Nov 25, 2018 10:56 am:Unrelated to @legoboyvdlp's question, regarding manufacturing buildings in an area that doesn't yet have any real buildings placed in OSM, I was wondering if you plan to provide any kind of check whereas if an actual building has been input in OSM it will override any random generated pseudo buildings.
What I am concerned with is, I like the idea of populating areas where OSM simply never had anybody input buildings yet but the track or street data is there so you can reasonably extrapolate and produce data for the area. But one thing that makes OSM so special is the realistic nature of the scenery where the data "has" been input into OSM. I hope that if an area is populated in OSM it would have priority over random generated pseudo buildings. Is that to be the case or are you diverging from using actual OSM data and opting more for data generated based on likely candidate based extrapolation logic?

What I am trying to say is I hope there will always remain an option in FG or osm2city, depending on where this eventually goes, to generate only the realistic "actual" data if one chooses, even if that means holes in the data due to lack of data. Maybe with the ideal situation being a combination of both, again as an option, where manufactured data will be generated in lieu of actual data if it doesn't exist yet.


I hope the following gives an answer:
  • There is a parameter OWBB_GENERATE_BUILDINGS, which decides whether or not plausible buildings are created in areas, where there is no OSM data, but where there is land-use data either in OSM or in BTG-files. I.e. if you do not like these buildings and rather have holes, then just set the parameter to False.
  • Once the buildings are created (no matter where they come from), they are treated the same way in later analysis.
  • There is right now a parameter FLAG_STG_BUILDING_LIST which decides, whether "some" buildings (hopefully > 70%) are rendered in shaders using Random Buildings or not. Setting this to False gives the possibility to have all in meshes (I might rename it, when the feature is "stable").
  • Even though it is "Random Buildings", it is not that random: the placement, size and nature (as far as Random Buildings allows it now or later) of the building will be based on an actual or plausible building. So it is a replacement.
  • Remember that for most buildings even from OSM the available data in tags is quite limited. So even though using simplified buildings in Random Buildings might hide some of the complexity of floor plans, all the rest is just interpretation based on heuristics.
  • Right now the textures used in Random Buildings are much better than the ones on osm2city-data. That will hopefully change some day, but I do not see a good way to regionalise osm2city-data. Also: IMO most regionalisation is relevant for detached buildings and row-houses. Large buildings etc. look "the same" all over - and make sense to include in meshes instead of Random Buildings.
  • So I believe that there will be enough parameters to generate just the scenery one wants. However the sceneries to be included in Terrasync need to be a bit optimised for both disk space and rendering performance - so I guess Random Buildings will be used somewhat aggressively.

Not sure whether I answered your questions, are there other concerns?
Maintaining osm2city
vanosten
 
Posts: 385
Joined: Sat Sep 25, 2010 5:38 pm
Location: Denmark - but I am Swiss
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby wlbragg » Sun Nov 25, 2018 11:23 pm

Yes it did and I like your approach, even the idea of shader based buildings that are loosely based on their detailed OSM counterpart.
Having the option to "opt out" and use only "absolute" data satisfies my concerns for absolute realistic OSM scenery.
The other thought was that should this ever be a built-in with the FG source application that we include the same customizations and opt-in/outs.
Thanks!
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
User avatar
wlbragg
 
Posts: 4495
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Debain/nVGT640

Re: osm2city.py development

Postby legoboyvdlp » Fri Nov 30, 2018 4:17 pm

Cars seem to be driving in opposite directions here?

Image

I wonder is it possible to force cars to drive on the left if the two carriageways are marked as a single road in osm?
User avatar
legoboyvdlp
 
Posts: 6416
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: osm2city.py development

Postby Thorsten » Fri Nov 30, 2018 5:16 pm

Actually the piece of road you show in the picture appears to be mistakenly on top of another road (?) - to the right there's proper two lane traffic into the same direction, which I suspect might also be underneath the left.
Thorsten
 
Posts: 10433
Joined: Mon Nov 02, 2009 8:33 am

Re: osm2city.py development

Postby wlbragg » Fri Nov 30, 2018 8:48 pm

Or an interchange where the top road is an offramp to change highways and supposed to be at a different elevation.
Kansas(2-27-15)/Ohio/Midwest scenery development.
KEQA (2-27-15), 3AU, KRCP Airport Layout
User avatar
wlbragg
 
Posts: 4495
Joined: Sat Aug 25, 2012 11:31 pm
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Debain/nVGT640

Re: osm2city.py development

Postby adam_one » Mon Dec 03, 2018 10:42 pm

stuart wrote in Sun Nov 18, 2018 10:08 pm:adam_one: I think you're just hitting the limitations of loading time for the large model files that osm2city is generating, unfortunately. There's also an interaction with OSG's PagedLOD which can result in models being unloaded earlier than you might want.

Rick and I have been discussing how to address this, and I've just pushed a change for 2018.4 which would allow using shaders for the OSM buildings: https://sourceforge.net/p/flightgear/ma ... /36470886/

-Stuart

Thanks for the answer.
adam_one
 
Posts: 64
Joined: Tue Jan 05, 2016 2:41 pm
Version: alpha
OS: pirated

Re: osm2city.py development

Postby danw » Sat Dec 22, 2018 9:34 pm

I'm trying out osm2city on Charleston SC, and I get a path error trying to generate the output:

******* Exception in tile 1629859 - to reprocess use boundaries: -80.2_32.6_-80.0_32.625 ******* at 2018-12-22_161801 -

Traceback (most recent call last):
File "C:\Users\daniel\osm2city\bin\osm2city\build_tiles.py", line 144, in process_scenery_tile
osm_buildings, file_lock)
File "C:\Users\daniel\osm2city\bin\osm2city\buildings.py", line 780, in process_buildings
write_buildings_in_meshes(coords_transform, buildings_in_meshes, stg_manager, stats)
File "C:\Users\daniel\osm2city\bin\osm2city\buildings.py", line 697, in write_buildings_in_meshes
cluster_elev, cluster_offset, prepare_textures.roofs, stats)
File "C:\Users\daniel\osm2city\bin\osm2city\building_lib.py", line 1500, in write
ac.write(ac_file_name)
File "C:\Users\daniel\osm2city\bin\osm2city\utils\ac3d.py", line 288, in write
f = open(file_name, 'w')
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\daniel\\flightgear\\customscenery\\kchs\\Buildings\\w090n30\\w081n32\\w090n30\\w081n32\\1629859city00200.ac'

It's odd in the end of the directory structure seems to be duplicated: Buildings\\w090n30\\w081n32\\w090n30\\w081n32

The output file in my params.ini file is defined as PATH_TO_OUTPUT="C:\\Users\\daniel\\flightgear\\customscenery\\kchs".

In the output directory the C:\\Users\\daniel\\flightgear\\customscenery\\kchs\\Buildings\\w090n30\\w081n32 path is created, but nothing is populated in the directories.

Note: this is a win 10 environment with an anaconda python env setup for python 3.5, and postgresql 9.6.

Any suggestions as to what the problem might be?
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 2:47 am

Re: osm2city.py development

Postby legoboyvdlp » Sat Dec 22, 2018 10:00 pm

Hi,
I had this as well and fixed it by changing parameters.py replacing line 387 with:

Code: Select all
 
    return re.sub('[\\]', '_', PREFIX)


I am on mobile so I may be incorrect as to how I fixed it, but this is almost certainly how. I will check in the morning.
User avatar
legoboyvdlp
 
Posts: 6416
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: osm2city.py development

Postby danw » Sat Dec 22, 2018 11:30 pm

I've found that the file path wasn't being computed correctly - or maybe it's an issue related to my file structure?

With the following changes, it seems to work properly:

diff --git a/buildings.py b/buildings.py
index 66eeb43..a3c9c8f 100755
--- a/buildings.py
+++ b/buildings.py
@@ -692,6 +692,8 @@ def write_buildings_in_meshes(coords_transform: coordinates.Transformation,
path_to_stg = stg_manager.add_object_static(file_name + '.ac', center_global, cluster_elev, 0,
my_clusters.stg_verb_type)

+ file_name = os.path.basename(file_name)
+ logging.info("path_to_stg %s %s" % (path_to_stg, file_name))
# -- write .ac and .xml
building_lib.write(os.path.join(path_to_stg, file_name + ".ac"), cl.objects,
cluster_elev, cluster_offset, prepare_textures.roofs, stats)
diff --git a/piers.py b/piers.py
index 53e1721..28463e7 100644
--- a/piers.py
+++ b/piers.py
@@ -94,6 +94,8 @@ def _write_piers(stg_manager, replacement_prefix, clusters, coords_transform: co
else:
_write_pier_line(pier, obj, cl.center)
path = stg_manager.add_object_static(ac_file_name, center_tile, 0, 0)
+
+ ac_file_name = os.path.basename(ac_file_name)
file_name = os.path.join(path, ac_file_name)
f = open(file_name, 'w')
f.write(str(ac))
diff --git a/pylons.py b/pylons.py
index 6af1530..3c7e760 100755
--- a/pylons.py
+++ b/pylons.py
@@ -1546,6 +1546,9 @@ def _write_cable_clusters(cluster_container: cluster.ClusterContainer, coords_tr
ac_file_lines.append("OBJECT world")
ac_file_lines.append("kids " + str(len(cl.objects)))
segment_index = 0
+ cluster_filename = os.path.basename(cluster_filename)
+ logging.info("path_to_stg %s %s" % (path_to_stg, cluster_filename))
+
for way_segment in cl.objects:
segment_index += 1
ac_file_lines.append("OBJECT group")
diff --git a/roads.py b/roads.py
index 791f6d9..ec47c86 100755
--- a/roads.py
+++ b/roads.py
@@ -1280,6 +1280,9 @@ def _process_clusters(clusters, replacement_prefix, fg_elev: utilities.FGElev, s
path_to_stg = stg_manager.add_object_static(file_name + '.ac', center_global, cluster_elev, 0,
stg_verb_type)
stg_paths.add(path_to_stg)
+ file_name = os.path.basename(file_name)
+ logging.info("path_to_stg %s %s" % (path_to_stg, file_name))
+
ac.write(os.path.join(path_to_stg, file_name + '.ac'))

for the_way in cl.objects:
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 2:47 am

Re: osm2city.py development

Postby legoboyvdlp » Sun Dec 23, 2018 12:27 am

When I tried fixing it in the separate files I got wierd results due to conflicts. I would suggest trying my fix above first to see if it works?
User avatar
legoboyvdlp
 
Posts: 6416
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

Re: osm2city.py development

Postby danw » Sun Dec 23, 2018 2:07 pm

legoboyvdlp wrote in Sat Dec 22, 2018 10:00 pm:Hi,
I had this as well and fixed it by changing parameters.py replacing line 387 with:


Code: Select all
 
    return re.sub('[\\]', '_', PREFIX)



This doesn't seem to be related to the problem that I'm seeing. I don't have slashes in my prefix, so this is a NOP either way.
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 2:47 am

Re: osm2city.py development

Postby legoboyvdlp » Sun Dec 23, 2018 3:30 pm

Hi,
I know, but each built tile has a individual PREFIX which also is called PREFIX in the source code (actually, I think the resulting PREFIX is equal to the parameter PREFIX plus the path plus the tile id):
Code: Select all
C:\\Users\\daniel\\flightgear\\customscenery\\kchs\\Buildings\\w090n30\\w081n32\\w090n30\\w081n32\\1629859city00200.ac
with this fix would turn into
Code: Select all
C:\\Users\\daniel\\flightgear\\customscenery\\kchs\\Buildings\\w090n30\\w081n32\\w090n30_w081n32_1629859city00200.ac

Try it, I had this exact problem and this is exactly how I fixed it :)
User avatar
legoboyvdlp
 
Posts: 6416
Joined: Sat Jul 26, 2014 1:28 am
Callsign: YV-LEGO
Version: 2018.3.1
OS: Windows 10 HP

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 1 guest