Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby vanosten » Tue Jun 20, 2017 3:58 pm

I am planning to remove the possibility to load OSM data directly from files, because:
  • Possibility to remove several 100 lines of code incl. switch logic -> less maintenance and fewer sources of error
  • I actually do not know, whether the file interface still works, as I have not used it for many months - and I have no intention to do testing on several data interfaces
  • A minor part of logic does not even filter correctly when using files
  • It is much easier to develop due to the SQL interface to the database
  • The DB approach works not only for me and is the future in order to support automatic generation of sceneries on the server
  • It is easier to create/update OSM data in a database than in files (e.g. I will soon migrate a better way of generating missing OSM landuse data)

The drawback is of course that in order to generate a scenery, you have to have a recent PostGIS installation, it takes additional disc space to store the data in the database and adds a few more steps.

If you have any reasons why I should not do so, then please tell me (all things equal the old code will still be available in GIT).
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby slawekmikula » Tue Jun 20, 2017 7:40 pm

thanks for release plan info. I fully understand :) Same for me.

Nonetheless regarding using OSM files directly. Frankly for me it did not work any time. Maybe for very limited areas it worked, but for larger areas processing failed. I do not remember but it was something about memory (still I have 16GB). That's why i've created these helper scripts to process areas directly available on geofabrik (https://github.com/slawekmikula/scripts-osm2city) via PostGIS DB. It has some drawbacks when there is no large storage (as in my case) for larger areas. I was thinking to change these scripts to provide whole area splitting and process it with loading to PostGIS db only partially. But right now it works and can process repeatedly anything from geofabrik in one run.

So for me files processing is unnecessary.
slawekmikula
 
Posts: 128
Joined: Sun Feb 19, 2017 10:31 am

Re: osm2city.py development

Postby Thorsten » Wed Jun 21, 2017 6:36 am

IMHO integration was a bit rushed the last 2 releases and now there are compatibility issues - however e-mail conversations on the devel-list tend to frustrate me. So unfortunately I cannot tell for how long these compatibility issues will exist - nor when embedding osm2city into Terrasync will be done for real.


I think we're facing a chicken and egg issue:

* Effect framework side, I'd like to be sure that things are stable - if there are no requests which affect the data structure, framework-side we're good to go (and the framework can always be extended with new features for the existing data).

* however, you asked for a testing and evaluation period when James suggested we go for mass production next

* TorstenD would probably like to be sure that there's no changes in what goes to TS, because the data is unwieldy and ill-suited for frequent repository updates, so he wants to avoid a situation where material gets committed, then a change announced and a request made to update the material to a new format three months later

So I think the case to go to TS is much better once it's forseeable that the data that is intended for TS won't be requested to be updated on a timescale of, say, a year at least and the whole data structure is declared stable (note that we can still change regional definitions, textures, maps, effects etc. indepdenently even for material already on TS). As long as you are not ready to commit to stable, large-scale TS uploads will be difficult to argue.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: osm2city.py development

Postby vanosten » Sun Jun 25, 2017 8:17 am

Thorsten wrote in Sun Jun 11, 2017 7:46 am:I've just pushed a change that dependent on view angle to the car you should see tail lights. Also, part of the road illumination is now determined by the amount of traffic on the road, roads will be more brightly lit if there's plenty of cars on them.

Please double-check that this plays nice with the dual-lane roads and generally gives a good impression (the traffic model is now rather sophisticated with daily peaks during rush hours, so dependent on whether the rush hour happens in darkness or you see roads at midnight, the visuals should be quite different).


The back lights now show nicely and the time-of-day effect is quite nice. One could then consider taking another step by osm2city giving a hint, whether a given street is within a big city or a small town etc. However that is currently not at the top of my list.

I have a few observations, which you might consider (maybe I have done something wrong - however in general I run "full" ALS):
  • During night it seems to me that on small roads the traffic almost exclusively run in one direction. It works fin on larger roads and motorway.
  • The motorway seems too dark to me when lit. You can barely see a hint of red light.
  • During day the embankment (green) somehow also get light effects - they should have none
  • It seems to me that there a quite significant differences in the darkness of streets during day - and it does not seem to be related to exposure to sun or traffic density.
  • There are V-shaped residuals on both streets and embankments

The following three pictures might illustrate it:
https://my.pcloud.com/publink/show?code=XZVKyvZC7aC9d87IGVSpqts9CtXIXa5OoMV
https://my.pcloud.com/publink/show?code=XZ0KyvZseW8gvcHs00QKR0sutoAXQdi6W3X
https://my.pcloud.com/publink/show?code=XZFKyvZeiEvDDwz22FoF9TmkhAKtbpmer8X
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby Thorsten » Sun Jun 25, 2017 8:38 am

Admittedly I have seen none of those in my test area (still Bergen). So I have to guess a bit.

During night it seems to me that on small roads the traffic almost exclusively run in one direction. It works fin on larger roads and motorway.


I can't come up with a reason how this could happen - the algorithm is symmetric in road side.

The motorway seems too dark to me when lit. You can barely see a hint of red light.


Could you clarify whether this is about day or night?

Also, whether the light flag is actually set or not? A motorway without street lights and low traffic is going to be very dark at night.

During day the embankment (green) somehow also get light effects - they should have none


How do they actually call the effect? They're part of the same mesh, so it's just a different texture slot, right? I suppose they do get stray light but not streetlights.

It seems to me that there a quite significant differences in the darkness of streets during day - and it does not seem to be related to exposure to sun or traffic density.


At least in one of the pics it seems to be one lane dark, the opposite lane bright - so is it possible you have the normal direction flipped in the generation process (one direction generates normals upward and gets direct light, the other direction has normals downward and has to make do with ambient light)?

There are V-shaped residuals on both streets and embankments


They look like artifacts in the surface normals as well (water sometimes gets similar things when slope bleeds into it from the surroundings). More specifically, it's probably the specular channel firing, the roads are set to be quite reflective, and that makes small mesh flaws show up prominently.

You can try looking at this in wireframe whether anything sticks out.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: osm2city.py development

Postby miguel » Thu Jun 29, 2017 3:10 pm

for my feature of vanosten is ok,and it works
miguel
 
Posts: 226
Joined: Wed Aug 19, 2015 5:05 pm

Re: osm2city.py development

Postby adam_one » Wed Jul 26, 2017 11:33 pm

Vanosten, do you happen to know why Swedish cities ran through buildings.py result in:

Code: Select all
EBUG:root:Replacing texture for flat roof despite ['roof:default', 'compat:roof-pitched']
Traceback (most recent call last):
  File "/home/adam/Downloads/osm2city/buildings.py", line 748, in <module>
    process(my_coords_transform, my_fg_elev, my_blocked_areas, my_stg_entries)
  File "/home/adam/Downloads/osm2city/buildings.py", line 708, in process
    cluster_elev, cluster_offset, prepare_textures.roofs, stats)
  File "/home/adam/Downloads/osm2city/building_lib.py", line 983, in write
    _write_roof_for_building(ac_object, b, roof_mgr, cluster_offset, stats)
  File "/home/adam/Downloads/osm2city/building_lib.py", line 920, in _write_roof_for_building
    roofs.separate_gable(ac_object, b)
  File "/home/adam/Downloads/osm2city/roofs.py", line 120, in separate_gable
    tang = (b.X[ind_X[1]]-b.X[ind_X[0]])/b.lenX[ind_X[1]] * inward_meters
AttributeError: 'Building' object has no attribute 'lenX'
adam_one
 
Posts: 64
Joined: Tue Jan 05, 2016 3:41 pm
Version: alpha
OS: pirated

Re: osm2city.py development

Postby vanosten » Thu Jul 27, 2017 7:30 am

adam_one wrote in Wed Jul 26, 2017 11:33 pm:Vanosten, do you happen to know why Swedish cities ran through buildings.py result in:

Code: Select all
AttributeError: 'Building' object has no attribute 'lenX'


It is a bug introduced by me ca. 3 weeks ago. It should work if you search/replace b.lenX with b.edge_length_x in roofs.py. However: could you please send me the coordinates of the tile, where it is failing, so I can test with the newest code?
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby Zequinha » Thu Jul 27, 2017 11:26 am

paju1986 wrote in Thu Apr 20, 2017 7:39 pm:Hello.

Will be possible to make roads to "connect" to static scenery bridges?


Concerning this bridge in specific (Ponte 25 de Abril), I was thinking of improving this static model myself. I made the connection of 'Vasco da Gama' bridge with tthe ground roads, so it should be similar for this one too. I have no idea how the connection of 'Vasco da Gama' bridge looks under osm2city.py though.

In the following image, it is possible to see that there is already a connection between Ponte 25 de Abril and the ground. However, this osm2city connection does not match up the static model height. This is a problem. First, how can we check which of the heights is correct?

Image
Last edited by Zequinha on Fri Jul 28, 2017 4:24 pm, edited 1 time in total.
Zequinha
 
Posts: 204
Joined: Sun Mar 13, 2016 5:26 pm
OS: GNU

Re: osm2city.py development

Postby vanosten » Fri Jul 28, 2017 7:12 am

Zequinha wrote in Thu Jul 27, 2017 11:26 am:
paju1986 wrote in Thu Apr 20, 2017 7:39 pm:Hello.

Will be possible to make roads to "connect" to static scenery bridges?
....


Could you please consider adding some new text? There were quite a few answers/discussions to that post. So what is the reason for reposting?
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby Zequinha » Fri Jul 28, 2017 4:23 pm

Sorry my mistake in the quotes! I've fixed it now!
Zequinha
 
Posts: 204
Joined: Sun Mar 13, 2016 5:26 pm
OS: GNU

Re: osm2city.py development

Postby vanosten » Mon Jul 31, 2017 7:45 am

Also your corrected question has been answered previously. I do not have any easy solution for this. The best thing might be to model ramps in the static model assuming there would be no osm2city (which should be the default assumption for making static models no matter what).
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby saul » Sun Aug 27, 2017 10:24 am

Hi,

I have finally found some time to test the latest developments (commit Aug 13). For me, it sounds natural to support only the database code: that's the way to go. However, many us have never used a psql database before, and most probably will only use it in this context. In my case it took me some time to figure out some details before I got it working. I would strongly suggest to expand a little bit the section "OSM Data in a PostGIS Database" in the documentation. I mainly faced 2 problems: how to set a valid password for the giuser, and the default listening port which was different in the guide. Here down are some notes that I wrote that my be helpful for a newbie:

Useful command line commands:

Open a database:
psql database_name

Delete a database:
sudo -u postgres dropdb database_name

Open a database as a database superuser:
sudo -u postgres psql database_name

quit database:
\q

See the user roles:
\du

Problem 1: giuser password not set


Change the roles:
https://www.postgresql.org/message-id/4 ... granch.com

Open the database:

sudo -u postgres psql stockholm


check the roles:

\du

change the password

alter user giuser password 'hola';



Problem 2: The listeing port in OSM instructions is wrong:

The default port is 5432

Working line:

~/osmosis --read-pbf file="sto.pbf" --log-progress --write-pgsql database=stockholm host=localhost:5432 user=giuser password=hola

Problem 3:

/home/vanosten/bin/osmosis-latest/bin/osmosis --read-pbf "/media/sf_fg_customscenery/projects/TEST/kbos.pbf" .....

should be

/home/vanosten/bin/osmosis-latest/bin/osmosis --read-pbf file="/media/sf_fg_customscenery/projects/TEST/kbos.pbf" ....


After these changes, the scripts were working. However, roads.py produced an error due to selfinteresection. My params.ini is:

PREFIX = "STOCKHOLM"

BOUNDARY_WEST = 17.0082
BOUNDARY_SOUTH = 59.0335
BOUNDARY_EAST = 19.1643
BOUNDARY_NORTH = 59.6830

BUILDING_MIN_AREA = 5.0
BUILDING_MIN_HEIGHT = 2.0

PATH_TO_SCENERY = "/home/saul/.fgfs/TerraSync"
PATH_TO_OUTPUT = "/home/saul/flightgear/custom_objects/OUTSVER"

PATH_TO_OSM2CITY_DATA = "/home/saul/flightgear/osm2city/osm2city-data"

# OSM_FILE = "buildings.osm"
USE_DATABASE = True
DB_HOST = "localhost"
DB_PORT = 5432
DB_NAME = "stockholm"
DB_USER = "giuser"
DB_USER_PASSWORD = "hola"

NO_ELEV = False
FG_ELEV = "/home/saul/flightgear/next/install/flightgear/bin/fgelev"

TEXTURES_REGIONS_EXPLICIT = ["de"]
FLAG_2017_2 = True

And the log of roads.py is:


WARNING:root:hstore for osm_id=32148315 has not valid key/value pair: '['construction_date"', '"2009-09-22 ', '"']' in '"building"=>"apartments", "operator"=>"http://www.SKB.org", "architect"=>"Nils Smedmark Arkitekter", "roof:shape"=>"gabled", "start_date"=>"2011", "roof:colour"=>"red", "name_kvarter"=>"kvarteret Glottran", "building:colour"=>"lemonchiffon", "building:levels"=>"5", "construction_date"=>"2009-09-22 =>"'.
INFO:root:Reading OSM way data for ['building'] from db took 11.5300 seconds.
Self-intersection at or near point -27092.710166496017 19324.843601587061
WARNING:shapely.geos:Self-intersection at or near point -27092.710166496017 19324.843601587061
INFO:root:Candidate land-uses with sufficient area added: 12834
INFO:root:Spawning fgelev
Found hole of minimum diameter 0.16384m at lon = 18.0364deg lat = 59.3041deg
INFO:root:Originally lit 3298 - generated lit 41256 - no lit 28502
INFO:root:Added 22789 new streets to existing 50267 highways
INFO:root:There were 28730 existing highways with at least 1 intersection
INFO:root:Originally lit 0 - generated lit 19960 - no lit 10836
INFO:root:Added 8007 new streets to existing 22789 highways
INFO:root:There were 25423 existing highways with at least 1 intersection
Traceback (most recent call last):
File "roads.py", line 1386, in <module>
process(my_coords_transform, my_fg_elev, my_blocked_areas, my_stg_entries)
File "roads.py", line 1337, in process
roads.process(blocked_areas, stg_entries, landuses_lit, stats) # does the heavy lifting incl. clustering
File "roads.py", line 418, in process
self._cleanup_topology()
File "roads.py", line 876, in _cleanup_topology
way.refs.remove(ref)
ValueError: list.remove(x): x not in list

Please let me know if there is something that I am doing wrong.
Many thanks!

Saul
saul
 
Posts: 40
Joined: Tue Nov 26, 2013 10:57 pm

Re: osm2city.py development

Postby vanosten » Mon Aug 28, 2017 7:46 pm

saul wrote in Sun Aug 27, 2017 10:24 am:... and the default listening port which was different in the guide ...


Thanks for the input regarding the documentation. I will update. With regards to the DB-port: apparently Postgres changed it from 5432 to 5433 between version 9.5 and 9.6 or so.

saul wrote in Sun Aug 27, 2017 10:24 am:And the log of roads.py is:


WARNING:root:hstore for osm_id=32148315 has not valid key/value pair: '['construction_date"', '"2009-09-22 ', '"']' in '"building"=>"apartments", "operator"=>"http://www.SKB.org", "architect"=>"Nils Smedmark Arkitekter", "roof:shape"=>"gabled", "start_date"=>"2011", "roof:colour"=>"red", "name_kvarter"=>"kvarteret Glottran", "building:colour"=>"lemonchiffon", "building:levels"=>"5", "construction_date"=>"2009-09-22 =>"'.
INFO:root:Reading OSM way data for ['building'] from db took 11.5300 seconds.
Self-intersection at or near point -27092.710166496017 19324.843601587061
WARNING:shapely.geos:Self-intersection at or near point -27092.710166496017 19324.843601587061
INFO:root:Candidate land-uses with sufficient area added: 12834
INFO:root:Spawning fgelev
Found hole of minimum diameter 0.16384m at lon = 18.0364deg lat = 59.3041deg
INFO:root:Originally lit 3298 - generated lit 41256 - no lit 28502
INFO:root:Added 22789 new streets to existing 50267 highways
INFO:root:There were 28730 existing highways with at least 1 intersection
INFO:root:Originally lit 0 - generated lit 19960 - no lit 10836
INFO:root:Added 8007 new streets to existing 22789 highways
INFO:root:There were 25423 existing highways with at least 1 intersection
Traceback (most recent call last):
File "roads.py", line 1386, in <module>
process(my_coords_transform, my_fg_elev, my_blocked_areas, my_stg_entries)
File "roads.py", line 1337, in process
roads.process(blocked_areas, stg_entries, landuses_lit, stats) # does the heavy lifting incl. clustering
File "roads.py", line 418, in process
self._cleanup_topology()
File "roads.py", line 876, in _cleanup_topology
way.refs.remove(ref)
ValueError: list.remove(x): x not in list


Please try replacing line 876 with
Code: Select all
if ref in way.refs: way.refs.remove(ref)
. It might work.

Unfortunately there seem to be a few logical errors in roads.py - I might have to write thorough unit tests with synthetic test data to get rid of them.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby saul » Fri Sep 01, 2017 9:44 pm

Hi Vanosten,

The script finished correctly after the change. I will create a database with a larger area and test it again.

Many thanks,

Saul
saul
 
Posts: 40
Joined: Tue Nov 26, 2013 10:57 pm

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 9 guests