Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby danw » Sun Dec 23, 2018 5:11 pm

I had to add an extra back slash for the regex to compile, but it worked.

Code: Select all
return re.sub('[\\\]', '_', PREFIX)


Thanks for your help.
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby danw » Sun Dec 23, 2018 10:39 pm

So now I'm able to generate osm2city scenery for Charleston, and it seems to work well for the downtown area (peninsula area down to the harbor). However I'm finding it difficult to get osm2city to generate residential areas.

The settlement type map for the area is as follows:
Image

and the wouldbe buildings debug output (yellow indicates where buildings would be generated):
Image

The areas marked in yellow (corresponds to periphery on map legend) are the areas where I'm finding it difficult to generate the residential areas. I get larger buildings like aparments and commercial buildings, but there doesn't seem to be many single-family type dwellings.

My config is as follows:

NO_ELEV=False
PROBE_FOR_WATER=True
FG_ELEV='C:\\fgmeta\\archive\\fg-project\\install\\msvc141-64\\bin\\fgelev.exe'
BUILDING_MIN_AREA=10.0
BUILDING_REDUCE_CHECK_TOUCH=True
OWBB_GENERATE_BUILDINGS=True
OWBB_USE_BTG_LANDUSE=True
OWBB_GENERATE_LANDUSE_BUILDING_BUFFER_DISTANCE=10
BUILDING_NEVER_SKIP_LEVELS=2
FG_ELEV_CACHE=True
WRITE_CLUSTER_STATS=True
DEBUG_PLOT_LANDUSE=True
DEBUG_PLOT_GENBUILDINGS=True

Does anyone have suggestions on parameters that I should try to get better fill in the residential areas?

Thanks,

Dan
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby vanosten » Mon Dec 24, 2018 9:16 am

Hi Dan

Thanks for the try and the details. It always works in the area where I develop - so good to see another place. I will give it a try over x-mas days. And I will look at the Windows vs. *nix path issue.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby vanosten » Tue Dec 25, 2018 8:00 am

I have pushed a fix to the path problem (which works on Linux, but is not tested on Windows).
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby danw » Tue Dec 25, 2018 5:16 pm

Your new fix works on windows. Thanks.
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby vanosten » Tue Dec 25, 2018 10:29 pm

danw wrote in Sun Dec 23, 2018 10:39 pm:The areas marked in yellow (corresponds to periphery on map legend) are the areas where I'm finding it difficult to generate the residential areas. I get larger buildings like aparments and commercial buildings, but there doesn't seem to be many single-family type dwellings.
...
Does anyone have suggestions on parameters that I should try to get better fill in the residential areas?


There are two answers to the question:
  • The buildings generated are hard-coded - maybe they are too large. Making that configurable is not the plan - the plan is to use configurable shaders for it. So you need a bit of patience.
  • Play with smaller values for OWBB_RESIDENTIAL_PERIPHERY_TERRACE_SHARE and OWBB_RESIDENTIAL_PERIPHERY_APARTMENT_SHARE.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby vanosten » Wed Dec 26, 2018 9:14 am

2 small improvements:
  • If the population is not included in the tags for a Place in OSM, then the population from WikiData is fetched over the internet if available.
  • If residential building with 1-3 levels is in steep terrain, then an extra level is added.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby danw » Thu Dec 27, 2018 3:24 pm

It looks like I get an exception querying wikidata with your latest update:

******* Exception in tile 1629883 - to reprocess use boundaries: -80.23_32.875_-80.0_33.0 ******* at 2018-12-27_081924 -

Traceback (most recent call last):
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\http\client.py", line 1198, in getresponse
response.begin()
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\http\client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\socket.py", line 576, in readinto
return self._sock.recv_into(b)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\contrib\pyopenssl.py", line 294, in recv_into
raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\adapters.py", line 440, in send
timeout=timeout
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\util\retry.py", line 357, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
raise value
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 389, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 309, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=1.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\daniel\osm2city\bin\build_tiles.py", line 138, in process_scenery_tile
lit_areas, osm_buildings = ol.process(the_coords_transform, my_airports)
File "C:\Users\daniel\osm2city\bin\owbb\landuse.py", line 699, in process
urban_places, farm_places = m.process_osm_place_refs(transformer)
File "C:\Users\daniel\osm2city\bin\owbb\models.py", line 1353, in process_osm_place_refs
my_place.parse_tags_additional(relation.tags)
File "C:\Users\daniel\osm2city\bin\owbb\models.py", line 117, in parse_tags_additional
my_population = wio.query_population(my_id)
File "C:\Users\daniel\osm2city\bin\utils\wikidata_io.py", line 40, in query_population
r = requests.get(url=query, timeout=1.)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\adapters.py", line 521, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=1.0)
******* Exception in tile 1646248 - to reprocess use boundaries: -80.0_32.625_-79.75_32.75 ******* at 2018-12-27_082004 -

Traceback (most recent call last):
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\http\client.py", line 1198, in getresponse
response.begin()
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\http\client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\socket.py", line 576, in readinto
return self._sock.recv_into(b)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\contrib\pyopenssl.py", line 294, in recv_into
raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\adapters.py", line 440, in send
timeout=timeout
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\util\retry.py", line 357, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
raise value
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 389, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\urllib3\connectionpool.py", line 309, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=1.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\daniel\osm2city\bin\build_tiles.py", line 138, in process_scenery_tile
lit_areas, osm_buildings = ol.process(the_coords_transform, my_airports)
File "C:\Users\daniel\osm2city\bin\owbb\landuse.py", line 699, in process
urban_places, farm_places = m.process_osm_place_refs(transformer)
File "C:\Users\daniel\osm2city\bin\owbb\models.py", line 1353, in process_osm_place_refs
my_place.parse_tags_additional(relation.tags)
File "C:\Users\daniel\osm2city\bin\owbb\models.py", line 117, in parse_tags_additional
my_population = wio.query_population(my_id)
File "C:\Users\daniel\osm2city\bin\utils\wikidata_io.py", line 40, in query_population
r = requests.get(url=query, timeout=1.)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "C:\Users\daniel\AppData\Local\conda\conda\envs\osm2city_py35\lib\site-packages\requests\adapters.py", line 521, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=1.0)
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby danw » Thu Dec 27, 2018 3:31 pm

  • Play with smaller values for OWBB_RESIDENTIAL_PERIPHERY_TERRACE_SHARE and OWBB_RESIDENTIAL_PERIPHERY_APARTMENT_SHARE.


I incorporated your suggestions and added a few more tweaks, and it's working better, as I now get decent fill in the periphery areas.

My current settings:

NO_ELEV=False
PROBE_FOR_WATER=True
BUILDING_MIN_AREA=10.0
BUILDING_REDUCE_RATE = 0.01
BUILDING_MIN_HEIGHT = 1.0
BUILDING_REDUCE_CHECK_TOUCH=True
LOD_ALWAYS_ROUGH_ABOVE_AREA=10.0
LOD_ALWAYS_ROUGH_ABOVE_LEVELS=0
LOD_PERCENTAGE_DETAIL=0.01
MIN_ABOVE_GROUND_LEVEL=0.1
POINTS_ON_LINE_DISTANCE_MAX=250
OWBB_GENERATE_BUILDINGS=True
OWBB_USE_BTG_LANDUSE=True
OWBB_MIN_CITY_BLOCK_AREA=50.0
OWBB_RESIDENTIAL_PERIPHERY_TERRACE_SHARE=0.9
OWBB_RESIDENTIAL_PERIPHERY_APARTMENT_SHARE=0.05
BUILDING_NEVER_SKIP_LEVELS=1
HIGHWAY_TYPE_MIN=3
FG_ELEV_CACHE=True
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby danw » Thu Dec 27, 2018 3:34 pm

Roads were going up onto the helipads, so I made the following change:

Code: Select all
$ git diff
diff --git a/utils/aptdat_io.py b/utils/aptdat_io.py
index 9e5ec71..45a7188 100644
--- a/utils/aptdat_io.py
+++ b/utils/aptdat_io.py
@@ -99,7 +99,7 @@ class Helipad(Runway):

     def create_blocked_area(self, coords_transform):
         my_point = Point(coords_transform.to_local((self.center.x, self.center.y)))
-        return my_point.buffer(math.sqrt(self.length + self.width) / 2)
+        return my_point.buffer(math.sqrt(self.length + self.width) * 2)


 class Airport(object):
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby vanosten » Thu Dec 27, 2018 10:02 pm

danw wrote in Thu Dec 27, 2018 3:34 pm:Roads were going up onto the helipads, so I made the following change:


Could you please tell the coordinates?

I just pushed another change, which should help remove roads on airports like Glasgow.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby danw » Fri Dec 28, 2018 12:09 am

I saw it here (from apt.dat):

17 15 0 1 SC71 [H] Medical University of South Carolina
102 H1 32.78712083 -079.95258889 0.00 15.24 15.24 2 2 0 0.25 0
19 32.78732655 -079.95234418 1 WS

I also see that same issue in the released code at the following location in Oahu:

17 175 0 1 HI40 [H] Kuakini Medical Center
102 H1 21.31350166 -157.84725695 0.00 27.43 27.43 2 2 0 0.25 0
19 21.31387196 -157.84685946 1 WS
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby vanosten » Fri Dec 28, 2018 3:28 pm

I pushed a fix, which should be more "correct". However I cannot find the helipads above in the scenery: could you please check?
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 540
Joined: Sat Sep 25, 2010 6:38 pm
Location: Denmark - but I am Swiss
Pronouns: he/his
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby danw » Fri Dec 28, 2018 4:21 pm

You should be able to pull them up in the launcher using the location search or from fgfs using search in the set location dialog.

Screenshot from Kuakini:

Image
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

Re: osm2city.py development

Postby danw » Fri Dec 28, 2018 5:08 pm

vanosten wrote in Fri Dec 28, 2018 3:28 pm:I pushed a fix, which should be more "correct". However I cannot find the helipads above in the scenery: could you please check?


Your fix looks good at the medical university helipad.
danw
 
Posts: 17
Joined: Wed Aug 08, 2018 3:47 am

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 2 guests