Board index FlightGear Development Scenery

osm2city.py development

Questions and discussion about enhancing and populating the FlightGear world.

Re: osm2city.py development

Postby montagdude » Thu Mar 12, 2020 5:16 pm

merspieler wrote in Wed Mar 11, 2020 9:51 pm:I've reported the vanishing buildings as well as the flickering months ago... seems to be forgotten.... (like so many thing in fg) btw... if you upgrade to get rid of these bugs... don't go with amd... i've just got an amd card which work nicely but the bugs are there as well (not sure if it works on envidia tho)
I was able to test it on a machine with an Nvidia card, and there was no flickering, and nothing disappeared when I turned up the model shader. Maybe this is just an issue with older video cards not having some GL extensions. On my system, it says the OpenGL version is 3.0.
montagdude
 
Posts: 154
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Thu Mar 12, 2020 7:04 pm

My GPU is from 2017... not that old....
I've got opengl 4.5... so unlikely that it's the version.
The new GPU got rid of the flickering tho... but the buildings still disappear.
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 585
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby stuart » Thu Mar 12, 2020 9:28 pm

merspieler wrote in Wed Mar 11, 2020 9:51 pm:I've reported the vanishing buildings as well as the flickering months ago... seems to be forgotten.... (like so many thing in fg) btw... if you upgrade to get rid of these bugs... don't go with amd... i've just got an amd card which work nicely but the bugs are there as well (not sure if it works on envidia tho)


Not completely forgotten. :)

I think I know how to fix it, but it requires re-writing the shaders and FG code to use fewer varying parameters by packaging some of the texture coordinates together. To do that I need a clear head and some clear time.

-Stuart
G-MWLX
User avatar
stuart
Moderator
 
Posts: 1544
Joined: Wed Nov 29, 2006 9:56 am
Location: Edinburgh
Callsign: G-MWLX

Re: osm2city.py development

Postby vnts » Sat Mar 14, 2020 1:11 pm

The way the shader buildings are done is pretty powerful in terms of potential iteration. :mrgreen:

stuart wrote in Thu Mar 12, 2020 9:28 pm:it requires re-writing the shaders and FG code to use fewer varying parameters by packaging some of the texture coordinates together

Do you mean using fewer vertex attributes? The model fragment shader handled buildings when I had a brief look at shader buildings right after shader buildings were added last year.

For whenever you have space to tackle this problem (no need to feel you have to look at the problem now):

The general way of packing multiple values of fewer bits into a float or int:

Edited: a wrong think on the shader part, but the idea is to get bits out by multiply or dividing and using floor or fract
Code: Select all
Formula for values v1, v2, v3 with 8 bits, packed into v with 32 bits

C++ side:
v = (v1 << (8*0)) + (v2 << (8*1)) + (v3 << (8*2));

Shader:
v1 = 256*fract(v/256); v=(v-v1)/256; v2 = 256*fract(v/256); v=(v-v2)/256; v3 = 256*fract(v/256);

IIRC reading (when I tinkered a little with FG shaders) this trick will work to pack even 4 8-bit values into floats and unpack in shaders.

Maybe this trick could work to reduce the space taken by trees in RAM and any pauses when loading trees into VRAM.

it requires re-writing the shaders


(From some experiments on the old shader buildings)

It might be simpler to move the texture coordinates and calculating texture gain (tex units per meter) to shader side at least for ALS. That would leave the per building data.
A future iteration of building shaders to take more advantage of procedural capability, within the next couple of years or whenever, would change the texture sheet format anyway.

This would have the advantage of buildings loading up quicker with minimal c++ processing. There might be less frame spacing issues when 2km(?) blocks of buildings are loaded to VRAM as buildings use less space.

The same formula can be implemented in the vertex shader.

The building properties could maybe all use 8 bits: Rotation (<2 degree accuracy). Length/width/height/etc with 255m range and 1 meter sensitivity, or 127m range and 0.5 meter sensitivity. If OSM2CITY output has more decimal palces it could be rounded off on c++ side for now.

The building position data should use 16 bit ints in building lists to reduce space. 2km/65535 = 0.030m = 3 cm.

Data sent to shaders could be packed to reduce 3 vertex attributes to 2. If packing 2 16 bit values per float works, or 3 8 bit values per float are used and rejoined into 2 floats.

As I see it, the smallest and simplest amount of vertex attributes would be: a). A face number - 8 bit, b). the number of the vertex in the face, c). the rest of the packed properties.

a) and b) can be packed into 1 float. Having b) seperate early allows easy texture coords. In a future iteration, it'll allow knowing where in a quad you are to decide if there's going to be a door, window, decoration, or skylight there or if a part of a wall is lit by lights.

In a future iteration the texture selection and texcoord calculation code is likely to be in the fragment shader, as only the fragment shader knows where in a wall or roof the fragment is to calculate wall, lighting, or object there. Textures can be randomly selected from a row of texture variants. Currently vertex shader is faster(?) unless trying to reduce varyings.

It's possible to use an array of vec2 and simply look up the texture coords for each corner of a quad based on an index made from a) and b). I found this was fast during geometry shader experiments earlier.

In a far future (!) iteration, it might(?) be convenient to load the model verticies and any info idenifying face and corner numbers as vertex attribute data. This should allow new shaders and texture sheets to be created by tweaking existing ones without modifying the core, for use with different or more complex building models in different parts of the world. The c++ side would just load the list of verticies and attributes from the appropriate model and add the building properties as vertex attributes from the OSM2City building list data. The appropriate model could be defined in regional definitions, or maybe chosen based on additional info from the OSM2City building list.

From the other thread:
merspieler wrote in Thu Feb 13, 2020 6:42 pm:A quick update:
I've just started to provide the files in the way they will be included into terrasync.
...
And the some stats on the scenery so far (same as on the ml):
Total disk usage: 168.6 GiB
Apparent size: 140.2 GiB
Items: 1892988

Size distribution:
Roads 96,6GiB
Pylons 58.4GiB
Buildings 11.6GiB
Details 2.1GiB

Interesting :) That's not too much as compressed files.

Wondering hat happens when they're downloaded. From the Australia demo scenery, 156 MB download uncompressed to 778 MB. Roads was the biggest at 445 MB. Will FG uncompress OSM2City data, or will it keep files compressed, maybe slowing loading of data a lot?

There's a lot of room for further compression and if necessary. Building lists are stored as strings? .ac files are text too. A binary file format conversion might help. There may be a compressed file format that specialises in dealing with numbers and can reduce to binary ints. Loading speed would be reduced too. It will perhaps be most noticeable the most on systems with fewer spare cores like old systems or laptops.

.ac files seem to use floats, 16 bit ints are more than good enough (3 cm accuracy on 2 km blocks). Reducing size is lossy compression. Even with good file compression data should be at least somewhat smaller.

It may be that current compression and loading speed are good enough :)

Kind regards
Last edited by vnts on Sat Mar 14, 2020 7:40 pm, edited 1 time in total.
vnts
 
Posts: 152
Joined: Thu Apr 02, 2015 12:29 am

Re: osm2city.py development

Postby stuart » Sat Mar 14, 2020 4:34 pm

Hi vnts,

Yes, I meant vertex attributes - my brain was slightly dead when I wrote that.

Thanks very much for the code snippet - that's exactly what I had in mind, but I hadn't started coding it. And also for the ideas to reduce occupancy.

On the question of compression, FG will uncompress the files so that load time is not impacted on future runs. I had thought that if we are compressing the building lists there would not be much additional benefit in having a binary format? Particularly as most of the data is with very limited set of characters.?

-Stuart



I will do so think evening.
G-MWLX
User avatar
stuart
Moderator
 
Posts: 1544
Joined: Wed Nov 29, 2006 9:56 am
Location: Edinburgh
Callsign: G-MWLX

Re: osm2city.py development

Postby vnts » Sat Mar 14, 2020 9:21 pm

That wasn't a code snippet (not a programmer professionally) as much as the idea from what I recall. I've fixed the shader part to have more of a chance of working : P
stuart wrote in Sat Mar 14, 2020 4:34 pm:I had thought that if we are compressing the building lists there would not be much additional benefit in having a binary format? Particularly as most of the data is with very limited set of characters.?

Reduction in bits is lossy and might make a difference to building lists (16 bit positions data and 8 bit property data). Binary formats would also be faster to load. The string processing code is possibly (?) getting a workout currently. Not sure how effective the current compression method is compared to calculating the size as a binary format: adding up the binary size by counting entries, and assuming a 8-16 bit table is used for text strings. Binary data will be compressed too.

Currently as it stands the building lists don't take up space relative to roads or pylons. That might change later.

Whether further size optimisation is worth it depends on whether current space requirements an issue. I'm not nearly familiar enough with FG, and the types of hardware or bandwidth people have. So far I noticed people do seem to use FG on laptops and old systems (smaller HDDs).

Road data can be made much smaller by instancing road segments similar to buildings like I mentioned here (link. If not, an alternative is to a) reduce the floating point .ac data to 16 bits or 8 bits, b) convert text strings to binary - maybe harder than instancing.

I haven't really looked at pylons. The pylon data appears in .stg. If there's a .btg that may help. If this is not done already(??), it maybe possible to specify lines of pylons: with start and end points. The number of pylons in between or the spacing could be a property. C++ side would then have to spend CPU time populating model positions and maybe query terrain elevation to place models.

There may be a sizeable space saving in terrain too, if terrain data is currently stored as 32 bit floats (??). Terrain geometry data won't be very accurate at all. 16 bit data can represent a 65km sphere with 1m accuracy. That's likely much more accurate than needed. I also recall reading on the wiki that the .btg format stores terrain coordinates as double precision+single precision offset (?). Dealing with smaller blocks of data would then allow 16 bit terrain geometry data with 1m+ accuracy. Rounding off is equivalent to snapping to a grid. It shouldn't create holes in terrain, and might patch existing holes. The double & single precision offset could be stored as a integers too I guess. Converting to integers should work as a post processing stage after everything else. Not sure how this compares to just compressing the data instead.

If this double & single precision offsets are available as uniforms they can be used as stable IDs for generating random properties for all objects - even trees e.g. with slightly different leaf colours.

Kind regards
vnts
 
Posts: 152
Joined: Thu Apr 02, 2015 12:29 am

Re: osm2city.py development

Postby vanosten » Sun Mar 15, 2020 5:48 am

If you are considering binary data,then please also mind the testing part of it. At least during a period (or as a start-up parameter text files should be possible in parallel.
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 457
Joined: Sat Sep 25, 2010 5:38 pm
Location: Denmark - but I am Swiss
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby vnts » Wed Mar 18, 2020 1:01 pm

If binary formats are worth it for performance or client/server space the data in AC files might be worth attention. Since AC files are text, the loading speed penalty is probably similar to building lists and stg files.

In Merspeilers OSMCity for Sydney region the uncompressed AC files are:

Buildings: 73.5 MB out of 69.3 MB
Pylons: 191.1 MB out of 192.4 MB
Roads: 445.5 MB out of 445.7 MB
Details: 63.7 MB out of 66.7 MB
Total: 769.7 MB out of 788.4 MB

- OSM2City building coverage isn't really good in this area, or it's missing/not included.
- AC files in pylons seem to be cabling between pylons. Cable model vertices seem to have a lot more significant figures in text.

Ways of avoiding AC files :) :

AC files in roads can be sidestepped by instancing.

Maybe AC files in pylons (cables) can be sidestepped by listing lines of pylons by specifying start/end/spacing. FG would have to populate pylons and cabling. The fastest way to do cables between two pylons might be a 2d rectangle that is rotated by the shader on it's axis to sort of face the camera, or would that not work? A bit like how billboards rotate completely to face the camera. If sag due to gravity in cables between pylons is to be simulated in the shader, the rectangle would need more vertices.

Details are cables, piers, and platforms. Cables take up a lot of space, maybe a cable list file might help. Not familiar enough to say what AC files can be sidestepped.

AC files in OSM2City buildings are detailed building models that are harder to sidestep. If there are office buildings that are approximated by 2 or 3 buildings together, maybe multiple rectangular buildings can be used to make a L or U shape. Not sure how much space would be saved, or the rendering speed up because of more buildings being instanced.

To make an L or U instanced building scheme work with future iterations all buildings would need a). a 16 bit ID so L or U building components get the same ID b). a manual way of specifying which of the 4 sides of buildings faced streets and if that side was a primary or secondary entrance-shopfront. Example of b) with a 4 2bit number: 0 building side with no entrance, 1 small side entrance, 2 secondary entrance/back exit, 4 primary entrance. b) is generally useful for houses and buildings, and could be complemented by 4-bit distance from street from 0-15 meters for lighting purposes.

Google doesn't seem to indicate there's a binary version of AC files(?), so no existing loading or conversion code.

As for how much space is saved by 8-16 bit integer binary files compared to text files and compressed text files, here is /quick/ a look at shader buildings example:

Shader building list file entry format:

- X/Y/Z: 16 bits *3
- Rotation/H/W/D/Pitch height: 8 bits *5
- Building size/Roof shape/roof orientation/wall tex/roof tex/num floors: 8 bits or less*6

- Total: 18 bytes: 17 bytes per entry (136 bits)+End of row entry byte?.

Example file:
Buildings\e150s40\e151s34\e150s40_e151s34_5426688_buildings_shader.txt.

7440 entries (lines) , 953.4 KB

Binary size (uncompressed): 133.9 KB (7440*18 bytes).
BZip2 dict 900kb: 205.7 KB
7z LZMA2 ultra: 227.0 KB
Zip deflate 64 ultra: 254.0 KB
GZip: 258.4 KB
Rar best: 265.4 KB

An uncompressed 8&16 bit precision binary gives a 48% size reduction on GZip. Roughly half. Other text files with lots of numbers might show similar benefit.

I suspect the lossy nature of the bit reduction helps a lot here. Compressing the binary format should give a further space saving, but not sure how much.

vanosten wrote in Sun Mar 15, 2020 5:48 am:If you are considering binary data,then please also mind the testing part of it. At least during a period (or as a start-up parameter text files should be possible in parallel.

Something to also consider is versioning the data. A future iteration might benefit from more information. What type of information depends on what shader building features are worth the spending the limited GPU budget for buildings on. That would likely be known later in the process. For example it might require someone like Thorsten to think of features for buildings in different parts of the world, and do a cost benefit analysis, or maybe it simply becomes clearer after implementing things. An example of more information: walls could be lit slightly at night from street lamps and traffic as they are not completely dark. If the distance to the next building was known there could be a bit of ambient occlusion included at daytime. If there's any information on whether the building is an old/historical one, or if the building was a log cabin, or in a small town far away from a city, different textures could be chosen by the shader as defined by the regional definitions/texture sheet. I guess extra or unneeded information doesn't need to be used, although it will increase file sizes slightly in binary form.

@stuart: Something I forgot to mention with regards to reducing VRAM/attributes:
The building basement doesn't need it's own set of quads if texture coordinates are assigned or tweaked in the fragment-shader. Just interpreting all z values below the ground as basement is enough to assign new tex coords. It is also possible to save 3 vertex attributes by not specifying vertex position attributes in model space. In my old experiments I used a building version of the model ALS ultra fragment-shader.

It's enough to specify face number of each quad & corner number of each vertex as vertex attributes. The vertex coords can be stored shader side in an array of vec3s representing vertices. An array of ivec4s can contain the index number for each vertex at the 4 corners of each quad. It's then a simple matter of using the face & corner number to look up the index number of the vertex for each corner. Normals can be derived in shader saving 3 attributes. The shader can do a cross product of two quad vectors, or look up a table. If the face and corner number isn't available in explicit form, then it has to be reverse engineered from the tex coords and vertex positions in a future iteration that uses a custom texture sheet (when I did some experiments I just assumed a front face for all walls as the face was convoluted to derive from tex coords and vertex positions).

Kind regards
vnts
 
Posts: 152
Joined: Thu Apr 02, 2015 12:29 am

Re: osm2city.py development

Postby montagdude » Tue Mar 24, 2020 1:11 am

Just wanted to share a couple osm2city scenery images. They are from Richmond, VA, which is within an area I'm working on now (also with custom terrain shapefiles). I think they look really nice. The only issues are:
  1. It's hard to keep the roads from going in and out of the terrain. I set POINTS_ON_LINE_DISTANCE_MAX = 100 and MAX_SLOPE_ROAD = 0.15. I don't think I want to add any more points to the roads, because they are already taking up the most space of anything in the scenery. Any other way to fix that?
  2. Some buildings are on top of each other in the dense areas. Is that because of the underlying OSM data, or maybe because buildings in the default scenery are being shown too?
Anyway, thanks again to all those working on this great tool. It really helps make FlightGear look great.
Image

Image
montagdude
 
Posts: 154
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby merspieler » Tue Mar 24, 2020 3:52 am

1: Decrease POINTS_ON_LINE_DISTANCE_MAX ... that's it
2: There are some more complex shaped buildings which are build by using more than one generic building. What it makes seem wierd is the fact that they don't share the same texture thus make them look like two different buildings.
Love at first flight A<380
Checkout Autopush. An improvment to the pushback to make your life easier.
Attempting an osm2city worldbuild... Coming soon to YOUR terrasync!
merspieler
 
Posts: 585
Joined: Thu Oct 26, 2017 10:43 am
Location: Wish to be in YBCS
Callsign: JST935. ORI1711
IRC name: merspieler
Version: next
OS: Debian Bullseye

Re: osm2city.py development

Postby montagdude » Tue Mar 24, 2020 12:20 pm

1) Okay, well I think I will have to live with the roads as is, because I don't really want them taking up more space. Also POINTS_ON_LINE_DISTANCE_MAX = 50 didn't seem to make much difference in a test area I tried. I'm speaking from ignorance regarding how the road creation algorithm works, but I wonder why the roads can't just actually follow the terrain? It seems backwards that there would be a max distance between points instead of a min distance. If the terrain is perfectly flat for 100 miles and the road is straight, it should only need two points in that distance, but if it's hilly and curvy, it would need more.

2) If they are the same building, shouldn't they use the same texture or split the textures so there is no overlap? When there are two textures on top of each other, it causes z-fighting.
montagdude
 
Posts: 154
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby montagdude » Fri Mar 27, 2020 12:40 pm

Sorry about all the questions, but here is another issue I'm facing. Very long bridges fail to be generated successfully. For example, here is a screenshot showing what is supposed to be the bridge crossing the Chesapeake Bay in Maryland. It starts but then gets cut off. It seems to happen for all bridges close to this length or longer.

Image
montagdude
 
Posts: 154
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby vanosten » Sat Mar 28, 2020 6:28 am

Please consider creating issues with detailed information:
* What is it you are seeing (e.g. picture) and what is wrong? If you suspect conflicts with non osm2city scenery, then please make a picture of the same spot with osm2city turned off.
* Coordinates
* OSM IDs

And please really do look into the OSM. Especially for buildings they can be tricky to understand and they are modeled inconsistently across the world. The problem with the bridge could be that it crosses a tile boundary - IDK.

I am aware of the fact that I have not been very active lately. The more information I get, the higher the chances that I will look into it or who knows, maybe someone else).
Last edited by Johan G on Sat Mar 28, 2020 11:16 am, edited 1 time in total.
Reason: Please do not quote the entire preceeding post, it is right above your's
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 457
Joined: Sat Sep 25, 2010 5:38 pm
Location: Denmark - but I am Swiss
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

Re: osm2city.py development

Postby montagdude » Sat Mar 28, 2020 11:24 am

Hey vanosten,

The bridge in the picture is part of a multilinestring labeled "US 301 (MD)," osm_id 2297412. The coordinates of the middle of the bridge are -76.3664, 38.9893. A picture is attached showing it on a map. I don't think this is due to a conflict with non-osm2city scenery. This is just one example though; there are several long bridges in the Maryland - Virginia region that did not get generated.

Image
montagdude
 
Posts: 154
Joined: Tue Dec 31, 2019 6:04 am

Re: osm2city.py development

Postby vanosten » Mon Mar 30, 2020 5:56 pm

I checked. The reason is that one end of the bridge is in the water according to FG scenery. So osm2city needs to assume that it would look odd and therefore does not place a bridge. Once we get a world scenery with cost lines much more aligned with OSM, it will work. Until then: bad luck or place a custom scenery object
Maintaining osm2city. Contributing with ground attack stuff to the OPRF FlightGear military-simulation community.
vanosten
 
Posts: 457
Joined: Sat Sep 25, 2010 5:38 pm
Location: Denmark - but I am Swiss
Callsign: HB-VANO
Version: latest
OS: Win 10 and Ubuntu

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 2 guests