osm2city.py development

[merspieler]

Is there a way to only build bridges instead of all roads?

[Johan G]

No, but you can either submit a patch or create a new issue of type "Feature" with a reasoning why this needs to be available (also: bridges in general or road-bridges?)

[vnts]

With regards to analysis on the mailinglist about reducing space usage by changing compression [1]..

Previously [2] I found Bzip2 to be more efficient at compressing building lists than alternatives. I did another search on algorithms potentially better than Bzip2. This time I used the sample tile e135n34 Merspieler linked [3]

I used peazip ( https://www.peazip.org/ ). It's crossplatform, has windows binaries, and has a variety of different algorithms.

Results on 4 folders of e135n34 which is 1.5 GB:

Code: Select all

Paq [Zpaq variant] (ultra) ~30 min compression on 4-5 year i5,          165MB       
Arc (level 9 compression, solid) ~noticeably slower,                    221MB
Arc (level 4 compression) ~ pretty fast,                                237MB
zstandard[zst] (level 19 compression),                                  262MB
Bzip2 (ultra,900kb dict, 10 passes),                                    311MB
Brotli (level 9 compression,                                            315MB
Paq [Zpaq]  (normal compression),                                       325MB


Things to consider if any of these results seem promising: compression level versus compression time tradeoff (see results of Arc or Paq, and faster variants of Paq), CPU time available to FG for decompression, faster implementations of algorithms than peazip used, threading possibilities, etc. I didn't check if different file formats or object types compressed better with certain algorithms (*.ac, *.cables*.ac, *roads*.ac, *.stg, *buildings_shader.txt). Currently file size is dominated by various AC files as objects aren't instanced.

I also came across lzbench ( https://github.com/inikep/lzbench ) which has more algorithms. It doesn't have windows binaries I could find. The main thing useful for FG is the compression ratio results when given the sample file we are using.

When it comes to compression the order of space saving is probably: 1) instancing, 2) using binary files instead of string text (see [here], 3) better compression, 4) eliminating unneeded string text simply by rounding vertex positions to e.g. 0.1m or 0.001m, 5) reduction of data sizes in instanced object lists whether they are binary or not.

1-4 have varying amounts of work needed to implement, and some side benefits like faster rendering, less time decompressing files, less server time compressing files, faster loading from decompressed binary files, less space usage on peoples hard drives, and less bandwidth. 5 might not make too much difference until other options have been applied.

1,2 and 3 could be used together - 4 will also help with compressibility even with compressed binary files as it's turns it into a lossy compression.

Kind regards

[merspieler]

To be honest, I wouldn't want to use any compression, that a normal distro doesn't support out of the box as it would complicate development even more, if you've got to install another thing, just to get the files.
As well, if you require additional software I strongly oppose anything that isn't available in any major distro branch like Debian or Red Hat based systems via their package manager(apt and rpm respectively).

As for compression times, in my experience, even running on a Sata SSD, the IO is still the bottleneck (on an Intel i5 4670K 4 Core, 7 years old).

[vnts]

To be honest, I wouldn't want to use any compression, that a normal distro doesn't support out of the box as it would complicate development even more, if you've got to install another thing, just to get the files.

Yes, that would be a complication. I thought FG would just use statically linked code, as compressor algorithm implementations would have been tested thoroughly and the only benefit in using libraries provided by the OS distributions may be marginal(?) performance improvements.

As well, if you require additional software I strongly oppose anything that isn't available in any major distro branch like Debian or Red Hat based systems via their package manager(apt and rpm respectively).

These are all opensource options AIUI, they should have implementations with source code available from their projects for static linking, as well as probably command-line/library implementations on from Linux/MacOS package managers (Probably the lzbench repo contains all the source in one place. And lzbench could probably even be used as a commandline compressor. Peazip is probably available too.). I was only looking for windows binaries as I'm not a programmer professionally and currently don't have a C++ IDE/compiler installed.

Whether there's enough space savings to make it worth bothering using a different (likely slower) compressor, or if other space savings approaches would make a bigger difference, is another matter.

Kind regards

[portreekid]

As well, if you require additional software I strongly oppose anything that isn't available in any major distro branch like Debian or Red Hat based systems via their package manager(apt and rpm respectively).

These are all opensource options AIUI, they should have implementations with source code available from their projects for static linking, as well as probably command-line/library implementations on from Linux/MacOS package managers (Probably the lzbench repo contains all the source in one place. And lzbench could probably even be used as a commandline compressor. Peazip is probably available too.). I was only looking for windows binaries as I'm not a programmer professionally and currently don't have a C++ IDE/compiler installed.

Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.

[merspieler]

Yes, that would be a complication. I thought FG would just use statically linked code, as compressor algorithm implementations would have been tested thoroughly and the only benefit in using libraries provided by the OS distributions may be marginal(?) performance improvements.

On windows maybe... the benefit of using the system once include things like
* Being able to update them independently
* Don't need to be shiped with fg, which saves space as well...
* Less code maintenance as we would else need to keep our copy of $lib up to date.
* Trust... the packages from the distros are all signed and when you use a distro, you trust it... at least to some degree... other sources miss that trust. (probably not that understandable for a windows user)

These are all opensource options AIUI, they should have implementations with source code available from their projects for static linking, as well as probably command-line/library implementations on from Linux/MacOS package managers (Probably the lzbench repo contains all the source in one place. And lzbench could probably even be used as a commandline compressor. Peazip is probably available too.).

It's not about they being open source or not, it's about being easily accessible... There are already complains in the community about the tooling... Adding another thing that we can't just apt-get install just worsens the situation (again, maybe not that understandable for a windows user).

... and just that you know, both, lzbench and peazip are NOT available on debian (haven't checked rpm).

Anyway, we shouldn't discuss this here but rather on the mailing list... After all, this thread is about the osm2city development.

[vnts]

Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.

any fallback for people maybe not wanting to update to 2020.X.

That was a conversation about compressing terrrain data [1] (and a compressor that was faster but not noticeably more effective at compressing)? I was talking about the global OSM2City build, and OSM2City data. Older FG versions should work fine once the new OSM2City data is added to Terrasync. AIUI OSM2City data will be contained under a different folder. Earlier versions of FG cannot run the new building shader lists - and will probably generate an error or crash(?). OSM2City data will likely be under a new folder downloaded only by compatible future FG builds e.g. OSM2Cityv2. Incompatible versions of FG should work fine as they will download nothing, or download the old folders with outdated OSM2City data formats on Terrasync right now (IIRC Terrasync has outdated OSM2City data format for places like England, Norway, and Alps/LOWI using 3d models in AC files).

Yes, but as I stated on the mailing-list, this would kill TerraMaster

Because Terramaster cannot decompress other formats and place in scenery folders? Is that a problem that's hard to work around? If Terramaster can download the compressed files, maybe FG can be told to decompress first if FG finds data in the format downloaded from Terrasync. It may(?) even be possible to (unconventionally) include 3 binaries for the 3 OSes as external command line decompression tools and select the correct one at runtime - the binaries should be tiny <shrug>.

If Terrasync data is to be compressed, people with less harddrive space (e.g. on laptops, or installing FG on an SSD with the OS), may find an option to keep terrasync data compressed useful - especially if data size goes up due to OSM2City and maybe improved terrain resolutions in WS3.

Compressing terrain data on Terrasync is another topic though..

It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.

I personally can't comment on whether it's worth it for OSM2City data as the issue is complicated as mentioned above (compression time, decompression time, implementation work, exploring different compression levels, download bandwidth, internet speeds, space on peoples SSDs/hard drives etc.). There are also more powerful ways reducing data size, for example by using lists for instanced objects instead of models in AC files for roads/cables.

I simply ran the example tile past some more content unaware compressors, as I suspected some might have better luck with our specific data than others, and reported results. The tar.gz data I downloaded [2] was 345 MB - and results seem to range from 165MB-325MB.

Compressing terrain data in BTG files is a whole different topic. As James said it would make earlier versions of FG in-compatible (unless there was at least one Terrasync source with old data, or maybe if some tools were provided to convert data for old FGs). Even if 2018.x was updated to use the newer format there would be complaints from people who suddenly could not use Terrasync. I recall people using 2016.1 as it was the last release to support 32 bit OSes(?) - not sure if that's changed, or if there have been a newer 32 bit version released. There seem to still be people in the forum with older FG versions under the profile section. People also tend to use older versions of FG on very old systems as newer versions either come with default aircraft that are performance intensive, internal property defaults that are too intensive for extremely old systems, and maybe even as the new versions use WS2 and people have not thought to try WS1 with newer FG. There would probably(?) be a rush of complaints.

For future terrain formats, WS2.0+, it's probably more powerful to avoid unnecessary BTG data in the first place and then try to see if a compressor can reduce size further. For example vertex lists in BTG files seemingly use single precision (32 bit) [3]. As I mentioned before [4], there's no need to use anything more than 16 bits for terrain (even 1 meter resolution is far higher than the accuracy of the source data). It's possible use large blocks of 16bit vector data with a 32bit vector to the start of each block. This will save ~2x the space for vertex lists without needing compression. Additionally rounding off 16bit vertices to a value like 1-10 meters /may/ allow a compression algorithm to save space further as some bits are 0s. A bit reduction like this can be done by post processing using a script on existing data AIUI, but it would break compatibility. (Further space reductions are trickier or more effort, as it involves techniques like using data with custom bit depths, or converting back and forth between coordinate systems so, for example, the local up direction can be quantised with less bits to take advantage of elevation changing slowly..i.e. more advanced solutions).

But compressing terrain data is a different issue maybe useful for future worldscenery.

@Merspeiler I meant about using static libraries for specific problem cases, like with compression where libraries are very mature/tested. The general Linux philosophy of downloading & compiling updated components does have upsides - including being able to compile specifically for the target CPU architecture, something that doesn't happen with closed source code with M$ windows (it works best if whole process optimisation is also possible, and the updated source of libraries are also available to be used part of the main program in cases where inlining functions makes a difference).

Kind regards

[merspieler]

In my opinion, we shouldn't look too much at backwards compatibility. If someone is really interrested in keeping that old version, he should take care of the maintenance of it them selfs. If you have an old PC, you won't run osm2city anyway....

Keep in mind, that modern games often take up no less than 30GB... some even over 50GB... Flightgear is small in comparison (less than 3GB when installed from apt) then it's up to the user, which scenery he/she wants and wants to keep around.

If you ask me, we care too much about old versions which holds us back from what we really could do.

[portreekid]

Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.

any fallback for people maybe not wanting to update to 2020.X.

That was a conversation about compressing terrrain data [1] (and a compressor that was faster but not noticeably more effective at compressing)? I was talking about the global OSM2City build, and OSM2City data. Older FG versions should work fine once the new OSM2City data is added to Terrasync.

Good luck then. The discussion about tgz for terrasync has been running for years. I doubt a second compression for terrasync will be implemented if the first one isn't live yet. Judge the timeframes https://github.com/Portree-Kid/TerraSyncTarTest and https://github.com/Portree-Kid/terramaster/commit/d204910e25d9c797265e42b2bc95753654f1c036

[vnts]

I was just pointing out possibilities, and even those come with tradeoffs. As I said removal of bloat and unneeded data can often be more effective than trying luck with a content unaware brute-force compression algorithm, with added bonus benefits.

In my opinion, we shouldn't look too much at backwards compatibility...If you have an old PC, you won't run osm2city anyway....

That part of the post was about the discussion about compressing terrain (James said there wasn't enough space on mirrors to store the old and new terrain). I wasn't talking about OSM2City. There isn't much OSM2City data on Terrasync now anyway (and maybe it's best to clean up and remove). Future data versioning can just request new folder names without disrupting old versions. OSM2City is sort of an added luxury, so it should(?) be fine to just replace a OSM2City global build with a newer version.

Kind regards

[portreekid]

I was just pointing out possibilities, and even those come with tradeoffs. As I said removal of bloat and unneeded data can often be more effective than trying luck with a content unaware brute-force compression algorithm, with added bonus benefits

Then I'd advise you to implement this in FG fast as a new LTS release has already been cut. You might get someone to cherry pick it as it is a feature cemented for years.

[merspieler]

That part of the post was about the discussion about compressing terrain

The terrain is already compressed...

Then I'd advise you to implement this in FG fast as a new LTS release has already been cut. You might get someone to cherry pick it as it is a feature cemented for years.

I agree... only actions count...

[vnts]

I'm not sure you follow this OSM2City conversation thread exactly, as you came in concerned about a discussion about terrain/airports/models compression

Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.

any fallback for people maybe not wanting to update to 2020.X.

That was a conversation about compressing terrrain data [1] (and a compressor that was faster but not noticeably more effective at compressing)? I was talking about the global OSM2City build, and OSM2City data. Older FG versions should work fine once the new OSM2City data is added to Terrasync.

Good luck then. The discussion about tgz for terrasync has been running for years. I doubt a second compression for terrasync will be implemented if the first one isn't live yet. Judge the timeframes https://github.com/Portree-Kid/TerraSyncTarTest and https://github.com/Portree-Kid/terramaster/commit/d204910e25d9c797265e42b2bc95753654f1c036

As I follow, you are saying a 2nd compression for OSM2City data would take a long time to gain traction

I was just pointing out possibilities, and even those come with tradeoffs. As I said removal of bloat and unneeded data can often be more effective than trying luck with a content unaware brute-force compression algorithm, with added bonus benefits.

By this I meant choosing a different compression was complicated due to various tradeoffs and complications mentioned above. Additionally it may require more research and checking e.g. decompression wouldn't slow FG on slower systems or 2 core laptops. I couldn't say if benefits were worth the costs, as I'm not familiar with all the engineering implications. I'm also not as familiar with the social implications as others: distribution of older/slower systems and slower internet among contributors as well as other things, so I can't weigh that up - I personally have a recent system with access to fast-ish internet & no download limits.

Then I'd advise you to implement this in FG fast as a new LTS release has already been cut. You might get someone to cherry pick it as it is a feature cemented for years.

I was just reporting findings about compression and leaving judging whether it was worth it to others (as I said it wasn't even clear to me that compression was even the most powerful option) - I just pointed it out as the discussion indicated people were looking to save space by looking at compression options for OSM2City (I suspected better compression was possible from earlier investigations).

If you mean removing bloat and unneeded data, then that requires instancing similar to buildings (C++ core support) for things like roads. There's previous context to OSM2City & waiting for core support. I've already talked with vanosten in PM about shaders for instanced roads (that will also open up possibilities of toggle-able roadside objects including fast light sprites when viewed from far away, and even moving cars along concepts I prototyped before). A generic instancing support system should help multiple things (including special vegetation models for crops). Such a system should go a long way towards reducing C++ core support each time something is changed much like the effect system already does - one possible system is one that allows defining the model to be instanced as multiple triangle strips or points, in a simple text file with a custom amount of per vertex data that ends up as vertex attributes.

To be clear, having said that, a lot of people are away or busy due to the situation with the coronavirus, and there are multiple big systems being implemented that will serve FG for 5+ years and which need C++ and GPU code time, like VPB terrain generation&lod&airports&drawing, next gen scenery shader tech, compositor, and the big features that the compositor was created to enable in the first place that are probably worked on/planned behind the scenes. So time for C++ code support and review of GPU code for OSM2City related features will seemingly have to fit in-between that.

That part of the post was about the discussion about compressing terrain

The terrain is already compressed...
.

I see, thanks. It wasn't apparent as the discussion on the mailinglist at that point implied otherwise.

Kind regards

[merspieler]

If you want to play a flightsim on a 2 core laptop, use banks... Forget the low end... You can't play fg seriously on any older none gaming laptop... so why even bother... other flightsims have higher minimum requirements.

If you leave the judgment to others... we deem the efforts not worth.

As for core changes, I think it's best to wait, till the new worldbuild is around since there will be changes and new opportunities.

As we said before, be our guest and do the changes your self... and having no experience is no excuse... we all have learned (or are still learning).