Paq [Zpaq variant] (ultra) ~30 min compression on 4-5 year i5, 165MB
Arc (level 9 compression, solid) ~noticeably slower, 221MB
Arc (level 4 compression) ~ pretty fast, 237MB
zstandard[zst] (level 19 compression), 262MB
Bzip2 (ultra,900kb dict, 10 passes), 311MB
Brotli (level 9 compression, 315MB
Paq [Zpaq] (normal compression), 325MB
merspieler wrote in Tue Jul 07, 2020 9:00 am:To be honest, I wouldn't want to use any compression, that a normal distro doesn't support out of the box as it would complicate development even more, if you've got to install another thing, just to get the files.
merspieler wrote in Tue Jul 07, 2020 9:00 am:As well, if you require additional software I strongly oppose anything that isn't available in any major distro branch like Debian or Red Hat based systems via their package manager(apt and rpm respectively).
vnts wrote in Tue Jul 07, 2020 9:57 pm:merspieler wrote in Tue Jul 07, 2020 9:00 am:As well, if you require additional software I strongly oppose anything that isn't available in any major distro branch like Debian or Red Hat based systems via their package manager(apt and rpm respectively).
These are all opensource options AIUI, they should have implementations with source code available from their projects for static linking, as well as probably command-line/library implementations on from Linux/MacOS package managers (Probably the lzbench repo contains all the source in one place. And lzbench could probably even be used as a commandline compressor. Peazip is probably available too.). I was only looking for windows binaries as I'm not a programmer professionally and currently don't have a C++ IDE/compiler installed.
vnts wrote in Tue Jul 07, 2020 9:57 pm:Yes, that would be a complication. I thought FG would just use statically linked code, as compressor algorithm implementations would have been tested thoroughly and the only benefit in using libraries provided by the OS distributions may be marginal(?) performance improvements.
These are all opensource options AIUI, they should have implementations with source code available from their projects for static linking, as well as probably command-line/library implementations on from Linux/MacOS package managers (Probably the lzbench repo contains all the source in one place. And lzbench could probably even be used as a commandline compressor. Peazip is probably available too.).
portreekid wrote in Wed Jul 08, 2020 6:26 am:Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.
portreekid wrote in Wed Jul 08, 2020 6:26 am:any fallback for people maybe not wanting to update to 2020.X.
portreekid wrote in Wed Jul 08, 2020 6:26 am:Yes, but as I stated on the mailing-list, this would kill TerraMaster
portreekid wrote in Wed Jul 08, 2020 6:26 am:It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.
vnts wrote in Thu Jul 09, 2020 12:51 am:portreekid wrote in Wed Jul 08, 2020 6:26 am:Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.portreekid wrote in Wed Jul 08, 2020 6:26 am:any fallback for people maybe not wanting to update to 2020.X.
That was a conversation about compressing terrrain data [1] (and a compressor that was faster but not noticeably more effective at compressing)? I was talking about the global OSM2City build, and OSM2City data. Older FG versions should work fine once the new OSM2City data is added to Terrasync.
merspieler wrote in Thu Jul 09, 2020 9:16 am:In my opinion, we shouldn't look too much at backwards compatibility...If you have an old PC, you won't run osm2city anyway....
vnts wrote in Thu Jul 09, 2020 11:21 pm:I was just pointing out possibilities, and even those come with tradeoffs. As I said removal of bloat and unneeded data can often be more effective than trying luck with a content unaware brute-force compression algorithm, with added bonus benefits
vnts wrote in Thu Jul 09, 2020 11:21 pm:That part of the post was about the discussion about compressing terrain
portreekid wrote in Fri Jul 10, 2020 4:57 am:Then I'd advise you to implement this in FG fast as a new LTS release has already been cut. You might get someone to cherry pick it as it is a feature cemented for years.
vnts wrote in Thu Jul 09, 2020 12:51 am:portreekid wrote in Wed Jul 08, 2020 6:26 am:Yes, but as I stated on the mailing-list, this would kill TerraMaster and any fallback for people maybe not wanting to update to 2020.X. Currently we can think about this breaking change, as there is always the option of people using TerraMaster as a stopgap. It currently supports tgz. I'd guess you need something like 30% space reduction to convince them.portreekid wrote in Wed Jul 08, 2020 6:26 am:any fallback for people maybe not wanting to update to 2020.X.
That was a conversation about compressing terrrain data [1] (and a compressor that was faster but not noticeably more effective at compressing)? I was talking about the global OSM2City build, and OSM2City data. Older FG versions should work fine once the new OSM2City data is added to Terrasync.
portreekid wrote in Thu Jul 09, 2020 7:45 pm:Good luck then. The discussion about tgz for terrasync has been running for years. I doubt a second compression for terrasync will be implemented if the first one isn't live yet. Judge the timeframes https://github.com/Portree-Kid/TerraSyncTarTest and https://github.com/Portree-Kid/terramaster/commit/d204910e25d9c797265e42b2bc95753654f1c036
vnts wrote in Thu Jul 09, 2020 11:21 pm:I was just pointing out possibilities, and even those come with tradeoffs. As I said removal of bloat and unneeded data can often be more effective than trying luck with a content unaware brute-force compression algorithm, with added bonus benefits.
portreekid wrote in Fri Jul 10, 2020 4:57 am:Then I'd advise you to implement this in FG fast as a new LTS release has already been cut. You might get someone to cherry pick it as it is a feature cemented for years.
merspieler wrote in Fri Jul 10, 2020 7:23 am:vnts wrote in Thu Jul 09, 2020 11:21 pm:That part of the post was about the discussion about compressing terrain
The terrain is already compressed...
.
Users browsing this forum: No registered users and 3 guests