Board index FlightGear Development Aircraft

Slowdown in the loading .ac files

Questions and discussion about creating aircraft. Flight dynamics, 3d models, cockpits, systems, animation, textures.

Slowdown in the loading .ac files

Postby abassign » Thu Jan 19, 2017 8:07 pm

I made with Blender .ac a sphere with 63K vertices, one with 253K vertices and a third sphere with 1051K vertices.
The first was quickly loaded, the second il loaded in about 4 minutes, the third is loaded in about 60 minutes!

A few months ago I did the same test and the loading times were absolutely acceptable, only the third ball took a couple of minutes, the load times seemed linear with the size of the files.

I noticed the problem because, developing a plane, I realized that, after updating the version of the FGFs to 2016.4 and then to 2017.1.0, the model of charging times seem to follow an exponential logic with the size and not more linear as it was before. During this time I have not changed anything in my operating system and drive NVIDIA 367.57 (Linux Mint 18) I7200 quad-core machine with 32GB of RAM. This problem has been lamented by another developer with a Windows 7 machine.

Someone knows something? tests have been done, by others, of .ac file size greater than 5-6 MB?
My impression is that the problem can be partially solved, breaking the .ac files into smaller segments, but unfortunately this is not always possible.

Do these tests is not difficult, I used as a base plane UFO, which I replaced the ufo.ac file with the ball. Moreover, once the file is loaded, the execution times are absolutely acceptable. The sphere with vertices from 1051K, with maximum definition and quality and active ALS, 9-16 fps.

This is the test with 1051K vertices sphere, the speed is 12-16 fps:

Image
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Slowdown in the loading .ac files

Postby wlbragg » Thu Jan 19, 2017 10:13 pm

The shuttle has in excess of 700k and doesn't take more than a couple minutes on a slower core- duo system with a mid-range GPU + two 4096 texture sheets using bleeding edge fgfs and data.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Slowdown in the loading .ac files

Postby abassign » Fri Jan 20, 2017 12:32 am

wlbragg wrote in Thu Jan 19, 2017 10:13 pm:The shuttle has in excess of 700k and doesn't take more than a couple minutes on a slower core- duo system with a mid-range GPU + two 4096 texture sheets using bleeding edge fgfs and data.


Absolutely true, if you do a simple sphere with a large number of polygons, the loading file .ac will be very slow?
Before it was not so, I made this test a couple of years ago, and everything worked pretty well.
Can you do this test ? Is a test that is done quickly, creating a sphere with Blender and then using the modifier "Subdivision surface" several times until you get a high number of polygons, then exports in .ac format
... I would be curious to figure out if it's a problem we have only in Italy or not ... ;)
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Slowdown in the loading .ac files

Postby wlbragg » Fri Jan 20, 2017 5:59 am

I killed the loading at the 5 minute mark with a 491,000 sphere. That was longer than I wanted to wait. So I don't think it's an Italian thing. :)
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7588
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Slowdown in the loading .ac files

Postby Thorsten » Fri Jan 20, 2017 7:19 am

It's likely again some OSG mesh optimizer which is run over the raw mesh that's loaded (remember I never saw this for the Shuttle for some reason)?

The optimizer doesn't process vertex by vertex, it needs to compare one vertex with the nearby ones, so that's at least a task that goes like n^2 in the number of vertices, hence the non-linear scaling of loading times.

And of course if the mesh is 'good' to begin with, you don't get any framerate improvement for your trouble. It seems to be the kind of idea that's good for a small 3d mesh, but it just doesn't fly for larger models.

@abassign: May I suggest you present your findings on the devel list? This really is a problem, and I suspect we need to get rid of the optimizer.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: Slowdown in the loading .ac files

Postby abassign » Fri Jan 20, 2017 8:38 am

Thorsten wrote in Fri Jan 20, 2017 7:19 am:@abassign: May I suggest you present your findings on the devel list? This really is a problem, and I suspect we need to get rid of the optimizer.


Observ that the slowing effect is quadratic with the number of vertices, I agree that the problem is the presence of some optimizer enabled in the latest versions of OSG. However, before writing, I want to do some tests, because, by analyzing the .ac files you've done for the shuttle, I have the impression that the problem is reduced if you reduce the size of the individual subparts. Then, when I have these data, I write a note in devel list.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Slowdown in the loading .ac files

Postby abassign » Sun Jan 22, 2017 10:41 am

Thorsten wrote in Fri Jan 20, 2017 7:19 am:It's likely again some OSG mesh optimizer which is run over the raw mesh that's loaded (remember I never saw this for the Shuttle for some reason)?
The optimizer doesn't process vertex by vertex, it needs to compare one vertex with the nearby ones, so that's at least a task that goes like n^2 in the number of vertices, hence the non-linear scaling of loading times.


I did some checks, the .ac file contains subparts which are processed individually. The extraction program definitely performs an analysis process that has a quadratic cost than on Module size. I found in my .ac modules that contain 5-6000 vertices, not many seemed to me at the rendering level, but this number becomes devastating to an algorithm that operates with a quadratic load on the CPU. At this point, when I removed these modules everything is back in place. The loading time has been reduced by about 10 times!
It is obvious that those who designed the did the software to very different types of objects, small and fractionated, for which a quadratic algorithm is still quite effective.

Since I'm working on a high-definition model, this fact is a problem, as it is common to work with objects composed by thousands of polygons. A GPU level, once the module is loaded, as I showed you before, there are no problems (The magnificent Shuttle amply demonstrates that fact), current GPUs (when they have a sufficient amount of RAM) very well run hundreds of thousands of polygons.

So now there is a clear separation performance of SW used in FGFS. On one side of the loader file .ac it remained tied to an old logic modules with smaller about a thousand polygons. While the rendering engine, based on the GPU, has made enormous progress.

There can be various strategies:

1. Delete the optimizer present in OSG
2. Move the optimization process directly into the GPU
3. Limit the use optimizer only for smaller modules of a certain size.
4. Check if the problem is related to an algorithmic defect of the optimization algorithm.

Now my curiosity is to know: "How do I report all this and to whom exactly?"
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Slowdown in the loading .ac files

Postby zakalawe » Tue Jan 24, 2017 5:14 pm

This is quite an interesting result, please do share it on the developer list. I am worried that part of the cost is the BVH construction which we cannot avoid of course, it's necessary for collisions and picking support.
zakalawe
 
Posts: 1259
Joined: Sat Jul 19, 2008 5:48 pm
Location: Edinburgh, Scotland
Callsign: G-ZKLW
Version: next
OS: Mac

Re: Slowdown in the loading .ac files

Postby Thorsten » Tue Jan 24, 2017 6:55 pm

I am worried that part of the cost is the BVH construction which we cannot avoid of course, it's necessary for collisions and picking support.


But we had that in all along, whereas this dramatic slowdown has appeared only fairly recently according to abassign.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: Slowdown in the loading .ac files

Postby PINTO » Tue Jan 24, 2017 6:57 pm

I tried this with a 500k vertex non-sphere (looked like a brain sort of) and I gave up after 35 minutes of load time.
Actively developing the MiG-21bis (github repo) (forum thread) (dev discord) (fg wiki)

http://opredflag.com is an active flightgear dogfighting community (using a system that isn’t bombable)
User avatar
PINTO
 
Posts: 966
Joined: Wed Oct 21, 2015 7:28 pm
Callsign: pinto
Version: stable
OS: Win10

Re: Slowdown in the loading .ac files

Postby abassign » Wed Jan 25, 2017 8:16 am

In the documentation of OSG, you can find this reference:
http://trac.openscenegraph.org/projects/osg/wiki/Support/UserGuides/OptimizerOptions

Searching on Google, I saw that the problem was also discussed in November for the Shuttle:
https://forum.flightgear.org/viewtopic.php?f=4&t=25747&hilit=cockpit&start=1560

Unfortunately the automatic optimizations can lead to "side effects" that are not always acceptable.
The optimization feature is certainly useful if the .ac file was not already optimized, but it could be redundant if it already is. My question is this:
You can delete or change this options selectively while loading .ac files?
I thought you might enter in the xml file an option to set the flag optimizer to activate according to the needs of those developed.

These are the flag that can be activated for the optimizer:
Image
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Slowdown in the loading .ac files

Postby bugman » Wed Jan 25, 2017 8:33 am

Abassign, if you discuss this on the FlightGear development mailing list, you will be able to talk directly to the people who have made these optimisation changes. Such discussions are what that mailing list is for. Otherwise your problem WILL take much longer to solve.

Regards,
Edward
bugman
Moderator
 
Posts: 1808
Joined: Thu Mar 19, 2015 10:01 am
Version: next

Re: Slowdown in the loading .ac files

Postby abassign » Wed Jan 25, 2017 9:38 am

I sent an excerpt of the discussion in the developers forum.

I hope the problem will be solved in the best way.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Slowdown in the loading .ac files

Postby Hooray » Sat Mar 11, 2017 12:37 pm

Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU


Return to Aircraft

Who is online

Users browsing this forum: No registered users and 13 guests