Microsoft Flight Simulator, a small graphics study

[icecode]

However zero of the core developers consider that a CPU bottleneck is firstly a "problem"

I consider it a problem. Why people sometimes try to pass opinions as facts just because X person said it?

the benchmarking that has been done does not point to this

What did you benchmark? Systems, Nasal, rendering, all of them? How did you benchmark it? Where are the results? I would like to inform myself so I can change my opinion on the subject.

[legoboyvdlp]

Hooray has done a lot of benchmarking work along these lines.

While that's fine and helpful, does that not indicate that Hooray's system isn't CPU limited? Also, is that relevant anymore - as best I remember his system can't run recent versions of FlightGear -- are those statistics still relevant to new versions?

For instance, I would be very surprised if the IDG-A32X's low performance wasn't to do with inefficiency in IO to the property tree - that's on the CPU (although admittedly that's partly our fault

[bugman]

I have GPU and not CPU limited systems. What I'm trying to say is that we actually need some proper and detailed benchmarking. I.e. just like the MS flight sim development team, or any commercial development team, will be constantly doing as they develop. This is needed to make the statement that multi-threading will solve issues. I realise Hooray's system is/was old. But in 3 years looking at subsystems and considering parallelisation, I haven't seen any benchmarking to provide a convincing argument that parallelising subsystems L, M, and N will solve CPU bottleneck X, Y or Z, verses GPU bottleneck A, B, or C. This is important because maybe only 1% or less of the code would theoretically benefit from parallelisation (the practicality of slow data messaging between threads/nodes/etc. would make this even less). The only person looking at such bottlenecks in detail, with benchmarking, is Richard, and he is looking at real solutions (note - multi-threading is not on the radar).

Detailed benchmarking with reproducible and very specific conditions is needed to see where in the codebase any benefit could be potentially extracted. Otherwise I know that I could put in 2-3 years of full-time effort to make the property tree thread safe (or rewrite it be an isolated and robust subsystem, using HLA or other messaging interfaces for communication with the rest of the codebase), to make all subsystems robust enough to work in isolation (an essential step if they were to be parallelised), to convert the last parts of FG core into subsystems (there are some major rendering components that James has still not converted into subsystems, e.g. SGSky and, importantly, FGRenderer), and to maximally parallelise the subsystems. And I'm pretty sure that the end result of my hard work will have zero frame rate improvements from now, but rather more likely cause a decrease in frame rates due to the cost of passing data between the parallelised components (and the CPU cost of making the subsystems robust enough for this). I have a lot of practical experience with the disappointments of theoretical vs. practical parallelisation. I have lost a solid year of my life on the issue!

@legoboy: In coding, the algorithm implemented is almost always more important than the speed of the underlying language. So optimisations of specific Nasal code have orders of magnitude greater benefit than saying that we need to activate threaded Nasal. I'm guessing that will be the case with the IDG-A32X and I guess you guys already know the Nasal IO and property tree pitfalls.

Regards,
Edward

[icecode]

I have GPU and not CPU limited systems.

Unless you are running FG on an integrated GPU, it's very unlikely that you are GPU limited. If you are, I don't think your system is a good system to benchmark on as FG is an intensive 3D application, not a web browser.

What I'm trying to say is that we actually need some proper and detailed benchmarking.

I agree, that would be the first thing I would do if I were to start changing things around, but I'm rather busy with other stuff. I'm just trying to paint a simple picture of what the problems are with rendering in FlightGear, since I consider myself involved enough with the project both at a user-level and at a developer-level. End-users don't usually have the knowledge of FG internals to describe their problems with detail, so they just complain about "low fps".

I haven't seen any benchmarking to provide a convincing argument that parallelising subsystems L, M, and N will solve CPU bottleneck X, Y or Z, verses GPU bottleneck A, B, or C.

I've never said that parallelising subsystems as they are would bring any performance improvements, and you are right it probably wouldn't, but then again I've never even mentioned that approach to multi-threading in FlightGear. What you are describing is an implementation approach, and it may or may not work. Same thing can be said for the approach I proposed earlier in the thread. The best way to find out, as you said, is to run benchmarks and compare them. Until then I don't think it's wise to completely discard multi-threading and its possible perks.

[bugman]

No one is permanently discarding multi-threading. But note that, on the other hand, no one is demonstrating a specific CPU bottleneck that would be best eliminated via multi-threading (and which cannot be fixed via some other mechanism, e.g. a better algorithm). So there is no point implementing any multi-threading right now.

Making the property tree thread safe will have a CPU cost. Making the subsystems robust for parallelisation will have a CPU cost. Parallelisation itself will have a significant data transfer cost that is currently not present. And threading has a real human cost, being probably an order of magnitude harder to code, debug, and maintain. Hence there is no reason to consider changing the core in this way, for now.

Regards,
Edward

[icecode]

no one is demonstrating a specific CPU bottleneck that would be best eliminated via multi-threading

Because in this whole thread I've only been talking about high-level architectural changes to FlightGear, not about specific places where you can optimize with multi-threading.

Hence there is no reason to consider changing the core in this way, for now.

Your opinion, not mine. And unless I see some (proper) benchmarks to prove otherwise, I'd leave this as a purely speculative discussion, not as facts that the reader must believe.

[bugman]

Multi-threading is not a "high-level architecture". It is also one of many different parallelisation techniques. It is just a tool that is used to target a very specific spot or sequence in the code. Multi-threading is not a magic bullet. You can't just make FG "multi-threaded" and then take full advantage of X cores running asynchronously from the rendering. Let's look at the specific example of the FlightGear architecture and discuss different levels of granularity:

1) Groups of subsystems. This is the highest level of granularity and such specific groups are being considered for HLA parallelisation.
2) Individual subsystems. This would be the next level of granularity. No one is considering this for any parallelisation tools.
3) Threading within a single subsystem. This would be the simplest, lowest level of granularity and the most likely to result in improvements. If carefully designed, the FlightGear main loop and property tree do not need to be modified to be thread-safe.

If you implement one of the above, you cannot simply switch to another level of granularity without a complete rewrite. Especially when benchmarking shows that your complex implementation leads to performance degradations (which is a common occurrence when the problem is simply better executed serially rather than in parallel). So you cannot implement granularity level X and then say that FlightGear is prepared for the future of multi-core systems. Unless of course you belong to a marketing team

Regards,
Edward

[icecode]

Multi-threading is not a "high-level architecture". It is also one of many different parallelisation techniques.

I didn't mean high-level architecture as HLA (didn't notice they shared the same name). I meant the high-level structure of FlightGear, literally.

Multi-threading is not a magic bullet.

I literally used that same expression in one of my previous posts. You don't need to convince me of that.

Let's look at the specific example of the FlightGear architecture and discuss different levels of granularity:

That architecture is implicitly flawed for multi-threading because it hasn't been thought out from the beginning to support threading. Yes, it would take a huge rewrite to change this. I didn't say it was a nearly impossible task for nothing.

[curt]

An off topic post and several related follow up comments were moved to the trash bin. Please think about how your posts contributes to moving the discussion forward and please be kind to each other. Please carry on with the on-topic aspects of this discussion!

[www2]

A good sub system that is a good candidate for multi-threading is the sound system.
Other candidates are Weather system and AI system and run there own Nasal code in a container.
As a for the communication between threads for my view is a socket like approach (maybe i talk about HLA interface).

For blocking code: the nasal function findNavaidsWithinRange have some blocking problems.
This function hangs flightgear for about 2 minutes if i set the range to 3000nm.

[bugman]

1) The sound system is already multi-threaded due to using OpenAL, just see Erik's mailing list post:

Re: [Flightgear-devel] Designing a thread-safe property tree API (was Re: A FGPythonSys implementation: ...)

2) Thorsten's weather system is the one that may possibly benefit, but that is written in Nasal code rather than C++, so converting it to be multi-threaded would require a complete rewrite of how Nasal is embedded within FG (to enable Nasal's inbuilt threading), followed by a complete redesign of AW to use it. But from what Thorsten has mentioned previously, the problem sounds like it is naturally non-parallelisable. I.e. it is not suitable for breaking into small chunks (it has a granularity of 1).

Such a system might benefit from the subsystem robustness I've been looking at. Imagine spinning it out so that the subsystem can run stand-alone on a server somewhere, and multiple machines then connect to it via HLA so that they all see the same weather and cloud formations. I think Thorsten has previously mentioned that such ideas would, however, require a completely rewrite of the AW routines to shift it from a local to non-local algorithm. Hence this will require some volunteer to invest a huge quantity of time to properly design and implement.

3) Durk has already made the AI system very robust. It can run stand-alone. But from what Richard has mentioned, I have a feeling that the loading of AI models into your local OSG scene graph is the biggest bottleneck. As far as I am aware, this is not a problem that multi-threading can solve. Different tools are needed for that (I'm not sure if Richard is still working on that one).

4) Blocking code vs. inefficient code. Using findNavaidsWithinRange with 3000 nm is inefficient. We are talking about multi-threading in this unlocked thread so the term "blocking" should be reserved for the thread concept of locks Inefficient code is not alleviated by multi-threading - that just partly hides the problem. You fix inefficient code by changing the underlying algorithm. The Navaids are in an SQL database - specifically for fast lookups - so you need to design a fast algorithm to take advantage of the SQL database fast lookups.

Regards,
Edward

[bugman]

I just saw an interesting commit by Richard about the Nasal garbage collection issue I mentioned earlier:

Commit: 92a3c8

Added Nasal garbage collection background thread

This uses an SGThreadExclusive controlled by Emesary notifications that are received from the main loop.

When active at the end of a frame the garbage collection thread will be released; if it is already running this will do nothing. Optionally at the start of the mainloop we can wait for the previous GC to finish.

The actions of the background GC is controlled by notifications - again received from the main loop which in turn uses properties.

I initially thought that the wait at the start of the frame would be necessary; however in 100 or so hours of flight without the await for completion at the start of frame no threading problems (or any other problems) were shown; so nasal-gc-threaded-wait is defaulted to false which gives a slight boost in performance.

So what this does is to it removes the GC pause of 10-20ms every 4 seconds (test using the F-15). This change doesn't really give much extra performance per frame because normally GC is only performed when needed.

Regards,
Edward


P. S. If you search the simgear and flightgear code base for "SGThread", you will see that we actually do have some quite specifically targeted multi-threading already present in FG. But, as I previously hinted, you only put this stuff where it makes sense.

[www2]

1)
4) Blocking code vs. inefficient code. Using findNavaidsWithinRange with 3000 nm is inefficient. We are talking about multi-threading in this unlocked thread so the term "blocking" should be reserved for the thread concept of locks Inefficient code is not alleviated by multi-threading - that just partly hides the problem. You fix inefficient code by changing the underlying algorithm. The Navaids are in an SQL database - specifically for fast lookups - so you need to design a fast algorithm to take advantage of the SQL database fast lookups.

Regards,
Edward

I know that 3000nm is overkill I use only as a benchmark for showing this problem.

[Richard]

As has been said I've been doing a lot of research work on multi-threading and how to improve performance.

There are really two things that are important, firstly (and most importantly to me) is to avoid frame pauses[1], secondly the overall frame rate. Recent changes to the model loader (DDS texture cache) have reduced frame pauses but there are still things to improve. These two things are also referred to a system load and latency - and are equally important.

For research purposes I managed to get the rendering to run in a separate thread to the main simulation loop- and this gives a small increase in performance - effectively the bit at the end of the OSGStats histogram - on the F-15 this is around 5-7ms[2].

I've spent a lot of time optimising the F-15 over the last few years in general and the last few months in particular - both the 3d and also the Nasal / Canvas to get to the 5-7ms figure and not all aircraft are this well optimised - but there's not a lot we can do about that in the core.

What Fernando is saying makes a lot of sense to me - that we should be looking to find a way to get more performance on the rendering side as that's what generally limits the frame rate - on my system I did once manage to get a 90% GPU load and a 75% CPU load for FG by enabling 20 OSG Database threads but generally both the CPU (single core) and GPU are running at close to 90% on my system with OSM and high random vegetation.

What Thorsten says also makes a lot of sense - as to take advantage of any rendering improvements generally requires artistry and contributions. Maybe we should be thinking of someway to restart a coordinated scenery development project as there are definitely contributors that are interested in working on scenery. Scott has been doing good work making Terragear stable, but it might be that we should also investigate using Virtual Planet Builder as that is something that Robert has said he'll be working on for VSG (but I'm not a scenery expert so I don't understand the complexities in doing this)

Again I think it is generally understood that the lack of LOD in the terrain mesh is contributing to overall system load - and good progress was made on fixing that with the nextgen scenery project but work on that is currently stalled.

What I think is the best long term solution is to split the rendering into a separate process and keep rest the same - as long as an aircraft model can consume no more than 6.2ms per frame (peaking at 8.3ms) we can have at 120hz simulation core that can then feed the rendering process possibly using CIGI[3]. Now it maybe that I'm biased towards this approach because it's what the big sims have been doing since the advent of digital image generation.

The only way I can see of truly making use of lots of cores is switching to Vulkan because OpenGL is essentially single threaded. I've been following the VSG (Vulkan SceneGraph) development lists and progress is being made and this is likely the long term direction that we will need to take; and it might be that this is the time to separate out the rendering.

In the short to medium term we can only improve what we've got, firstly with compositor, possibly with nextgen scenery but multithreading isn't going to play a major part unless we can separate out the rendering so we'll be using 2.5 cores.

---------------------------------------------------------------------------
[1] a frame pause is where the system stops rendering for a few frames
[2] this figure can vary with extra system load such as AI Traffic, AW, MP
[3] there is a good opensource CIGI implementation reference http://cigi.sourceforge.net/

[V12]

Some about frame pause - When I start FG for example at LOWI with manual METAR without clouds, my machine can do 30 fps without significant amout of frame pause. But after 1 hour flight in same weather, after land at LOWI I have 13-15 fps and many frame pauses. With MSFSX, fps at take off fromLOWI is 20 fps, on landing after 2 hours long flight again at LOWI with same weather is 20 fps, there is almost none frame pauses.
I can accept framerate as low as 10 fps, but without frame pauses.
Or another example - I have 45 and more fps, but there are still visible and annoying frame pauses. Something is wrong in the rendering pipeline. IMHO, FGFS needs multithreaded core, rendering, nasal, weather.