Board index FlightGear Support Graphics

Increase CPU usage to more than 15%

Graphics issues like: bad framerates, weird colors, OpenGL errors etc. Bad graphics ar usually the result of bad graphics cards or drivers.
Forum rules
In order to help you, we need to know a lot of information. Make sure to include answers to at least the following questions in your initial post.

- what OS (Windows Xp/Vista, Mac etc.) are you running?
- what FlightGear version do you use?
- what graphics card do you have?
- does the problem occur with any aircraft, at any airport?
- is there any output printed to the console (black window)?
- copy&paste your commandline (tick the "Show commandline box on the last page of FGRun or the "Others" section on the Mac launcher).
- please upload a screenshot of the problem.

If you experience FlightGear crashes, please report a bug using the issue tracker (can be also used for feature requests).
To run FlightGear on old computers with bad OpenGL support, please take a look at this wiki article. If you are seeing corrupted/broken textures, please see this article.

Note: If you did not get a reponse, even after 7 days, you may want to check out the FlightGear mailing lists to ask your question there.

Re: Increase CPU usage to more than 15%

Postby V12 » Thu Feb 20, 2020 5:40 pm

OK, thx.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: Increase CPU usage to more than 15%

Postby Hooray » Thu Feb 20, 2020 5:53 pm

Speaking in general, Nasal code registered to be invoked via so called "timers" will show up in the performance monitor via the "events" subsystem, whereas listeners will be executed whenever a property is updated/written to,so that there is no single location where Nasal listeners (or rather their callbacks) would show up.

It would not be such a bad idea to have context-specific SGEventMgr instances, so that Nasal code may be invoked in a corresponding context specific to aircraft-events, scenery-events or gui-events - the same setup would also work for timers that are addon-specific, i.e. "addon-events" - and possibly even event manager instances per add-on.

This kind of setup would approximate the functionality of a simple "process monitor", so that people would at least be able to tell in what context/environment Nasal code is adding up. Obviously, the GC is a global thing, so this doesn't help with GC issues - however, the Nasal garbage collector would definitely be one of those things that could/should be added to the built-in on-screen stats, to enable end-users to easily tell if they're running into GC issues or not.

Equally, the number of property reads/writes would make sense to be exposed there, too - as well as the number of listeners/timers executed.

These 3 metrics in combination, should help people spot Nasal related issues much more easily, and they would be comparatively straightforward to add to the OSG on-screen stats, too.
Besides, we have a handful of related patches floating around here on the forum, on the wiki and in the devel list archives.

viewtopic.php?f=6&t=28517
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Increase CPU usage to more than 15%

Postby Richard » Thu Feb 20, 2020 11:44 pm

V12 wrote in Wed Feb 19, 2020 10:36 pm:Where in the Your graphs are Nasal jobs, AI management jobs etc ?

We have Oracle database for realtime production process monitoring with write data rate around 1000 values per second. No problem. I can monitor all subprocesses via SQL queries, make online graphics visualization of material flows etc. With good commit policy You never fall into problems.


All of the rest (apart from rendering is in the section highlighted);
Image

1000 values per second is next to nothing at 120hz - around 8 values per frame; it's trivial.

Fast, lock free, sharing of data between real time threads is a fundamentally hard problem to solve.

There are aircraft (777, A320) that require a lot of Nasal and canvas and these can put a heavier load; but also I think I've shown that it is possible to make a lot of improvements within the aircraft model as the F-15 used to also be heavy because of Nasal and canvas.
Richard
 
Posts: 810
Joined: Sun Nov 02, 2014 11:17 pm
Version: Git
OS: Win10

Re: Increase CPU usage to more than 15%

Postby Hooray » Fri Feb 21, 2020 11:52 am

Richard wrote in Thu Feb 20, 2020 11:44 pm:There are aircraft (777, A320) that require a lot of Nasal and canvas and these can put a heavier load; but also I think I've shown that it is possible to make a lot of improvements within the aircraft model as the F-15 used to also be heavy because of Nasal and canvas.


right, with a corresponding background, Nasal and Canvas can be heavily optimized - but that's not the point: the majority of aircraft developers don't have a 10+ year track record of contributing to FlightGear, and they rarely have a degree in software engineering, computer science or maths/physics.

Which is to say, the execution model for Nasal/Canvas is currently far too low-level, so that providing additional stats may indeed be useful (Nasal GC objets/references/pools, timers, listeners) - unless a better/alternate execution model is implemented, to run certain Nasal/Canvas code outside the main loop.

What has been happening in the Nasal/Canvas department in the last couple of years is reflecting badly upon both, Nasal and Canvas - because we are providing a way to create fancy displays, that simply doesn't scale at all, because people still need a strong background in computing to write code that isn't turning FlightGear into a sluggish piece of software.

For the same reason, it makes sense to provide an option to disable Nasal/Canvas entirely, so that people can tell how much impact their MFDs are really having.

Just saying that all performance issues are due to the current scenery/terrain scheme, and that LOD will make everything better is far too simplistic - to see for yourself, run any complex aircraft (spacecraft) in a location without any scenery/terrain loaded. Likewise, there are still issues completely unrelated to Nasal/Canvas.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Increase CPU usage to more than 15%

Postby Richard » Fri Feb 21, 2020 4:31 pm

Hooray wrote in Fri Feb 21, 2020 11:52 am:...with a corresponding background, Nasal and Canvas can be heavily optimized - but that's not the point: the majority of aircraft developers don't have a 10+ year track record of contributing to FlightGear, and they rarely have a degree in software engineering, computer science or maths/physics.


The point was that these craft that are unusually heavy because of Nasal/Canvas need to have a solution - I didn't mean to imply that all developers would be able to optimise Nasal and Canvas in the way that I've done; in fact I'm hoping to publish some notes to the wiki on how to optimise Nasal/Canvas to minimise the load.

As a separate activity it certainly would be nice to get some of this logic running on another CPU thread - but that's even more complicated than optimising Nasal/Canvas - I don't think that it is something we can hope to do in the core any time soon because of the lack of thread safety.

There are extra debugging aids to alert the developer when modules overrun; again I probably need to write some notes about how to diagnose performance problems.

Hooray wrote in Fri Feb 21, 2020 11:52 am:Just saying that all performance issues are due to the current scenery/terrain scheme, and that LOD will make everything better is far too simplistic


I realise that, but my point was really that until we have LOD the rest of the optimisations aren't as important.
Richard
 
Posts: 810
Joined: Sun Nov 02, 2014 11:17 pm
Version: Git
OS: Win10

Re: Increase CPU usage to more than 15%

Postby Hooray » Fri Feb 21, 2020 4:53 pm

As long as these "debugging aids" are provided in the form of additional properties, these can be trivially used using existing "machinery", such as the built-in logger - i.e. exposing Nasal GC metrics as properties (propertyObject<uint>) would be pretty simple to do, and it would go a long way - the next step would be turning currently hard-coded GC defaults into properties, so that these can be individually tweaked - these are ideas that Andy Ross and Thorsten Brehm discussed several years ago, Andy specifically suggested to allocate Nasal objects into "generations", i.e. keeping the existing mark/sweep collector, but reusing the naPool infrastructure to turn the GC into an incremental/generational one.

In addition, whenever we turn hard-coded defaults into properties, it will also be possible to tell people to tinker with different startup settings, i.e. by overriding defaults via --prop: - for instance, last I checked, Nasal was using 16+ MB of memory, and none of the legacy defaults were suited for that.

Exposing such internals at the property tree level will make it possible for people to look behind the scenes, i.e. by using the property browser, but having such data in the form of properties is also the prerequisite for any kind of benchmarking, and feature scaling - i.e. by hooking up a PID controller to tweak Nasal GC defaults while targeting a specific frame rate/spacing.

For the time being, people/developers working on such *craft, have no good method to tell what's going on behind the scenes - i.e. they cannot t easily know how often certain timers/listeners are registered/triggered, and they don't know anything about GC internals like pool-reallocations/resizing.

Besides, once we have such data available as properties, these can also be hooked up to the existing osg-stats, or alternatively plotted via the Canvas system.

As a matter of fact, we could even register a custom OSG::StatsHandler per Canvas texture and sample data per canvas, to show what's going on there.


Richard wrote:As a separate activity it certainly would be nice to get some of this logic running on another CPU thread - but that's even more complicated than optimising Nasal/Canvas - I don't think that it is something we can hope to do in the core any time soon because of the lack of thread safety.


I agree, I don't think there is a "simple" solution to deal with legacy code - however, providing a new execution model for new Canvas modules where a Canvas texture has a private property tree that can only be updated by a Nasal script that runs outside the main loop would be feasible, and is in line with ideas previously discussed on the developers mailing list - furthermore, that approach is also in line with the way web browsers have come to address the long-standing issue of JavaScript blocking tabs, by coming up with the "web extension" framework, using a message-passing based approach - with one script context running outside the main thread ("background scripts"=, and another one ("content scripts") running inside the main loop communicating only via "events" (messages).

This kind of setup could be made to work by providing a new/alternate Canvas mode, where the Canvas-tree would never show up in the global tree, but instead bound to a private FGNasalSys instance, minus all the global extension functions.

With the exception of nested canvases, i.e. those referencing another canvas via a raster image lookup - those canvas textures could be updated/re-drawn outside the main loop, and would only require a few well-defined synchronization points, i.e. those fetching updated properties/navaid info, and providing the final texture to the main loop, and this is where Emesary could become a real asset.

In and of itself, this won't help with legacy aircraft/code - at least not directly, but it would provide an alternative that people interested in better performance could adopt over time, while investigating how legacy code could be dealt with, so that it can benefit without too much manual work (such as providing a list of subscribed properties, that are automatically copied to the private property tree running in the background context) - this won't be as efficient, but having a list of input/output properties could work well enough for most people's code: viewtopic.php?f=30&t=36614&p=357243&#p357243

Subject: Running Nasal in a separate thread for better performance

Hooray wrote:[...]there is a thing called the global property tree, and that there is a single global scripting interpreter. The bottleneck when it comes to Nasal and Canvas is unnecessary, because the property tree merely serves as an encapsulation mechanism, i.e. strictly speaking, we're abusing the FlightGear property tree to use listeners that are mapped to events, which in turn are mapped to lower-level OSG/OpenGL calls - which is to say, this bottleneck would not exist, if a different property tree instance were used (per Canvas (texture)).

This, in turn, is easy to change - because during the creation of each canvas, the global property tree _root is set, which could also be a private tree instead.

Quite literally, this means changing 5 lines of C++ code to use an instance-specific SGPropertyNode_ptr instead of the global one.

At that point, you have a canvas that is inaccessible from the main thread (which sounds dumb, but once you think about it, that's exactly the point).
So, the next step is to provide this canvas instance with a way to access its property tree, which boils down to adding a FGNasalSys instance to each Canvas - that way, each canvas texture would get its own instance of SGPropertyNode + FGNasalSys

Anybody who's ever done any avionics coding will quickly realize that you still need a way to fetch properties from the main loop (think /fdm, /position, /orientation) but that's really easy to do using the existing infrastructure, you could really use any of the existing I/O protocols (think Torsten's ajax stuff), and you'd end up with Nasal/Canvas running outside the main loop.

The final step is obviously making the updated texture available to the main loop, but other than that, it's much easier to fix up the current infrastructure than fixing up all the legacy code ...

[...]
telling the canvas system to use another property tree (SGPropertyNode instance) is really straightforward - but at that point, it's no longer accessible to the rest of the sim.
You can easily try it for yourself, and just add a "text" element to that private canvas.
The interesting part is making that show up again (i.e. via placements).
Once you are able to tell a placement to use such a private property tree, you can use synchronize access by using a separate thread for each canvas texture (property tree).
But again, it would be a static property tree until you provide /some/ access to it - so that it can be modified at runtime, and given what we have already, hooking up FGNasalSys is the most convenient method. But all of the canvas bindings/APIs we have already would need to be reviewed to get rid of the hard-coded assumption that there is only a single canvas tree in use.

Like you said, changing fgfs to operate on a hidden/private property tree is the easy part, interacting with that property tree is the interesting part.

Also, it would be a very different way of coding, we would need to use some kind of dedicated scheduling mechanism, or such background threads might "busy wait" unnecessarily.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Increase CPU usage to more than 15%

Postby V12 » Fri Feb 21, 2020 5:53 pm

Richard wrote in Thu Feb 20, 2020 11:44 pm:Fast, lock free, sharing of data between real time threads is a fundamentally hard problem to solve.

Many industrial software systems this problem solved. Only database can solve high speed multiclient shared access. Many years ago I workedon simulation of product lines for waffles, it was on 486DX66 with almost 1000 technological values per second (temperatures, conveyors speeds, fan speeds etc. on poor DOS DBF database. Database never made bottleneck. Modern well designed databases have not any trouble with parallel write, nor parallel read. But this new system need total rewrite of FG core...
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: Increase CPU usage to more than 15%

Postby Hooray » Fri Feb 21, 2020 6:47 pm

wow, dBase ... that must have been the early 90s then - brings back old memories ;-)

Anyway, while those anecdotes are indeed interesting - they don't have much bearing in the context of this discussion.

When it comes to FlightGear, the challenge is identifying the real bottlenecks - i.e. with and without graphics enabled, with and without scenery/terrain and Nasal/Canvas enabled - and then we'd ideally use a test suite with pre-recorded flights / flight plans to see how different aircraft perform in terms of frame rate, frame spacing, memory/VRAM utilization.

Tim Moore previously suggested that something like this could be done using FlightGear itself, i.e. by running a "headless" version of FlightGear in a scripted fashion, loading different combinations of aircraft/airports (scenery) while executing certain flight plans or pre-recorded flights.

There is support for a built-in profiler, the system/performance monitor, the built-on OSG stats - those data points can then be used to compare different combinations of FlightGear settings to see which subsystems/settings are the obvious candidates when it comes to optimizing FlightGear, which may include threading out some portions, but which may also include other optimizations that have little to do with threading, e.g. memoization/caching - which is something that more sophisticated Nasal frameworks are using to "cache" properties/values and avoid unnecessary re-reads.

The whole database approach may admittedly sound a bit far-fetched, even though recent versions of FSX do indeed use a distributed approach - and I believe Richard mentioned elsewhere that he's also got a professional background in distributed simulation, where it is also not that uncommon to have the equivalent of distributed systems that provide different data to various parts of the simulation.

As a matter of fact, the terrain/scenery system could already be said to be the equivalent of a database, and that also applies to the so called "navigation database" - these could indeed be implemented as "service providers" running in their own threads/processes - and technically, that would also solve a number of long-standing issue when it comes to setups that involve inter-linked fgfs instance that work in a master/slave setup or other multi-instance setups (FSWeekend/LinuxTag) or multiplayer setups that might need a central resource provider, no matter if that means navigational data, terrain/scenery data or environmental data.

Finally, let's keep in mind that the property tree is basically "just" a database - it can be queried, and values can be put into - and it can even be accessed from different processes via the props/telnet protocol. And based on Erik's original idea of having "remote property trees", it could also be distributed, at the cost of implementing support for "mounting" remote property trees inside a global tree, and registering other trees in a private tree to mirror read/write updates.


Under the hood, the property tree isn't much more than a simple LDAP-like key/value database.
So, there's that, too - even though, the approach would still look very different to dBase-style 90s code ;-)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Increase CPU usage to more than 15%

Postby V12 » Fri Feb 21, 2020 8:17 pm

Hooray wrote in Fri Feb 21, 2020 6:47 pm:Under the hood, the property tree isn't much more than a simple LDAP-like key/value database.
So, there's that, too - even though, the approach would still look very different to dBase-style 90s code ;-)


Property tree is not like LDAP like database, because all components runs in one loop and there is problem with memory sharing between more threads.
And yes, You are right, it is very different to dBase. But idea is same :
- db server as independent process
- main loop reading input devices, clicks etc. other independent process writing needed data into db tables, ofcourse not dBase :mrgreen:
- fdm another independent process reads needed data (aircraft position and orientation, control surface position, input device position etc) and writes into db new computed aircraft position, orientation etc.
.
.
. etc
- Weather manager
- AI manager
- renderer - again independent process reads all needed data about view orientation, aircraft position, scenery geometry etc, assemble scenery and send data to the GPU
All that processes are fully independent "database clients", doesn't share memory etc. and therefore they can run on any CPU cores. With this system You can't get stuck on db locks, because there is not concurency write into one table and one record and one field.

It is only idea.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: Increase CPU usage to more than 15%

Postby Hooray » Fri Feb 21, 2020 8:26 pm

You're missing quite a few showstoppers, such as synchronization between threads/processes - also, there is no such thing as a "parallel write" without also doing check-pointing, i.e. serializing such writes to ensure state is valid at all times. The issue isn't trivial, even though it may seem obvious when designing something from scratch - but it's a completely different matter once you are talking about an existing legacy code base, and re-architecting that according to your ideas. Basically, you are talking about a complete rewrite, possibly using dBase and BASIC ? :D

Again, it might be a good idea to read up on what's been previously discussed in the scope of FlightGear + better use of multi-core hardware, to get a more comprehensive picture of the various issues involved here:

http://wiki.flightgear.org/Modularizing ... FlightGear
http://wiki.flightgear.org/Multi-Thread ... FlightGear
http://wiki.flightgear.org/Property_threading
http://wiki.flightgear.org/Multicore_Pr ... d_Clusters
http://wiki.flightgear.org/Distributed_ ... FlightGear
http://wiki.flightgear.org/Distributed_ ... Simulation
http://wiki.flightgear.org/High-Level_Architecture
http://wiki.flightgear.org/FlightGear_C ... _Interface)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Increase CPU usage to more than 15%

Postby Hooray » Sat Jun 13, 2020 1:00 pm

Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Previous

Return to Graphics

Who is online

Users browsing this forum: No registered users and 3 guests