Board index FlightGear Development

[SUGGESTION] Multi-core FlightGear support

FlightGear is opensource, so you can be the developer. In the need for help on anything? We are here to help you.
Forum rules
Core development is discussed on the official FlightGear-Devel development mailing list.

Bugs can be reported in the bug tracker.

[SUGGESTION] Multi-core FlightGear support

Postby Vinura » Thu Mar 15, 2012 1:57 pm

I have heard on these forums from various people that FG does not utilise all the resources available to it very effectively.

Now before I start, I am well aware that FG can run quite fast on a decent computer but there are still little annoyances even when there are still plenty of resources for FG to use.

FG developers have made very GREAT advancements (and I thank them for enhancing FG) in terms of graphics rendering, plane realism, etc but there has been a lack in multi-core support.

I am certainly no developer myself but I understand that it would take a mammoth effort to modify FG's core to better support multi-cores and other resources effectively.

I am putting in a request to see if someone could start on this and improve FG's utilisation of resources more effective.

If someone has already started a topic about this, then by all means, delete this post but otherwise, could someone start on this? Your work would be much appreciated not only by me but also by all the other regular FG users.


Cheers,
Vinny
Vinura
 
Posts: 293
Joined: Sat Feb 19, 2011 2:28 am
Version: GIT
OS: Windows 7

Re: [SUGGESTION] Multi-core FlightGear support

Postby Thorsten » Thu Mar 15, 2012 2:45 pm

The vast majority of the computing load is generated due to rendering, i.e. if you have GPU support (as with any modern graphics card) it doesn't touch the CPU at all. The framerate is usually determined exclusively by the GPU capability - distributing CPU load across several cores may be elegant conceptually, but doesn't make anything run faster.

The graphics card on the other hand has parallel rendering pipelines in the first place.

So there really doesn't seem any point in better multi-core support.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: [SUGGESTION] Multi-core FlightGear support

Postby curt » Thu Mar 15, 2012 3:07 pm

FlightGear is built on top of OSG so actually we do have some fairly sophisticated multicore support. OSG can divide up the work load among your cores in a variety of ways. I believe there is a property you can set to specify how you want OSG to do this. (I'm typing from my tablet and away from my computer today so I can't look it up right now.) In addition, things like paging scenery and fetching weather data are already handled in separate threads.

I think it's a common misconception that flightgear doesn't have good multicore support. Contrast this with how x-plane uses additional cores -- it burns them on running detailed physics for it's AI aircraft. We could spend all day discussion what makes sense to shove into additional cores and how to do it and where the performance bottlenecks are, etc.

Regards,

Curt.
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: [SUGGESTION] Multi-core FlightGear support

Postby Hooray » Thu Mar 15, 2012 3:25 pm

Thorsten wrote:The vast majority of the computing load is generated due to rendering, i.e. if you have GPU support (as with any modern graphics card) it doesn't touch the CPU at all. The framerate is usually determined exclusively by the GPU capability - distributing CPU load across several cores may be elegant conceptually, but doesn't make anything run faster.
So there really doesn't seem any point in better multi-core support.


Actually, what Thorsten wrote isn't entirely correct, i.e. it may be true to a large degree in the context of systems like his local weather system because it is now GPU-limited (due to all his shader work) most of the time, but there are many advantages by moving certain stuff out of the main loop so that it can be run separately.

And FlightGear core developers are VERY aware of that, there have already been hundreds of discussions actually.
You can even find an in-depth discussion in a paper that dates back to 6 years ago: http://wiki.flightgear.org/images/1/1e/ ... ecture.pdf

You can find some links to related discussions here: http://wiki.flightgear.org/Modularizing ... FlightGear (apart from the collection of links, I consider that article outdated however).

Curt wrote:FlightGear is built on top of OSG so actually we do have some fairly sophisticated multicore support. OSG can divide up the work load among your cores in a variety of ways. I believe there is a property you can set to specify how you want OSG to do this.
Yes, there is - but, the OSG threading support is largely specific to offloading rendering-related computations to different threads, it will NOT directly help conventional FlightGear subsystems runing in the main loop, not without explicit coding - i.e. multi-threading or multi-processing and IPC.

Curt wrote:I think it's a common misconception that flightgear doesn't have good multicore support. Contrast this with how x-plane uses additional cores -- it burns them on running detailed physics for it's AI aircraft. We could spend all day discussion what makes sense to shove into additional cores and how to do it and where the performance bottlenecks are, etc.


Actually, when comparing the multi-threading support in X-Plane and FlightGear, it makes sense not to focus on controversial features such as detailed FDM computations for AI aircraft - besides, this has also been suggested for FlightGear by core developers such as David Megginson or Alex Perry, already 10 years ago (and they made some very good points):


At the moment, Durk has already implemented his own "custom" AI FDM logic, exactly like David predicted a decade ago:
David Megginson wrote:You can go through and try to support all of this, but in the end, it
will probably be easier just to create a new instance of an FDM and
autopilot and let them fly the plane. Note that I'm not suggesting
that you do this now -- what you've done already is quite impressive.
However, be careful that you don't end up unintentionally creating
your own new FDM in its stead.


You can probably find 50+ postings by long term contributors suggesting that AI traffic with FDM support would be a good idea: http://wiki.flightgear.org/An_Integrate ... AI_Traffic
So, I am not sure if we should even make that comparison in the first place?

However, it's important to also see how X-Plane is currently MUCH better at providing consistent frame rates and an overall more deterministic run time behavior than FG.

When you start up FlightGear on a 8-12-core machine, you'll find that most cores aren't really utilized at all. This is in stark contrast to X-Plane. There are many computations being run as part of the FG main loop, that don't stricly belong there, and which could obviously run outside the main loop, either in a separate thread or a dedicated process (think HLA).

Just think of all the talks related to moving the AI system out of the main loop: http://wiki.flightgear.org/Decoupling_t ... fic_System

This is something that requires explicit re-architecting, it cannot be just delegated to OSG assuming that it will just magically work. OSG parallelization is specifically related to rendering, it will not directly help you otherwise. Even if you could run 99% of the rendering work asynchronously, that will not have any positive effect, because obviously it's your main loop that's becoming the showstopper here.


Vinura wrote in Thu Mar 15, 2012 1:57 pm:I am certainly no developer myself but I understand that it would take a mammoth effort to modify FG's core to better support multi-cores and other resources effectively.

I am putting in a request to see if someone could start on this and improve FG's utilisation of resources more effective.

If someone has already started a topic about this, then by all means, delete this post but otherwise, could someone start on this? Your work would be much appreciated not only by me but also by all the other regular FG users.


There are a number of things already going on here actually, the most promising work is related to HLA, the idea is to divide parts of the FlightGear architecture such that multiple simulations collaborate and can be run in different threads or even processes. Basically, you would not have a single monolithic FG application with a conventional main loop, but rather a handful of modules (FDM, AI, weather/environment, scripting) that get to communicate with each other by using a central managing component, called a "Real-Time Infrastructure".


FG core developer Mathias Frohlich is working on this (the same guy who started the OSG port!), you can find lots of info by searching the devel archives for "HLA".
There's an introduction in the upcoming newsletter here: http://wiki.flightgear.org/Next_newslet ... pment_news
I tried to summarize things here: http://wiki.flightgear.org/FlightGear_H ... tecture%29

The wiki doesn't mention everything that is planned, there are many more ideas actually:

[Flightgear-devel] FlightGear Manager...
Mathias Frohlich wrote:I can see several components in this area:

  • The viewer. That was the original reason I started fgviewer. I want to have a
    pure viewer application that does not do any simulation.
  • The simulated aircraft. This includes *all* subsystems that belong to this
    particular aircraft. This must be done in a way that you can run several of
    these in a single federation.
  • The weather module. That needs to provide wind and atmosphereic parameters for
    all simulation entities.
  • The AI module. Provides all the AI traffic that is around.
  • A manager. Probably without any gui. Is responsible to give unique startup
    positions and such. Using the unique object names that are owned by exactly
    one federate this will be easy to do.
  • A multiplayer gateway. Currently I am talking about a local federation that
    should help runnning in a paralell environment and for things like the
    LinuxTag booth. Components might need to be coupled with a hard time
    syncronization. At least carefully selected components. Since this would just
    hang the whole federation if one of the federates hangs, this is impossible to
    do for the multiplayer part over the wide internet. Therefore the multiplayer
    server needs to be coupled loosely into the local federation. That is what
    this is for in the longer term.

And many more that make sense to move into its own component.
The above additional AI ogle with a nice, easy to adapt may be python scripted
logic ............

Here we need to start thinking carefully. And this is the reason I ask for
OpenRTI. I have started this project, because I found that setup of the usual
rti libraries is way to complex. So OpenRTI provides one mode where you can
just access a process local federation from multiple threads. There is no
network configuration needed and you do not setup any server in this operation
mode (sure it also provides the usual networking mode). So the plan is to use
this mode as an aid to parallellize flightgear on a local machine. The basic
advantage is that each federate is strictly programmed single threadad. All
the thread syncronization is handled by the rti library and hidden in that
thing. The trick is that each of these threads must be done in a way that you
can just compile that alternatively in a single standalone binary and run the
same component in a networked rti - the LinuxTag booth for example.

Now I start returning to your initial question. From my point of view, such a
gui manager application must be included somehow in the basic set of
functionality that is still running in the threaded mode. You would just
control your simulation with that component. May be restart with a different
aircraft by just shutting down the running federate and start up a new one
with a different fdm.
And as such, doing this core component in python is something I am not sure
about.
So, python in the area of *optional* rti components, is a great tool.
If I remember well, you do have some bigger install at home that might benefit
from such components very much! And in fact this is one reasons I am pushing
this direction.

Python also has one major problem with threads. There is the big interpreter
lock in python which makes python essentially single threaded. While this is
not a problem when such a component runs in its own process, python in core
components that need to run in threaded mode is essentially a no go.

OTOH, I am very curious what creative people will do with that easy python
stuff once we can easily integrate that into a federation in a seperate process
:)

So, yes. There is some work left :)

Greetings

Mathias


As you can see, someone is already working on this - despite some people thinking differently and claiming further parallelization probably wouldn't be worth the hassle. Just keep in mind that the move to OSG also caused lots of regressions, frustration and objections within the community - so, this probably isn'T any different actually.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: [SUGGESTION] Multi-core FlightGear support

Postby HHS » Thu Mar 15, 2012 9:24 pm

curt wrote in Thu Mar 15, 2012 3:07 pm:I think it's a common misconception that flightgear doesn't have good multicore support.... We could spend all day discussion what makes sense to shove into additional cores and how to do it and where the performance bottlenecks are, etc.

Regards,

Curt.


Hmmmm...... :|

I had contact last week with the coordinator of an european research project which uses FlightGear 2.0.0 for visualization. They wish it would be able with FlightGear "to separate the visualisation from the FDM and any other simulation critical components (such as logging). In this way we would have 1 computer dedicated to FDM calculations and logging, whereas the visualisations would run completely independently."

Something which to my knowledge not really much changed with the newer versions, or am I wrong here?
Up, up and away
User avatar
HHS
 
Posts: 3625
Joined: Thu Jul 19, 2007 9:09 am
Version: GIT

Re: [SUGGESTION] Multi-core FlightGear support

Postby Hooray » Thu Mar 15, 2012 10:31 pm

No, you are not wrong. Nothing has changed at all. The opposite is the case, FlightGear's multithreading use is more likely to complicate a "distributed setup" (running a modular FG on multiple different computers) like the one you describe.

And this setup is in fact exactly the way professional simulators work, too: They are distributed and communicate using sockets. Mathias' ongoing HLA work is going to make that increasingly feasible however.

OSG threading is neat and dandy, but obviously it is only directly useful for rendering related parallelization and related work. On the other hand, none of this comes for free. OSG's threading support really is impressive, but all those OSG threads must be kept busy. Properly. That's exactly the work that FlightGear is responsible for in its main loop.

OSG's multithreading support doesn't just come "automagically", just because FlightGear uses OSG.
The multithreading support in OSG must be explicitly supported by following certain OSG coding patterns and coding protocols. Otherwise, you won't benefit at all.

Obviously, OSG really is extremely powerful and also pretty good at parallelizing things, but you cannot necessarily see that when running FlightGear, but there is other OSG-based software which is making really good use of this.

Regarding Curt's comments, you have to take his background in mind ...

Curt isn't just the head of FlightGear development, but he's also the head of the "anti-multithreading movement", too ;-)

Seriously, Curt has been giving soapbox speeches on avoiding multithreading in FlightGear for over a decade. So he is obviously happy to point out that there's plenty of parallelism to be leveraged just by using OSG rather than supporting explicit parallelism in FlightGear, right? :lol:

You can find countless discussions on multithreading FG in the devel archives, and Curt's always been suggesting not to introduce more threading into FlightGear.

He did that for good reasons: correct multithreading is incredibly hard to get right in any non-trivial application.

Especially in complex systems like FlightGear. This holds even more true because FlightGear is implemented in C and C++, none of which has language-level support for concurrency and parallelism.

Basically, there's no language-level support for multithreading in the C-family of languages. Everything needs to be built and emulated using library-functions and classes such as Boost (this is less so in Java).

Still, coding a threaded design in C or C++ with more than 3-5 threads is incredibly tricky to get right (this will also be a problem once more threading will be used in Nasal code, because it also lacks language-level threading support).

There are languages far better suited for multithreaded programming, such as Ada - because they have language-level primitives to create "tasks" and "protected types".

However, what's really been happening in the meantime is that people were politely (more or less) debating the pros & cons of multithreading on the devel list, while silently committing multithreaded code to SimGear and FlightGear...to keep up with the multicore revolution.

Unfortunately, half a decade later, Curt's getting proven right because we are seeing more and more segfaults related to this very multithreaded code in FG. Just look at all the bugs that ThorstenB tracked down recently, most of them are directly related to concurrency issues and multiple threads doing things in a fashion which isn't threadsafe.

Mathias' HLA approach is different and more promising in that a process-based parallelization effort actually forces people to think about the problem at hand. Also, using an existing industry standard (such as i.e. HLA or CIGI) is going to make FlightGear increasingly relevant and attractive to industry leaders.

Normally, a process-based modularization strategy would inevitably mean that modules need to be run as separate processes, but Mathias mentioned a clever scheme to directly run certain "component subsystems" in worker threads, which would then be a part of the main FG process, while still using HLA and a central RTI to handle the communications across all HLA federates.


Seriously though, I would suggest to take everything you can read here with a grain of salt.

The wiki is full with tons of outdated "suggestions" and "proposals", many of them should only be kept for reference because they do point to important mailing lits discussions, but apart from that, they're of little use in my opinion actually.

The only person who really knows more about this is obviously Mathias because he is intimately familiar with OSG and HLA.

So, what I have posted were quotes coming "from the horse's mouth".

Honestly, I am pretty convinced that Mathias HLA work is at least as important and pioneering as David Megginson's original property tree work.

FlightGear will become much more scalable, and much more so than MSFS or X-Plane.

Also, it's obvious that more and more companies are going to be interested in FlightGear once such an architecture is implemented.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: [SUGGESTION] Multi-core FlightGear support

Postby curt » Thu Mar 15, 2012 10:46 pm

I'm not going to be able to match Hooray's verbal effort here. Let me just say that there are some significantly complex issues here surrounding threading, quite a few pitfalls and landmines for the unwary, and a tremendous amount of misconception or over generalization. There is much that can be done currently. If someone has a specific problem or a particular need, we could talk about specific strategies. If Hooray would like to have a shock and awe online flame war -- I'm happy to let him be master of his own domain.
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: [SUGGESTION] Multi-core FlightGear support

Postby Hooray » Thu Mar 15, 2012 11:06 pm

curt wrote in Thu Mar 15, 2012 10:46 pm:I'm not going to be able to match Hooray's verbal effort here.[...] If Hooray would like to have a shock and awe online flame war -- I'm happy to let him be master of his own domain.


You are absolutely misunderstanding. I am not at all interested in flame wars - in fact, you'll have a hard time seeing me participate in such crusades... If you had taken the time to read my response, you would have noticed that I really agree with you.

I'm sorry for obviously having caused an unwarranted response, but that wasn't really my intention. I can perfectly understand if you don't want to respond to the points that I made, because I am sure that both of us understand them pretty well, and there's hardly any point arguing here. Your understanding of the real issue is quite obviously much more comprehensive than the points you made here.

However, you might want to respond to HHS.

It's just too simple to say that "it's a common misconception that flightgear doesn't have good multicore support..", especially given that obviously a number of long-term FlightGear contributors (and core developers) have been pointing this out for over half a decade, some of whom are even currently actively working on fixing this, as I demonstrated by posting the corresponding quotes from the devel list.

Technically, I am not even disagreeing with the points you made, nor with the soapbox speeches that you've been giving on multithreading issues.

You mentioned OSG's threading support, which is fine - but merely doing that fails to address the issue at hand, which isn't strictly related to rendering (i.e. OSG) at all.

None of this is primarily my own opinion, let alone an attempt at having an online flame war - I am seriously sorry that you are perceiving my responses as such actually ...I was pointing out facts as they can be gathered from the mailing list, the wiki and the git logs, most of them made known by people who've been contributing to FG for years.

So, I am certainly not the problem here - at worst, I am just a "symptom" that summarizes the issue.

I am sorry if that should have offended you!
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU


Return to Development

Who is online

Users browsing this forum: No registered users and 8 guests