Scheduling policy of thread.lock(lock) and thread.semup(sem)

[onox]

What is the scheduling policy of these two operations? FIFO, priority, or undefined?

[Hooray]

Unfortunately, most of the FG APIs cannot be considered to be thread safe, i.e. all extension functions - if they seem safe, it's "by accident" not by any conscious effort.

This kind of stuff has never been properly formalized unfortunately - so you're kinda in uncharted waters when playing with threading in FlightGear, which even applies to the C++ code by the way, where threads have also been added "on the go", without ever really documenting the whole thing (which is part of the reason for the increasing number of threading-related segfaults we're seeing):

https://www.mail-archive.com/flightgear ... 26276.html

Even more generally: To what extent is FG thread-safe?
At one extreme, is this merely an aspiration for the
distant future? At the other extreme, is this an
established fact and requirement? Are there perhaps
bits that are known to be thread-safe, and can be
threaded, so long as we stay away from the naughty
bits? Is the answer different for MSVC versus Linux?

This is a fundamental discussion that often separates the home
simulation use and professional simulators. So far concentration has
gone into the first I guess, but the second would not exclude the first
(while the first might exclude the second).

Thus, it makes sense to tell us exactly what it is you're trying to do here, so that we can provide some more specific advice.

Overall, it's nice seeing people playing with Nasal threading, because there are often very good options to offload some work onto worker threads which is a straightforward/simple thing to do, but due to the FlightGear side of things, people need to be aware of a number of issues/challenges, and how to work around them - sometimes,this may involve touching the C++ code.

Also, if your code contains any significant work, it might be better to use OpenCL bindings to use dedicated CL kernels instead (we do have some outdated OpenCL bindings in some FG branch on gitorious, from the pre-canvas/pre-cppbind days) - where all the synchronization would be explicitly handled.

Just saying this because there are already a number of segfaults in FlightGear due to race conditions, i.e. multi-threading issues - core developers are hoping to clean this up by moving to a process-based concurrency strategy with explicit IPC via HLA - but it seems that there's still quite some work ahead, even though we do have several prototypes interacting with the simulator via HLA: HLA in FlightGear

Regarding the Nasal threads module, you'll find the relevant code in $SG_SRC/simgear/nasal:

• https://gitorious.org/fg/simgear/source ... hreadlib.c (the top-level wrapper)
• https://gitorious.org/fg/simgear/source ... ad-win32.c (Win32 implementation)
• https://gitorious.org/fg/simgear/source ... ad-posix.c (POSIX implementation)

[onox]

Is HLA still being worked on then? I was trying to make a protected object by the way, just as an experiment.

[Hooray]

Yeah, HLA is being worked on - but the work is not particularly "visible", because it's mainly infrastructure code that is created/refactored, to see for yourself, check out the OpenRTI project, which is being developed and maintained by a FlightGear core developer (the same guy who handled the OSG port mostly on his own): https://gitorious.org/openrti

As you can see in the commit logs, there's definitely work going on - and the FlightGear side of things is also regularly updated accordingly.

I was trying to make a protected object by the way, just as an experiment.

Should be straightforward to do, but we obviously don't have any explicitly called ctors/dtor, otherwise check out this thread:
Subject: Improving Nasal accessibility

VM Events/Hooks:

• hooks for threading events would be incredibly useful to model higher-level tasking abstractions using Nasal without just offering crude primitives like mutexes or semaphores - for example, in Ada, there's the concept of a "protected type" (monitors), which have implicit synchronization, so that it is safe to access them from multiple threads without manual/explicit serialization - interestingly, that could be implemented as custom precondition/postcondition handlers for hashes that would be called by the VM to handle the serialization.

[onox]

HLA means that I can use languages other than Nasal, correct? Well, I'm looking forward to HLA then

Should be straightforward to do, but we obviously don't have any explicitly called ctors/dtor, otherwise check out this thread:

I don't think it's possible with just plain old pthread_mutex_*() functions, perhaps atomic operations are needed.

[Hooray]

yeah, HLA means that applications will expose an abstract interface so that different languages/processes can more easily be interfaced.
Basically, HLA is about doing RPC in a pretty fancy "message/service bus"-oriented way, where applications can expose different kinds of services and allow other federates to access/use those.

The wiki should contain more info - but on the FG side of things, the focus is primarily better modularization, such that the FG main loop can be cleaned up and non-rendering related stuff moved to separate processes/threads. The hope is to provide better latencies/perfornance, especially in terms of frame rate, and steadiness - i.e. frame latency (time required to create a single frame).

Some of the more obvious candidates for HLA-based modularization would be the AI system, having a weather system - i.e. everything that is fairly "self-contained" in and of itself.

Instead of having a single monolithic code base, and process, FlightGear would be a collection of standalone processes (or threads) that interoperate across HLA/RTI, which basically frees up main loop resources. Eventually, FlightGear would end up looking like professional simulators, where different subsystems run in their own address space - and possibly even on dedicated hardware, or at least CPU cores:

Subject: how can I control two aircraft?

Currently, there is exactly one FDM controlling exactly one aircraft model in one session of FlightGear.

We have a very clever guy working on integrating HLA into FlightGear. Once that is implemented, the idea is already there to be able to virtually walk along a number of parked aircraft on the apron, click on any of them, enter the cockpit and go for a flight. But that's definitely still a long way to go.

Torsten

At some point, this may also involve turning the Nasal scripting interpreter into a HLA component, so that scripting-induced issues (e.g. the GC) would not impact the main loop:

https://www.mail-archive.com/flightgear ... 18063.html

I have a long, long, long term plan to improve multi-threading
support, by enforcing subsystems to *only* communicate via the
property tree, which has light-weight locks thanks to some work by
Mathias. With a dependency graph between subsystems (which I want to
add for other reasons any way) it would then become possible to run
any 'clean' subsystem on a pool of worker threads (maybe just one,
maybe more).

I'm sure there's some other locking that would be required for global
state (eg, the AIManger objects), and of course there's awkward cases
that will never be clean (especially instruments that touch the scene
graph) but it still seems a worthwhile goal. It'd be worth identifying
which subsystems are the big time sinks (FDM? AItraffic?) to
prioritise this.

Another thing that would work well is to proxy all nasal script
invocations to a Nasal helper thread - again this assumes scripts
basically interact with the sim via properties (which they already do)
and that any system functions they call are thread safe - not very
hard to do. As more and more functions get moved to nasal, this might
become a very easy way to balance the CPU usage.

[...]

The 'Nasal on a helper thread' thing may be possible in the short
term, if someone wanted to look at it, but whether or not Nasal is
using a noticeable amount of CPU is highly debatable, and depends a
lot on the particular aircraft / models you test with, I expect.
Again, profiling and concrete numbers are needed before doing
potentially intrusive changes for performance's sake.

In general, FG has quite a few data dependencies internally which make
multi-threading challenging right now - there's groundwork to make the
data-dependencies more explicit (i.e, via the property tree) that has
to happen before pieces can easily move to other threads.

Ensuring that as much rendering work happens on other threads (OSG can
be very aggressive about this) is a likely area of investigation, but
one I'll leave to Tim.

James

Regarding pthreads, you could explore exposing additional primitives - or even port the existing threads module to use cppbind and use SGThread/OpenThreads directly.