Board index FlightGear Development Canvas

Multi-threaded FlightGear/OSG and Canvas flickering solved

Canvas is FlightGear's new fully scriptable 2D drawing system that will allow you to easily create new instruments, HUDs and even GUI dialogs and custom GUI widgets, without having to write C++ code and without having to rebuild FlightGear.

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby merspieler » Thu Sep 02, 2021 11:23 pm

crap... you're right, the dnc script didn't pull the latest commit... (I've overlooked the -d flag)

Works a bit... but screens stay black on the cloned view or are with a noise pattern (DCDU)
Image

Also the Upper ECAM and DCDU popup misbehave video

And while trying to test all popups, I've got this crash:
Code: Select all
Thread 102 "fgfs" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe36fed700 (LWP 1823838)]
0x000055558d916800 in ?? ()
(gdb) bt full
#0  0x000055558d916800 in ?? ()
No symbol table info available.
#1  0x00007ffff7461a81 in osg::GraphicsContext::runOperations() () from ../../openscenegraph/lib/libosg.so.162
No symbol table info available.
#2  0x00007ffff74e124d in osg::OperationThread::run() () from ../../openscenegraph/lib/libosg.so.162
No symbol table info available.
#3  0x00007ffff7463a7a in non-virtual thunk to osg::GraphicsThread::run() () from ../../openscenegraph/lib/libosg.so.162
No symbol table info available.
#4  0x00007ffff72ad32f in OpenThreads::ThreadPrivateActions::StartThread(void*) () from ../../openscenegraph/lib/libOpenThreads.so.21
No symbol table info available.
#5  0x00007ffff682aea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140729821091584, 3928925698729510639, 140737488339870, 140737488339871, 140729821088704, 8396800, -3928769558794664209, -3928911158673209617}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0,
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#6  0x00007ffff4593def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.


Next round, I've checked all popups and the rest works... but I've managed to trigger this by resizing the popup:
Image

Also note, that the issue with the popups happens also without added clone view.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2242
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby Hooray » Fri Sep 03, 2021 7:06 am

As far as I can tell, those issues are unlikely to be in any way related to the recent Canvas related changes or to the original Canvas.Path issue.
If in doubt, try to come up with a reproducible test case and see if others can confirm that they can reliably reproduce the issue, too.

Right now, I don't see anything specific to the Canvas system, or rather, something that could be linked back to the recent changes or the original issue.
It's best to reach out to others to determine if they can reproduce the issue or not.

Speaking in general, you seem to have been tinkering with a fair number of local changes/experiments and OpenGL related tools/debuggers - all of which are more likely to cause/contribute to this than the recent changes are.

Obviously, I might be wrong, so let's wait for SurferTim and others to report back whether they're seeing any regressions or not - but what you've reported doesn't make any sense to me in the given context. If in doubt, you could obviously also share your findings on the devel list/issue tracker to see if there are people seeing similar issues.

I would also suggest to share your exact git hashes (SG+FG+FGDATA and try a clean rebuild first) to ensure that we're talking about the same sources (there's a convenient button to copy/paste all relevant info to the clipboard that I once added):

https://wiki.flightgear.org/About_dialog
Image

PS: In that case, it does help sharing a BT with debug symbols included :wink:
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby merspieler » Sun Sep 12, 2021 9:12 pm

Took the time to do a bisect:
Code: Select all
83b0a31b765348116c079b585ee997b1b5c9d355 is the first bad commit
commit 83b0a31b765348116c079b585ee997b1b5c9d355
Author: Fernando García Liñán <fernandogarcialinan@gmail.com>
Date:   Fri Aug 27 18:28:16 2021 +0200

    Canvas cameras are slave cameras instead of scene graph cameras
   
    This was a long-standing issue where Canvas cameras were rendered
    multiple times, once per slave camera. In the case of the Classic
    pipeline they were being rendered twice.
   
    Making them slave cameras has the issue of them not being shared
    across GraphicsContexts anymore, so it's not possible to see Canvas
    displays in extra CameraGroup windows. This is fixed by using OSG
    context sharing feature and sharing the initial windows' context
    with all subsequent windows.

 src/Viewer/WindowBuilder.cxx |  5 +++++
 src/Viewer/renderer.cxx      | 18 ++++++++++++++++--
 2 files changed, 21 insertions(+), 2 deletions(-)


That's when the first error appears (unable to open the UECAM properly).

FG: 83b0a31b765348116c079b585ee997b1b5c9d355
SG: a97e148a9467ee33335e5c55c5c0daa48596f40d
FGDATA: 8d85476441501596db6947ddcd21fb3ac4b07a52

Also tested with commits from around the faulty fg one with the same result.

There's no backtrace, cause it doesn't crash...
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2242
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby Hooray » Mon Sep 13, 2021 5:58 pm

That's unlikely to be related to the original issue reported here (and, thus, to this thread).
If you can reliably reproduce the issue and if it isn't specific to any particular aircraft/setup, this would be something better discussed on the devel list/issue tracker respectively.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby merspieler » Mon Sep 13, 2021 6:10 pm

As said before, I can reproduce this 100% of the time... I don't have anyone else confirming it cause apparently no one wants to help (asked on mattermost, matrix, irc and ml)

I've reported the issue already on the ml and for the issue tracker... I don't feel taken serious there either... some of my issues are on Needed info but no comment on what's needed, others are stale without any info at all... and some don't even have a responce...

Also it got worse between next HEAD and the breaking commit above... I didn't bother to track the exact commit...
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2242
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby Hooray » Mon Sep 13, 2021 6:56 pm

well, I understand - but based on what I've seen/heard so far, reproducing the issue would be the key to determine if there is an actual issue or if it's specific to your particular setup/machine/GPU/OS or even just to a number of custom OpenGL related enhancements that you're using, without fully understanding their implications, or rather, their inner workings.

For instance, what you have been reporting so far, could easily be related to OpenGL hooking, but also to a build tree that contains binaries, that are not fully in sync with the latest git sources, which is why it's important to see if others are seeing the same issue or not - because then it's obviously not specific to your box.

Besides, the git bisect is nice, but also wasn't needed: the Canvas viewer cams are the only recent changes to the Canvas system, so checking if that has an impact or not, is the lowest hanging fruit.

For now, I would suggest to try different aircraft or different canvas related features, and get some of your teammates involved to see if they can reproduce the issue or not - at that point, you'll at least know if you are dealing with a local issue or an actual issue that can be reproduced by others - at which point, it will be much easier to get more attention, too :D

If you have multiple OS installed, dual booting into another OS would be another option to ensure that your issue is not specific to your main OS.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby merspieler » Mon Sep 13, 2021 11:02 pm

For instance, what you have been reporting so far, could easily be related to OpenGL hooking

No... as I said, at least half a dozend times, I disable it for those tests.

or if it's specific to your particular setup/machine/GPU/OS

Then how comes, that on older versions it works? Can you reproduce it?

If you have multiple OS installed, dual booting into another OS would be another option to ensure that your issue is not specific to your main OS.


I don't... I've got another PC around, tho with only an iGPU, on which I could install anything... which distro should I check? Debian, like my main? or rather something a bit different like Ubuntu?
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2242
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby eatdirt » Thu Oct 07, 2021 9:03 pm

I can confirm on Mageia that, with FG 2020.4.0, aircraft being the SpaceShuttle, and

Code: Select all
--prop:/sim/rendering/multithreading-mode=AutomaticSelection


as soon as you try to open a canvas window, the ones associated with the Shuttle, they are empty. If you open them a few times, you get a segfault.

Running without the above-mentioned option is fine.

I've tried to increase stacksize with ulimit -s and OMP_STACKSIZE, but no change.


PS: I also confirm that running 2018.3.* (without compositor), with AutomaticSelection, runs fine.

cheers,
chris.
eatdirt
 
Posts: 1012
Joined: Wed Aug 15, 2018 3:06 pm

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby SurferTim » Fri Nov 19, 2021 12:53 pm

Just stopping by to let you know about the status of my FG install. It's working great! Thanks to all who helped.

One unexpected benefit was the crashing during aircraft launch has stopped.
Previously, once in about every three times I selected "Fly", the aircraft splash screen would display for a couple seconds, then crash back to the Ubuntu desktop.
It hasn't done that since the patch.
Just FYI.

Edit: The launch crashing had nothing to do with the canvas panel AFAIK. This crashing was worse on the Twin Otter.
Fly low. Fly slow. Land on a dime. Twin Otter. https://github.com/SurferTim/dhc6p
My other aircraft is a Citation-X https://github.com/SurferTim/CitationX
PirateAir videos at https://www.youtube.com/user/SurferTim850
User avatar
SurferTim
 
Posts: 1711
Joined: Sun Dec 09, 2018 6:49 pm
Location: Miramar Beach, FL
Callsign: Pirate
Version: 2020.4.0
OS: Ubuntu 18.04

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby merspieler » Fri Nov 19, 2021 4:54 pm

I get that crashing too... what did you exactly change?
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2242
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby Hooray » Fri Nov 19, 2021 4:56 pm

Note that this is actually not the proper fix, what's happened instead is that the recent change to use OSG context sharing, also will implicitly serialize OSG threading - in other words, for the time being, FlightGear's use of OSG threading is somewhat crippled, i.e. it might not make much of a difference what threading mode you are selecting, under the hood it will implicitly fall back to shared contexts/single-threaded mode

You can probably tell for yourself if that's what's happening by watching how many threads FG is using, and monitoring/comparing the performance you're getting
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby SurferTim » Fri Nov 19, 2021 5:39 pm

The patch is gone! What happened here? There was a two-line addition to a file in SimGear, and now I can't find it in this thread.

Hooray! It was Hooray! It was in my other thread. Here is what I did.

Here's how to test the patch by manually editing a single file in simgear:

go to $SG_SRC/canvas/elements/CanvasPath.cxx
navigate to/search the line containing static const VGubyte shCoordsPerCommand[]
prepend/insert a new line before that one
containing: static OpenThreads::Mutex _mutex;
next, navigate to the line containing osg::State* state = renderInfo.getState();
prepend/insert a new line containing OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_mutex);


recompile sg+fg, test and report back :D

Here is a link to Hooray's post. Sometimes my brain doesn't work as well as it should.
viewtopic.php?f=4&t=39475&start=75#p390955
Fly low. Fly slow. Land on a dime. Twin Otter. https://github.com/SurferTim/dhc6p
My other aircraft is a Citation-X https://github.com/SurferTim/CitationX
PirateAir videos at https://www.youtube.com/user/SurferTim850
User avatar
SurferTim
 
Posts: 1711
Joined: Sun Dec 09, 2018 6:49 pm
Location: Miramar Beach, FL
Callsign: Pirate
Version: 2020.4.0
OS: Ubuntu 18.04

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby Hooray » Mon Feb 07, 2022 4:39 pm

FWIW, not sure if any of you filed a proper ticket for this or not, but this issue is currently being discussed/re-discovered on the devel-list (with most core devs seemingly being unaware of this issue first being reported back in August 2021 ?):

https://sourceforge.net/p/flightgear/ma ... /37604861/
Investigating why FG will often segfault when setting the OSG threading
mode to CullThreadPerCameraDrawThreadPerContext, I wondered whether we are
doing things in cull callbacks that may not be thread safe, which would
explain the crashes.

Does anyone have any deeper insights here, before I dive into the depth
of it?

https://sourceforge.net/p/flightgear/ma ... /37604984/
But the crashes I've seen so far all seem to be related to canvas and
culling - the trace either points to somewhere in the canvas namespace,
or to something FBO-related, or so something related to culling, or,
worst case, somewhere in the bowels of the nvidia drivers.

So my guess would be that the canvas code does things to its FBO's in
either the update or the cull callback. I thought initially that it's
probably in the cull callback, because that's the difference between
CullThreadPerCameraDrawThreadPerContext (which crashes almost every
time), and CullDrawThreadPerContext (which seems to either crash on
startup, or not at all). But it could of course just be that in the
latter threading mode, the timing is different and makes races much less
likely.

https://sourceforge.net/p/flightgear/ma ... /37605328/
So far however everything points at canvas, and looking at
canvas code, it seems that there's a lot of stuff going on in update and
cull callbacks that looks to me like it shouldn't. I don't really
understand the architecture well enough to tell for sure though, or to
figure out how to fix it.

https://sourceforge.net/p/flightgear/ma ... /37605328/
So far however everything points at canvas, and looking at
canvas code, it seems that there's a lot of stuff going on in update and
cull callbacks that looks to me like it shouldn't. I don't really
understand the architecture well enough to tell for sure though, or to
figure out how to fix it.

https://sourceforge.net/p/flightgear/ma ... /37605458/
As part of the Core profile migration, we need to replace ShivaVG (which is the functional guts of Path.cxx) with a shader based implementation, ideally NanoVG, although Scott has indicated this might not be as easy as I hoped.

https://sourceforge.net/p/flightgear/ma ... /37605758/
From a quick eyeball just now, the code in CanvasPath.cxx looks /mostly/ to me:

- all setXYZ methods set a dirty flag
- there is an OSG updateCallback which updates the custom drawable
- drawImplementation calls the rendering commands, with some extra work to synchronise the current OSG state with the Shiva use of state.

Importantly, the vgPath is created inside update, but this /should/ only touch CPU state. Whereas if the attributes are dirty for say fill or colour or opacity, the vgPaint is re-created, and this likely touches OpenGL commands inside Shiva, so it’s done inside drawImplementation.

So, I don’t see anything grossly incorrect here, with my understanding of what OSG expects, and what Shiva does. Importantly, I don’t see anything special related to culling at all. My recollection is that even in the most aggressive threading modes, OSG won’t update a node at the same time as drawing it, which is why there’s no internal locking: if this guarantee did not hold, I think almost every OSG Group/Node would need to internally lock its state, with enormous overhead.

(There is a complication here around STATIC vs DYNAMIC nodes, but the Path node is tagged DYNAMIC, so the above guarantee should still be correct)



This goes to show, that it's indeed a good idea to file proper bug reports using the issue tracker - the people now investigating this issue, seem to be completely unaware of the forum bug report and the corresponding troubleshooting/experimental patches discussed here, which is rather unfortunate and a huge waste of time in a volunteer based project like FG ...


https://wiki.flightgear.org/FlightGear_ ... 8OpenVG.29
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby Hooray » Tue Feb 15, 2022 4:11 pm

FWIW, there are now talks on the devel-list, including early-experiments, to replace the existing Canvas.Path back-end (called Shiva) with an alternative that is shader-based, and thus, compatible with OpenGL ES/Core Profile support. This is likely to also take care of existing threading issues / race conditions:

https://wiki.flightgear.org/Canvas_news ... ng_ShivaVG
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Multi-threaded FlightGear/OSG and Canvas flickering solv

Postby SurferTim » Mon Feb 21, 2022 2:49 pm

That's good to hear. I'm still using FG2020.4.0 with the SimGear patch, and it is working GREAT! Not a single crash since.
Thanks to all who helped.
Fly low. Fly slow. Land on a dime. Twin Otter. https://github.com/SurferTim/dhc6p
My other aircraft is a Citation-X https://github.com/SurferTim/CitationX
PirateAir videos at https://www.youtube.com/user/SurferTim850
User avatar
SurferTim
 
Posts: 1711
Joined: Sun Dec 09, 2018 6:49 pm
Location: Miramar Beach, FL
Callsign: Pirate
Version: 2020.4.0
OS: Ubuntu 18.04

Previous

Return to Canvas

Who is online

Users browsing this forum: No registered users and 2 guests