Remote canvas utility

[zakalawe]

(cross-posting from the mailing-list, please reply there if you have some technical comments, but I would like feedback from aircraft developers about this)

I’ve spent some of my time since FSWeekend figuring out how I might drive some ‘real’ CDU hardware from the in-aircraft CDU options, and went for a slightly ambitious approach: I’ve written an analogue of ‘FGPanel’ for the canvas. Analogue is actually not quite correct - unlike FGpanel this connects at the view level (canvas properties), it doesn’t run the canvas ‘model’ locally, since that would be very tough (Nasal + whole property tree needed). This distinction was unimportant for the canvas up until now, but it’s going to become important I expect.

This was prompted by my intention to buy something like this:

http://www.flightdecksolutions.com/comp ... color-vga/

Which means I need the CDU screen running on its own screen/device nicely. And I already wrote a page-based Canvas CDU over a year ago, it’s just sitting on Gijs’ hard-drive awaiting being merged.

This is currently proof-of-concept quality, but already works quite nicely:

- uses a websocket to send an entire property sub-tree (a canvas, but in principle could be anything) to another process / machine
- render the canvas at arbitrary size/resolution
- it’s ten source files and very simple code

It looks like this:



Of course, there are some issues:

- current code uses Qt+software rendering, I am improving this to use OpenGL rendering (but still Qt-based)

- there’s some artefacts from imported SVG which are more visible with my renderer than the ShivaVG based one. (As you can see in the screenshot above, the ‘tails’ on the compass rose ticks are present in the data indeed)

- I will investigate making a solution using Skia (https://skia.org) as the backend, because this would give good performance via its OpenGL backend, make people who don’t want to use Qt happier, and could also replace the OSG rendering of the Canvas within FlightGear itself, allowing us to ditch ShivaVG (which is unmaintained) and probably improve performance substantially. (Skia can do SSE3/NEON optimised software rendering in a background thread, so Canvas wouldn’t eat into our OSG / triangle budget)

(Of course there is no such thing as a free lunch, Skia is a large dependency too!)

I am very happy with this approach so far, since any Canvas display can be trivially run out-of-process, remotely on an Android tablet, Raspberry Pi, iPhone or whatever. (I am hoping to make an app available on the Play store and App store, but one thing at a time). The 777 ND and lower EICAS pages work and look great at larger sizes. I also investigated using a binary protocol for doing the mirroring, current one is an optimised JSON protocol but swapping in a binary protocol is very doable in the future.

This has prompted me to look at some more fundamental pieces of the Canvas, because there are some things the Canvas / Map layer do which are not very performance-friendly, especially the symbol-caching / texture-atlasing I think can be managed better on the rendering-client side. (Plus all the overhead that comes from doing lots of work in Nasal) The system would benefit from some knowledge about screen-space pixel size, so that each per-frame change in heading / aircraft position does not cause every symbol and route-path element to be recomputed in cartesian space and then re-transmitted over the mirror-socket. (But, even doing that, and with a software rendering, it’s still fast - computers are really quick….)

Unfortunately, the 757 and 777 PFD and upper EICAS are implemented as 3D animations, not Canvas, and hence currently can’t be remoted. (I need to try the 747 and MD-11 next as more test-cases) I know Thorsten R has done lots of Canvas work for the Shuttle, but I don’t know what (if any) can be generalised? I’m imagining a library of some standard elements like tapes and digital gauges, or at least best-practice examples. Or maybe someone already has a PFD and upper EICAS done than can be modified? The 777 is very similar the 787, 744 and 737NG, and the 757 one is similar to the 737-300/400/500.

Any comments on the current canvas behaviour would be interesting. And of course please try the code once I submit it - I am sure there’s many edge cases and bugs to fix. I know there was some work on HUD-via-Canvas already, it would be great if those pieces already exist in a reusable form since HUD and PFD elements are very similar.

Similarly if people have known test-cases where current Canvas performance is a limiting factor (again I know Thorsten mentioned this in the context of the shuttle) that would be interesting, since obviously there’s no point adapting more and more things to use the Canvas if the performance sucks.

[Hooray]

if people have known test-cases where current Canvas performance is a limiting factor (again I know Thorsten mentioned this in the context of the shuttle) that would be interesting, since obviously there’s no point adapting more and more things to use the Canvas if the performance sucks.

We did have that discussion a few years ago, so you are probably still aware of the details - i.e. the KNUQ taxiways rendered using Stuart's airports.xml dialog (using an embedded Canvas) - this ends up creating hundreds of Canvas path elements (OpenVG/ShivaVG):



For a more recent benchmark/test case (self-contained via the Nasal Console), see:

http://wiki.flightgear.org/Howto:Canvas ... nchmarking


Speaking in general, it doesn't make sense for us to use Nasal+Canvas and OpenVG paths at this level - it's too frine grained, it would make more sense to introduce dedicated elements handling edge cases that are too heavy from a performance standpoint, i.e. due to the resulting property tree overhead (think having elements for taxiways, runways etc to reduce the property tree overhead).

We also talked about that in the context of the shuttle's ADI ball (which is the other use-case with tons of property-throughput), i.e. introducing a new Canvas::Element specifically for loading/displaying and animating 3D models using an OSG::PositionAttitudeTransforrmMatrix (PAT):

http://wiki.flightgear.org/Howto:Extend ... _3D_models


Regarding the remote-canvas use-case, you may want to read up on a more recent development/patch inspired by your original FGCanvas idea:
http://wiki.flightgear.org/Read_canvas_image_by_HTTP


In summary, I agree that it would make sense to coordinate things a little more, because there is quite a bit of overlapping work taking place here currently - and the only sane mechanism to support external hardware like the one you linked to is indeed standardizing on a property-tree level interface - which would also be a requirement for a replication mechanism in a distributed setup with fgfs instances, e.g. those used at FSWeekend/LinuxTag, but also any MP-dual-pilot setup where certain state must be selectively replicated (AndersG's work).

This has prompted me to look at some more fundamental pieces of the Canvas, because there are some things the Canvas / Map layer do which are not very performance-friendly, especially the symbol-caching / texture-atlasing I think can be managed better on the rendering-client side. (Plus all the overhead that comes from doing lots of work in Nasal) The system would benefit from some knowledge about screen-space pixel size, so that each per-frame change in heading / aircraft position does not cause every symbol and route-path element to be recomputed in cartesian space and then re-transmitted over the mirror-socket.

That, too, has been previously discussed here - the main issue really is too much Nasal (props.nas in particular, i.e. a cppbind port may help with some issues), without its use being standardized/formalized (imagine all the use-case for handling animations using timers and listeners, which is unnecessary and a huge mess - we're sacrificing tons of optimization potential here). The main performance culprit really is property overhead (often due to more appropriate elements missing), as well as projection handling (which Gijs fixed for the hard-coded map/ND code you once wrote), and the navdb also showed up in several profiles.

Regarding the ND In general (being familiar with its code and the underlying MFC/MapStructure work), it is unlikely that it makes that much sense to continue with its development "as is", because we've already been working around the lack of core level changes for too long - e.g. introducing 2-3 new element types would make more sense, which would also benefit other aircraft, and use-cases - especially formalizing and standardizing the handling of animations at the property tree level would make sense - otherwise there is simply no sane way to get rid of all those Nasal-level dependencies.

Basically, it makes zero sense for a Canvas to depend on Nasal to this degree once it is in the tree - i.e. it could, and arguably, should be using SimGear-level native code, such as the StateMachine code or even property rules to get rid of Nasal dependencies, at that point it would become much easier to replicate/synchronize Canvas/MFD state without having too use pixel level data: http://wiki.flightgear.org/Canvas_Devel ... FlightGear

[zakalawe]

(Oh and I should have said, the yellow box visible in the screenshot is debugging a clip rect, which seems to be in a strange place!)

[Hooray]

the yellow box visible in the screenshot is debugging a clip rect, which seems to be in a strange place!

Note that Canvas, and thus canvas clipping, uses a slightly different coordinate system:

http://wiki.flightgear.org/Canvas_Element#clip.28.29
http://wiki.flightgear.org/Canvas_Nasal_API#set_2

(This sort of thing is easy to miss when doing per-pixel/streaming based synchronization/replication of the Canvas texture)