Board index FlightGear Development

Ask for ideas about annotating aircraft for machine learning

FlightGear is opensource, so you can be the developer. In the need for help on anything? We are here to help you.
Forum rules
Core development is discussed on the official FlightGear-Devel development mailing list.

Bugs can be reported in the bug tracker.

Ask for ideas about annotating aircraft for machine learning

Postby stingshen » Fri Aug 25, 2017 3:34 pm

Hi guys,

I would like to train a faster-rcnn by feeding annotated aircraft pictures. The following is an example of the annotated picture:

Image

Referring to https://sourceforge.net/p/flightgear/fgdata/ci/next/tree/Nasal/view.nas, I can now use property tree to get the current position of the plane. But I can't get the viewing size of the plane. Do you have any ideas of how to achieve that?

Thanks,
Yi
stingshen
 
Posts: 3
Joined: Sun Aug 20, 2017 2:37 pm

Re: Ask for ideas about annotating aircraft for machine lear

Postby wkitty42 » Fri Aug 25, 2017 4:24 pm

what do you mean "viewing size"?? the apparent size of the craft will vary between large and small craft at the same distance from the viewer... i suspect that at some point you will need to access the craft's actual dimensions... those are generally found in the craft's xml file(s) and may be in the property tree but i've not had occasion to go looking for them to see if they do exist in the property tree...

you can access the property tree with the '/' key in the sim... there's a lot there...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 4634
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 14.04.5

Re: Ask for ideas about annotating aircraft for machine lear

Postby Hooray » Sat Aug 26, 2017 5:36 pm

You may want to tell us in more depth what you are trying to do, I am a little familiar with ML, but I don't quite see the use-case you have in mind so far ?

Speaking in general, the nature of a ANN is/works such that it does not even matter if you -the programmer- know anything about the data that you provide, as long as normalize it such that it works in the form of doubles, you can literally pass in all sorts of properties from /orientation, /position, /sim/current-view, /sim/view and let it determine what is relevant/useful to come up with the kind of algo/heuristics that you want it to "learn" (back-propagate to).


Besides, assuming that you are looking for a way to compute/render a bounding box for the aircraft to crop the size of the image so that your ANN/CNN has less work to do, you may be better off using some C++/OSG code to render such a bounding box.

Alternatively, if you really don't want to touch any of FlightGear's C++ code, and want things to do the ML-only way, my suggestion would be to use a layered ANN, with one of the first layers trained to recognize airplanes/aircraft from different perspectives, and return the image coordinates accordingly - so that you can use the first two ANNs to extract the sub-image and/or render a bounding box.

If you literally need coordinates/dimensions in meters, I guess one thing that may work is parsing the FDM itself and annotating the image using the fdm sub-tree in the property tree - in addition you could dynamically create a livery that contains information/markers to extract dimensions from it.

As you can see, there are probably several ways to approach this, but what is ultimately feasible depends really on the concrete use-case you have in mind, and the tools and expertise available - e.g. a hobbyst will usually have very different constraints compared to a PhD student in machine learning ...
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11115
Joined: Tue Mar 25, 2008 8:40 am

Re: Ask for ideas about annotating aircraft for machine lear

Postby stingshen » Sun Aug 27, 2017 1:05 pm

Hi Hooray,

Thank you for your great suggestions! Let me explain the idea in my mind first. I am a fan to photograph air crafts near airports. I would like to build a gimbal to automatically track landing air crafts. So I think the first step is to automatically identify the plane coming into the scene. I plan to try using RCNN to do that, so I need to use lots of annotated images to train the RCNN first. I would use FlightGear to generate those annotated images. The bounding box isn't necessary to be rendered on screen. I just need to write the bounding box location and size to a file.

Among your suggestions, I prefer parsing the FDM itself. I am open to use C++. But in the fdm sub-tree in the property tree, I can't find the information about the aircraft model size on the screen. Are there any properties I can combine to get the apparent size of the air craft? Or should I create a new property in the property tree to extract the information?
Last edited by Johan G on Sat Sep 30, 2017 4:50 pm, edited 1 time in total.
Reason: Please do not quote the entire preceeding post
stingshen
 
Posts: 3
Joined: Sun Aug 20, 2017 2:37 pm

Re: Ask for ideas about annotating aircraft for machine lear

Postby Hooray » Tue Aug 29, 2017 9:45 pm

First, note that there is a screenshot utility, and a streaming facility (via Phi/mongoose httpd) to easily help create those screeenshots in a scripted fashion.
There also is /some/ support for reset/re-init, i.e. changing aircraft "on the fly".

Apart from that, there is a built-in view manager and a scripted view.nas module

So you could create all sorts of screenshots semi-automatically.

What is available in the FDM tree is likely to vary between FDMs, and FDM engines, but I guess you'd be better off asking in the FDM forum if/how and where to find such info - frankly, I have never done a single FDM, so I am not in a good position to tell you if that info is readily available or not.

As a matter of fact, not knowing if each FDM provides that data, I would be inclined to suggest touching the C++ code - but there are probably better ways that I am not aware of.
For now, I suggest to ask your question in the FDM forum, tell them exactly what you need in terms of properties/units.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11115
Joined: Tue Mar 25, 2008 8:40 am

Re: Ask for ideas about annotating aircraft for machine lear

Postby stingshen » Thu Aug 31, 2017 2:29 pm

Thank you for the help! I will do some research and ask my question in the FDM forum.
Last edited by Johan G on Sat Sep 30, 2017 4:50 pm, edited 1 time in total.
Reason: No need to quote the entire preceeding post
stingshen
 
Posts: 3
Joined: Sun Aug 20, 2017 2:37 pm


Return to Development

Who is online

Users browsing this forum: No registered users and 2 guests