Board index FlightGear Development New features

Integration of ''vision in loop'' with flightgear for automa

Discussion and requests for new features. Please note that FlightGear developers are volunteers and may or may not be able to consider these requests.

Integration of ''vision in loop'' with flightgear for automa

Postby salman2401 » Wed Oct 13, 2010 6:47 am

Integration of ''vision in loop'' with flightgear for automatic stabilization of unmanned/manned aircrafts using horizon detection.

Hello Friends!
As the topic suggests, i want to launch a project which aims at the implementation of algorithms for horizon detection into Flight Gear. This may prove as an auxillary aid for the autopilot in FG and ofcourse once developed to a higher degree, may promise its use in aircrafts , particularly hobby type RC planes.
I have a kind of an algorithm already developed in MATLAB , i just want some experienced members from FG team to help me in the integration process.

let this post serve as a feasibility check, please comment on wether this is possible or not, what areas would require work, like c++ codes to be added in the main FG source code or would NASAL do the job? or something else like a third party software, which may be able to feed FLIGHT GEAR with control comands?
Please elaborate.



if anybody volunteers to join hands, he is most welcomed and he can register his name here. Once the project gets some shape, we might write some research papers on this, bringing a good name to Flight Gear in the international community.
THANX.

Lookin Forward,
Sal
User avatar
salman2401
 
Posts: 85
Joined: Tue Apr 20, 2010 7:24 am
Callsign: Airborne

Re: Integration of ''vision in loop'' with flightgear for automa

Postby MaverickAlex » Wed Oct 13, 2010 9:21 am

Curt it seems would be the best person to speak to in relation to this.
See this thread.

http://www.flightgear.org/forums/viewtopic.php?f=19&t=9662
http://www.atlasvirtualairlines.com
Modeller of: TNCM TBPB MDPP TFFF TJSJ FMEE TKPK TFFG TQPF KLAX KATL EGNJ EGGW APACHE, EUROFIGHTER, HP VICTOR, DHC8
FGcom 122.75
creator Terragear GUI
User avatar
MaverickAlex
 
Posts: 1204
Joined: Sun Mar 23, 2008 4:46 pm
Location: Kingston Upon Hull. UK

Re: Integration of ''vision in loop'' with flightgear for automa

Postby AndersG » Wed Oct 13, 2010 3:04 pm

salman2401 wrote:As the topic suggests, i want to launch a project which aims at the implementation of algorithms for horizon detection into Flight Gear. This may prove as an auxillary aid for the autopilot in FG and ofcourse once developed to a higher degree, may promise its use in aircrafts , particularly hobby type RC planes.
I have a kind of an algorithm already developed in MATLAB , i just want some experienced members from FG team to help me in the integration process.


Hi,

Here is my 50 öre (swedish cent:),
First, Nasal is not suitable for image processing and there is currently no way to access the image FG renders from Nasal. As I see it that leaves two basic options: 1) implement the processing in FG's C++ core or 2) get snapshots from FG and process them in an external application that feeds control inputs (or just the derived sensor output) back to FG. You can get screenshots out of FG at a fairly low rate using the jpeg http option or a a script that triggers the builtin screenshot function. There are several options for feeding the control or sensordata back into FG.

If it was my project I would go for the 2nd approach first. There is a risk that it can't meet your timing and synchronizatino needs[0], in which cased approach 1 might be needed.

Cheers,

Anders

[0] I think that could also be avoided by not running in real-time.
Callsign: SE-AG
Aircraft (uhm...): Submarine Scout, Zeppelin NT, ZF Navy free balloon, Nordstern, Hindenburg, Short Empire flying-boat, ZNP-K, North Sea class, MTB T21 class, U.S.S. Monitor, MFI-9B, Type UB I submarine, Gokstad ship, Renault FT.
AndersG
 
Posts: 2465
Joined: Wed Nov 29, 2006 9:20 am
Location: Göteborg, Sweden
Callsign: SE-AG
OS: Debian GNU Linux

Re: Integration of ''vision in loop'' with flightgear for au

Postby Hooray » Wed Oct 13, 2010 3:41 pm

EDIT: Streaming, screen-grabbing and computer vision/openCV, see: http://wiki.flightgear.org/Canvas_Devel ... ter_Vision

This will almost certainly require C++ work, as prevoiusly mentioned by AndersG, remote processing of images (i.e. "screen shots") will not be especially fast at the moment.
A good work around might be to use shared memory for writing frames to a memory region. But shared memory is a pretty platform specific thing in general. Boost may have some helpers for IPC though.

Nasal cannot -currently- access the screen, and even if it could it would be VERY slow to do any image processing directly in Nasal, this applies definitely to the Nasal main thread that is run in the main loop.

Someone who is familiar with C++ and FG could probably get something done by using opencv or ITK:

http://sourceforge.net/projects/opencvlibrary/
http://opencv.willowgarage.com/wiki/
http://www.itk.org/ITK/project/technicalsummary.html

This would basically mean adding 1-3 new dependencies (all multi platform) and do all the image processing using a corresponding library, this could then take place in a dedicated worker thread that could be made accessible via Nasal wrapper functions.

Another option might be writing images/videos to a file (or pipe) and processing it in a program that is running in parallel to FlightGear. This would be a fairly easy enhancement for FG, one could probably use the jpeg/httpd facilities to provide a similar server that basically captures the screen and writes it to a file or pipe. Another program would then be running and get its data from the file and feed back control inputs into FlightGear by writing to the FG property tree.
Last edited by Hooray on Thu May 15, 2014 4:17 pm, edited 1 time in total.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12174
Joined: Tue Mar 25, 2008 8:40 am

Re: Integration of ''vision in loop'' with flightgear for automa

Postby Hooray » Wed Oct 13, 2010 4:03 pm

it would not be very complicated to change FG so that it writes a binary blob of each frame to a property, so that the rendered image could be retrieved from FG just by "getting" a property, one could also use this to create a "live stream" for capturing videos. With some tweaks one could probably even directly connect real player, windows media player or quicktime to a running fgfs instance.

However, without being pretty familiar with FG and OSG (OpenGL), I think it would be way easier to enhance the existing screen shot facility so that it can be used for writing screen shots at high rates (i.e. possibly at frame rate) to a configurable location. That would not be too complicated, because the rendered image could be added to a queue, so that all images get written to disk by a dedicated worker thread (instead of the FG main thread).

Other programs (or Nasal scripts) would then merely write to the property tree to configure the worker thread for taking the screen shots, this is pretty much like it works now already.
One could for example configure a screen shot directory, and always set a file name for each frame (e.g. the unix time stamp).

You would then end up with a directory of images that are all named according to the frame's unix time stamp. This could be either used in real time, or for post processing.

This would probably be pretty fast and not affect the FG main loop significantly. There would not be any I/O due to inter process communications.

The image processing program/scripts would then merely watch the corresponding directory for write access and process newly written images that match the current time stamp.
That would be pretty easy to parallelize and it would even be possibly to run things on different machines, but then you would need a way to synchronize the time stamps being used (i.e. by using a reference property that contains an offset)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12174
Joined: Tue Mar 25, 2008 8:40 am


Return to New features

Who is online

Users browsing this forum: No registered users and 2 guests