Integration of ''vision in loop'' with flightgear for automa

[salman2401]

Integration of ''vision in loop'' with flightgear for automatic stabilization of unmanned/manned aircrafts using horizon detection.

Hello Friends!
As the topic suggests, i want to launch a project which aims at the implementation of algorithms for horizon detection into Flight Gear. This may prove as an auxillary aid for the autopilot in FG and ofcourse once developed to a higher degree, may promise its use in aircrafts , particularly hobby type RC planes.
I have a kind of an algorithm already developed in MATLAB , i just want some experienced members from FG team to help me in the integration process.

let this post serve as a feasibility check, please comment on wether this is possible or not, what areas would require work, like c++ codes to be added in the main FG source code or would NASAL do the job? or something else like a third party software, which may be able to feed FLIGHT GEAR with control comands?
Please elaborate.



if anybody volunteers to join hands, he is most welcomed and he can register his name here. Once the project gets some shape, we might write some research papers on this, bringing a good name to Flight Gear in the international community.
THANX.

Lookin Forward,
Sal

[MaverickAlex]

Curt it seems would be the best person to speak to in relation to this.
See this thread.

http://www.flightgear.org/forums/viewtopic.php?f=19&t=9662

[AndersG]

As the topic suggests, i want to launch a project which aims at the implementation of algorithms for horizon detection into Flight Gear. This may prove as an auxillary aid for the autopilot in FG and ofcourse once developed to a higher degree, may promise its use in aircrafts , particularly hobby type RC planes.
I have a kind of an algorithm already developed in MATLAB , i just want some experienced members from FG team to help me in the integration process.

Hi,

Here is my 50 öre (swedish cent:),
First, Nasal is not suitable for image processing and there is currently no way to access the image FG renders from Nasal. As I see it that leaves two basic options: 1) implement the processing in FG's C++ core or 2) get snapshots from FG and process them in an external application that feeds control inputs (or just the derived sensor output) back to FG. You can get screenshots out of FG at a fairly low rate using the jpeg http option or a a script that triggers the builtin screenshot function. There are several options for feeding the control or sensordata back into FG.

If it was my project I would go for the 2nd approach first. There is a risk that it can't meet your timing and synchronizatino needs[0], in which cased approach 1 might be needed.

Cheers,

Anders

[0] I think that could also be avoided by not running in real-time.

[Hooray]

EDIT: Streaming, screen-grabbing and computer vision/openCV, see: http://wiki.flightgear.org/Canvas_Devel ... ter_Vision

This will almost certainly require C++ work, as prevoiusly mentioned by AndersG, remote processing of images (i.e. "screen shots") will not be especially fast at the moment.
A good work around might be to use shared memory for writing frames to a memory region. But shared memory is a pretty platform specific thing in general. Boost may have some helpers for IPC though.

Nasal cannot -currently- access the screen, and even if it could it would be VERY slow to do any image processing directly in Nasal, this applies definitely to the Nasal main thread that is run in the main loop.

Someone who is familiar with C++ and FG could probably get something done by using opencv or ITK:

http://sourceforge.net/projects/opencvlibrary/
http://opencv.willowgarage.com/wiki/
http://www.itk.org/ITK/project/technicalsummary.html

This would basically mean adding 1-3 new dependencies (all multi platform) and do all the image processing using a corresponding library, this could then take place in a dedicated worker thread that could be made accessible via Nasal wrapper functions.

Another option might be writing images/videos to a file (or pipe) and processing it in a program that is running in parallel to FlightGear. This would be a fairly easy enhancement for FG, one could probably use the jpeg/httpd facilities to provide a similar server that basically captures the screen and writes it to a file or pipe. Another program would then be running and get its data from the file and feed back control inputs into FlightGear by writing to the FG property tree.

[Hooray]

it would not be very complicated to change FG so that it writes a binary blob of each frame to a property, so that the rendered image could be retrieved from FG just by "getting" a property, one could also use this to create a "live stream" for capturing videos. With some tweaks one could probably even directly connect real player, windows media player or quicktime to a running fgfs instance.

However, without being pretty familiar with FG and OSG (OpenGL), I think it would be way easier to enhance the existing screen shot facility so that it can be used for writing screen shots at high rates (i.e. possibly at frame rate) to a configurable location. That would not be too complicated, because the rendered image could be added to a queue, so that all images get written to disk by a dedicated worker thread (instead of the FG main thread).

Other programs (or Nasal scripts) would then merely write to the property tree to configure the worker thread for taking the screen shots, this is pretty much like it works now already.
One could for example configure a screen shot directory, and always set a file name for each frame (e.g. the unix time stamp).

You would then end up with a directory of images that are all named according to the frame's unix time stamp. This could be either used in real time, or for post processing.

This would probably be pretty fast and not affect the FG main loop significantly. There would not be any I/O due to inter process communications.

The image processing program/scripts would then merely watch the corresponding directory for write access and process newly written images that match the current time stamp.
That would be pretty easy to parallelize and it would even be possibly to run things on different machines, but then you would need a way to synchronize the time stamps being used (i.e. by using a reference property that contains an offset)