Board index FlightGear Development New features

Export live render footage to train CV AI autopilot

Discussion and requests for new features. Please note that FlightGear developers are volunteers and may or may not be able to consider these requests.

Export live render footage to train CV AI autopilot

Postby aislan » Thu Oct 13, 2016 8:23 pm

Hi guys,

I'm PhD researcher and I have been using FG for a while to view simulated drone flights, using ardupilot SITL. Now, I want to train my UAV autopilot computer vision based AI using flightgear footage from the cockpit camera. In order to do it, my autopilot sends the UDP packages to FG with its telemetry data (it is already doing it through SITL), but it also needs to receive the image feedback from the camera footage in realtime.

As a short term solution, I'm pointing a camera connected to my autopilot to my computer screen to close the loop. Now I'm wondering if there is a way to grab the FG framebuffer of each frame and connect directly to my AI through OpenCV, GL, or other.

So is somebody did it or know how it can be done, please let me know. Otherwise, if somebody can give me a tip where to start in the source code I will also appreciate.

Thanks!

Aislan
aislan
 
Posts: 3
Joined: Thu Oct 13, 2016 8:01 pm

Re: Export live render footage to train CV AI autopilot

Postby PINTO » Thu Oct 13, 2016 8:30 pm

You'd probably be better off with something like Open Broadcaster Software or some-such screen capturing utility.
Actively developing the MiG-21bis (github repo) (forum thread) (dev discord) (fg wiki)

http://opredflag.com is an active flightgear dogfighting community (using a system that isn’t bombable)
User avatar
PINTO
 
Posts: 966
Joined: Wed Oct 21, 2015 7:28 pm
Callsign: pinto
Version: stable
OS: Win10

Re: Export live render footage to train CV AI autopilot

Postby curt » Thu Oct 13, 2016 8:31 pm

I have had some success with using ffmpeg to capture a portion of my screen (live) and format it as a web based video stream and then receive the video stream with an opencv program framegrabbing and processing.

Also, maybe you want to train with real video if there was a way to accurately connect the video with the flight data log? :-)



https://www.youtube.com/watch?v=CqS6Q6REBhA
Aerospace Engineering and Mechanics
University of Minnesota
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: Export live render footage to train CV AI autopilot

Postby aislan » Thu Oct 13, 2016 9:40 pm

Thanks for the answer guys. Both Open Broadcaster and ffmpeg are great. Will try ffmpeg first and hope to not add much latency. The only problem in using these methods and even my camera approach is I have to train my neural network in real world time. It means 1 hour or training will take 1 hour. If I can grab the frame from FG, I can speed up FG some order of magnitude and my 1 hour of training will take few minutes (if my hardware allows, of course).
aislan
 
Posts: 3
Joined: Thu Oct 13, 2016 8:01 pm

Re: Export live render footage to train CV AI autopilot

Postby Hooray » Sat Oct 15, 2016 5:33 pm

You could also do something like this (requires C++ changes): viewtopic.php?f=71&t=30642&p=296703#p296627


Apart from that, I'd suggest to run a forum/wiki search for "OpenCV", e.g. see: http://wiki.flightgear.org/Canvas_Devel ... ter_Vision
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Export live render footage to train CV AI autopilot

Postby Hooray » Sat Oct 22, 2016 2:38 pm

Here's another FlightGear related project that uses FlightGear created imagery, including the corresponding code: http://wiki.flightgear.org/FlightGear_W ... Box_Server

Note that the OP in other thread has meanwhile shared his changes for exposing any Canvas texture as a mongoose/httpd stream: http://wiki.flightgear.org/Read_canvas_image_by_HTTP

So you could basically integrate these two approaches to come up with a new Canvas::Element that allows scenery views (cameras) to be rendered to a Canvas texture, which could in turn be streamed to an external process and post-processed by OpenCV: http://wiki.flightgear.org/Canvas_Devel ... ter_Vision

Sooner or later, this may probably benefit from the groundwork detailed at: http://wiki.flightgear.org/CompositeViewer_Support
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Export live render footage to train CV AI autopilot

Postby aislan » Tue Jan 31, 2017 6:00 pm

Just to follow it up, I used the code described here http://stackoverflow.com/questions/2498 ... 0#24988220 (the last solution, where the author report 15000 fps) to grab the x11 part of the desktop. Thanks anyway for all the help.
aislan
 
Posts: 3
Joined: Thu Oct 13, 2016 8:01 pm

Re: Export live render footage to train CV AI autopilot

Postby Johan G » Tue Jan 31, 2017 7:07 pm

Thanks for the update. :)
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Some YouTube videos
Johan G
Moderator
 
Posts: 6625
Joined: Fri Aug 06, 2010 6:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 2020.3.4
OS: Windows 10, 64 bit


Return to New features

Who is online

Users browsing this forum: No registered users and 1 guest