Board index FlightGear Support Interfacing

Simulating and exporting target sensor data

Connecting two computers, using generic protocol, connecting with Matlab?

Simulating and exporting target sensor data

Postby wbstratton » Wed Mar 18, 2020 3:44 am

I am new to Flightgear and this forum, and am looking for assistance re a project I am working on for the US Air Force. I have outlined the context and concept below.

CONTEXT: We are working with the Air Force Engineering Center to develop analytic techniques to evaluate the accuracy (“truthfulness”) of various sensor systems to identify and track targets in complex operational environments. AETC has huge amounts of data that it collects from every sensor system on every type of aircraft in the US inventory (and foreign aircraft as well). We are evaluating the ability to apply machine learning to these huge data repositories to develop “scorecards” that evaluate the accuracy of different types of sensors on various aircraft to identify and track targets (ie. Ground based radar sites, enemy aircraft, missiles in flight, etc)

CHALLENGE: The actual data collected from aircraft during test exercises is often incomplete and inconsistent in format, contains a huge amount of extraneous data, and is classified at many different security levels (restricting access for code development). Info on the ORANGE FLAG exercises that AETC conducts quarterly:
https://www.edwards.af.mil/News/Article ... -services/
https://theaviationgeekclub.com/the-mos ... ange-flag/

CREATING SIMULATED DATA WITH FLIGHTGEAR: We are evaluating the ability to develop synthetic/simulated data using Flightgear (or other open source tool) that could represent basic operational scenarios and enable rapid development and testing of new code. By developing and validating basic scorecard building blocks with relevant, clean simulated data, we could accelerate development and the increase confidence in results when running the code on actual complex, classified data sets.

With the above context in mind, I am evaluating IF and HOW Flightgear could be used to generate flight track and sensor data that would represent the attached scenario (see screenshot on this link)
https://www.dropbox.com/s/15q8zzo5g46u3 ... o.png?dl=0

AIRCRAFT FLIGHT DATA (Timing, position, velocity)
- Flight tracks of two “Friendly” Aircraft
- Flight tracks of two “Enemy” Aircraft
- Flight Track of AWACS

SIMULATED TARGET SENSOR DATA
- Sensor data from AWACS tracking enemy aircraft and ground based radar locations (SAM sites)
- Sensor data from F-16’s tracking enemy aircraft and ground based radar locations (SAM sites)

The simulated sensor data would allow us to incorporate position error (lat, long, altitude change) and velocity error (course, speed, vertical change) that could then be evaluated against the known ground truth location of each target (based on the actual simulation scenario). Evaluating these errors for different sensors against different targets for different types of aircraft would be the basis for developing a scorecard building block for each type of aircraft.

Hopefully that makes sense! I greatly appreciate anyone taking the time to consider this and offer any insight on the feasibility of using Flightgear to generate the simulated data for the above scenario, and the associated approach and evel-of-effort for this concept.

If this proves feasible, there could be a consulting/subcontracting opportunity for a Flightgear developer/SME to assist us scenario development for our project.

Thanks very much for taking the time to consider this and respond!

You can also respond to me directly at benn(dot)stratton(cinnamon roll)juliacomputing(period)com
Last edited by wbstratton on Thu Mar 19, 2020 2:55 pm, edited 3 times in total.
wbstratton
 
Posts: 6
Joined: Thu Feb 13, 2020 6:14 pm

Re: Simulating and exporting target sensor data

Postby Johan G » Wed Mar 18, 2020 5:03 am

That sounds complex and interesting. I would say that those most capable of helping you probably are the Operation Red Flag (OPRF) team, a loosely knit group of users and developers that have done the more advanced work with combat aircraft in FlightGear. I would guess some of them might see your post, otherwise you could probably get in contact with them through their Discord server.

I guess you have looked around in documentation a bit but just in case, you might find these FlightGear wiki articles interesting:

  • Interfacing FlightGear (permalink) - Listing various ways to interface FlightGear.
  • Generic protocol (permalink) - A way to log data in a structured way.
  • Property tree (permalink) - A tree of key-value pairs (properties) and the main way parts of FlightGear communicate. Also the target when setting up FlightGear, its aircraft etc. from configuration files with subtrees of properties serialized as XML files.

Side note 1: Just to brag a bit, you may or may not be aware that our F-15, maintained largely by OPRF members, was selected for a DARPA project.

Side note 2: I took the liberty to obfuscate your email address

Side note 3: I took a look at the Wikipedia article Julia (programming language) (permalink). Somehow I had never heard of Julia. It seems impressive. To the point that I wonder if the scientific community will use it more and more for at least the heavy number crunching, like the Celeste case study. Nice to see that it is open source. :)
Low-level flying — It's all fun and games till someone looses an engine. (Paraphrased from a YouTube video)
Improving the Dassault Mirage F1 (Wiki, Forum, GitLab. Work in slow progress)
Johan G
Moderator
 
Posts: 6061
Joined: Fri Aug 06, 2010 5:33 pm
Location: Sweden
Callsign: SE-JG
IRC name: Johan_G
Version: 2020.3.4
OS: Windows 10, 64 bit

Re: Simulating and exporting target sensor data

Postby Hooray » Wed Mar 18, 2020 10:19 am

I am evaluating IF and HOW Flightgear could be used to generate flight track and sensor data that would represent the attached scenario (see screenshot on this link)
Image


flight data can be created by:
  • manually flying the aircraft
  • feeding traffic data externally (multiplayer protocol or existing track data)
  • using built-in autoflight systems (autopilot, route manager)
  • using scripted AI traffic (tanker.nas, fox2.nas)

flight data can be serialized in various way, the ones of primary interest are probably:



Finally, if you have track data already, the MP protocol provides more bang for the buck: http://wiki.flightgear.org/Multiplayer_protocol

If you need to create track data, you can use scripted AI traffic, but that won't be FDM-driven: search.php?st=0&sk=t&sd=d&sr=posts&keywords=scripted+traffic+nasal+fdm

To learn more about injecting traffic at the multiplayer level, see: search.php?st=0&sk=t&sd=d&sr=posts&keywords=multiplayer+inject+traffic

To learn more about scripted AI traffic and examples, refer to: search.php?st=0&sk=t&sd=d&sr=posts&keywords=fox2+tanker+scripted+traffic

You may also want to reach out Richard Harrison, who's been doing related things using his Emesary framework:

Image

http://chateau-logic.com/content/emesar ... flightgear
http://chateau-logic.com/content/emesar ... flightgear
http://zaretto.com/downloads.html

PS: There's a flightgear developers mailing list for people interested in FlightGear core development (C++)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11987
Joined: Tue Mar 25, 2008 8:40 am

Re: Simulating and exporting target sensor data

Postby wbstratton » Wed Mar 18, 2020 6:43 pm

@Johan: Thank you for the guidance and recommendations! Yes, Julia is an open source programming language that has its roots at MIT and allows coding in high level syntax (like Matlab) but performs like C++. For info on the open source project, see https://julialang.org/.

(Note: I am with Julia Computing, a company in Boston MA founded by the originators of the Julia programming language that provides custom development and support for the open source Julia language. Julia Computing is under contract with the Air Force to develop solutions for the above....)

@Hooray: Thanks very much for taking the time to respond and provide that info! Looks promising...I will dig in to it later today! Much appreciated
wbstratton
 
Posts: 6
Joined: Thu Feb 13, 2020 6:14 pm

Re: Simulating and exporting target sensor data

Postby Hooray » Wed Mar 18, 2020 7:58 pm

I don't think there's currently any solution to generate the kind of sensor data that you need, but it would be straightforward to create a scripted or XML based system (or a combination of both) to generate such data for each simulated aircraft.

The existing "tanker.nas" script would be easy to modify to create arbitrary flight tracks, associate a suitable 3D model and then use the position of the simulated entity to compute whatever target/sensor data you need.

Things will become a bit more complex if you need your aircraft to use actual flight dynamics, i.e. to respond to the surrounding weather/environment.

The "tanker" is a really simple piece of code that's written in a JavaScript-like language that runs directly inside the FlightGear main loop, see tanker.nas
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11987
Joined: Tue Mar 25, 2008 8:40 am

Re: Simulating and exporting target sensor data

Postby Richard » Thu Mar 19, 2020 3:24 pm

There are a number of ways that you could use FlightGear to generate data - the first idea that comes to mind is to have multiple instances of FlightGear on different computers each flown by pilots and use a private multiplayer server that would record the sensor data for each instance.

OPRF has automats (AI opponents) that are pretty good.

As hooray has mentioned you could use Emesary to communicate between the FlightGear instances - there is a bridge that will transmit over the multiplayer system.

We also have SAM simulation the main thing that is missing is decent AWACS simulation (and link 16) - but these are relatively straight forwards to add.

So using a modified flightgear multiplayer server (running on your LAN) you could record "AIRCRAFT FLIGHT DATA" and with modification to the aircraft / SAM models and some extra development on AWACS probably I'd still use the modified multiplayer server to record the "SIMULATED TARGET SENSOR DATA" - the models may need to be modified to present this data consistently but again that's not too difficult.
Richard
 
Posts: 785
Joined: Sun Nov 02, 2014 10:17 pm
Version: Git
OS: Win10

Re: Simulating and exporting target sensor data

Postby jano » Sat Mar 21, 2020 2:13 am

Not sure if you need only one scenario, or if multiple different scenario are needed, but one way to do this could be:

record a FG session with all the planes and SAMS as mp pilots (preferably using the "real time" mode available in recent next FG).

create a plane "sensor" , with only objectif to give you the sensor data , then you replay the flight as much as you want, giving you as much different sensor data as needed (eg one run per friendly aircraft, or per couple friendly/enemy)

the track data need only to be extracted once (when running the mp session, or later from the recorded flight)

I think the recorded flight would need the different planes to provide a bit of informations about their surrounding, to add errors (mask effects, ground proximity sam's radar beam shedule etc), and maybe there's something like that in the radar used by the belligerent guys from FG :)

jano
jano
 
Posts: 184
Joined: Thu Nov 29, 2007 11:32 pm
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Simulating and exporting target sensor data

Postby Michat » Sat Mar 21, 2020 5:19 am

i'll add that there is allready a written python script on git that inject traffic on mp. Without knowledge of the general public, due to danger that represent for the mp server.

Also https://fgtracker.ml/modules/fgtracker/?FUNCT=FLIGHT&FLIGHTID=7233001 provides low freq basic tracking ("due to a memory leak With immediate effect, the frequency was changed from 300 seconds to 1800 seconds"

The advantage is that the flight can be download as .kml xml https://fgtracker.ml/modules/fgtracker/?FUNCT=KML&FLIGHTID=7233001.

29.92 If that helps for the general perspective.
User avatar
Michat
 
Posts: 1041
Joined: Mon Jan 25, 2010 6:24 pm
Location: Spain
Version: 191b
OS: GNewSense

Re: Simulating and exporting target sensor data

Postby wbstratton » Wed Mar 25, 2020 3:31 am

@richard @jano @ michat Thank you very much for all the suggestions and insight! I am very impressed by the collaboration and response from the Flightgear community to my post.

I am reviewing all this with my team this week and then determine next steps. The idea of using simulated data for this project has suddenly become much more relevant with the impact COVID 19. Our primary contacts at Edwards AFB are currently restricted from going on base and are complying with the CA state work-from-home mandate. As a result, they cannot access their classified networks/data to run and debug code. Creating synthetic data maybe the ONLY way for us to continue to progress over the next several weeks/months.
wbstratton
 
Posts: 6
Joined: Thu Feb 13, 2020 6:14 pm

Re: Simulating and exporting target sensor data

Postby Hooray » Wed Mar 25, 2020 1:33 pm

FYI: If you'd like to use FlightGear itself to create any mapping/charting displays (think AWACS/RADAR), there is a dedicated 2D drawing API that is fully script-able using a built-in scripting language called "Nasal" (ECMA/JavaScript oriented), this mapping/charting framework is called MapStructure based on ARINC 661 concepts, and uses OOP (MVC) to create maps that consist of layers, which in turn display a list (vector) of symbols. The MVC approach means that there is a model that is responsible to populate the vector with elements, which are in turn passed to the layer to draw each symbol.

There are also existing layers, including layers to draw other traffic (think MP/AI). It would be straightforward to implement a dedicated "filtering" function to add/discard traffic entities based on their range, altitude, radar profile, or terrain obstructions.

We do have a number of related topics on the forum about implementing such a dedicated "radar" layer for the MapStructure framework based on adapting the existing layer that is responsible for drawing (all) MP/AI traffic.

Note that the MapStructure framework is using hashes to encapsulate hard-coded assumptions, which also means that it can be trivially used to display traffic/situations from arbitrary AI/MP nodes, i.e. by changing not just the reference point, but also using that reference point as origin for all other API calls (think terrain queries).

In addition, the use of hashes to encapsulate such hard-coded assumptions, also means that delegates can be easily injected into back-end code, without having to copy/paste things, which means that built-in data sources (navigation data, terrain data, weather etc) can be customized/overridden as needed.

The corresponding maps/charts can be used in cockpit displays/avionics, but also in arbitrary GUI dialogs:

Image

In other words, this framework is fairly flexible and could be used to create displays for arbitrary entities of your federation/simulation:

http://wiki.flightgear.org/Canvas_MapStructure
Image

Existing layers can be found in $FG_ROOT/Nasal/canvas: http://wiki.flightgear.org/Canvas_MapStructure_Layers

PS: The visuals/appearance are fully customizable, everything you see is either a part of a raster image or a set of OpenVG paths, which is also how SVG images are supported.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11987
Joined: Tue Mar 25, 2008 8:40 am

Re: Simulating and exporting target sensor data

Postby Hooray » Thu Mar 26, 2020 10:58 am

I just stumbled across this on the wiki: http://wiki.flightgear.org/MPRecorder

And we once created a dedicated "traffic generator" based on fgms, to inject live traffic feeds into the FlightGear MP environment:

http://wiki.flightgear.org/FGAIS
Image



I wasn't aware of the tool, and Michat mentioned another one that I wasn't of so far - so it'd be good to see if we could document these options.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11987
Joined: Tue Mar 25, 2008 8:40 am

Re: Simulating and exporting target sensor data

Postby Michat » Sat Mar 28, 2020 4:39 am

Hooray: I wasn't aware of the tool, and Michat mentioned another one that I wasn't of so far - so it'd be good to see if we could document these options.


The python mp traffic injector was introduced to me roughly a year ago by hindu SHM mumble user. It's been a long time since I haven't seen him in mumble. Not sure if he has presence on Discord.

Today I've been searching for that python code url address on my browser with no luck. Instead of I have found Mr. :wink: "Cowboy boots" Durk Talsma's traffic manager at sourceforge https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/src/Traffic/TrafficMgr.cxx#l3.

I don't have more information.

29.92
User avatar
Michat
 
Posts: 1041
Joined: Mon Jan 25, 2010 6:24 pm
Location: Spain
Version: 191b
OS: GNewSense

Re: Simulating and exporting target sensor data

Postby Hooray » Sun Mar 29, 2020 11:05 am

it would be awesome to get our hands on the source code - you mentioned Durk's fgtraffic effort, which has stalled as far we can tell, so such a Python script could become the foundation for fgtraffic in the future
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11987
Joined: Tue Mar 25, 2008 8:40 am

Re: Simulating and exporting target sensor data

Postby wbstratton » Wed Apr 08, 2020 10:44 pm

Hello Flightgear Team!

@hooray @michat @jano @richard @johan I apologize for going dark for the past week after the initial enthusiastic response... obviously there is a LOT going on in the world that required my immediate attention. I hope all who read this are healthy and safe in this unprecedented period we are all experiencing together.

*The Air Force has approved us developing some simulated data using Flightgear. To that end, I have drafted the following CONOPS that Id like feedback and help with.*

As described above, there are essentially two types of data we want to simulate in Flightgear:
- Flight track data of BLUE and RED forces, and
- Sensor data that each BLUE aircraft is receiving and sending over Link16 regarding the relative geographic position/track of of all the other aircraft/ targets in the exercise (air and ground based)

The actual ORANGE FLAG exercises that we are simulating are conducted at Edwards Air Force Base CA and are very complex operational environments with 12-24 BLUE FORCE aircraft (including an AWACS circling outside the AO) engaging with 6-12 RED FORCE aircraft and 3-6 RED FORCE Ground based radar (SAM sites) simultaneously. Exercises last 2-3 hours involving a wide variety of aircraft and generate a HUGE amount of raw data.

For simulation purposes, we just want to create the most basic components of the data to develop our analytic approach. As we develop and validate core components, adding complexity should be much easier. We view this as a basic “crawl-walk-run” approach, starting with the most basic scenario and then adding complexity as we develop confidence in the code and approach.

*CRAWL PHASE*
Fly a sortie of a single BLUE FORCE F-16 over R-2508 Complex (Edwards AFB air space). Takeoff, fly for 5-10 minutes and then land. Export data from Flightgear in whatever format is easiest and then we import data into our Julia development environment. (since Flightgear appears to allow us to fly anywhere in the world, we prefer to fly over the same terrain and airspace that is represented in the actual ORANGE FLAG exercise)

Complete information on R-2508 posted here:
Pilots Guide: https://drive.google.com/file/d/1SCDGp3 ... sp=sharing
User Guide: https://drive.google.com/file/d/16gwOku ... sp=sharing
Users Handbook: https://drive.google.com/file/d/1PTW6Oy ... sp=sharing

2. Fly sortie of two BLUE FORCE aircraft, one F-16 and one F-35 (or F-15 or F-22...just needs to be a different type fighter) for 5-10 min over R2508

*WALK PHASE*
3. Fly sortie of two BLUE FORCE aircraft, one F-16 and one F-35 and engaging with a single RED FORCE aircraft (with another F-16, Chinese, J-11/Su-27). Fly 5-10 min over R2508. Export data for all aircraft.

If we knock this out easily, we will then move towards RUN by adding sensor simulation for air and ground based targets, and then adding more aircraft to the environment

I am cross posting this to the OPFR Discord server since some of them expressed interest in supporting this effort, and we will likely need a multiplayer environment as we add complexity to the scenario.

As always feedback and collaboration is greatly appreciated!

Benn
Last edited by wbstratton on Thu Apr 09, 2020 12:47 am, edited 1 time in total.
wbstratton
 
Posts: 6
Joined: Thu Feb 13, 2020 6:14 pm

Re: Simulating and exporting target sensor data

Postby legoboyvdlp » Wed Apr 08, 2020 11:20 pm

In terms of exporting off the top of my head the choices you'd have would be

- fgtape mechanism - but thats more for saving flights and viewing them later
- export via UDP protocol: http://wiki.flightgear.org/Generic_protocol
- for JSBSim aircraft use the dedicated <output> mechanism to output to a .csv file (see viewtopic.php?f=36&t=33641 and http://wiki.flightgear.org/JSBSim_Logging)

One limitation is that I don't think we have any F22 unfortunately.
User avatar
legoboyvdlp
 
Posts: 7789
Joined: Sat Jul 26, 2014 1:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Next

Return to Interfacing

Who is online

Users browsing this forum: No registered users and 1 guest