Board index FlightGear Development Weather

KUDOS to weather people

Everything related to weather simulation, visuals should be discussed in the shader subforum.

Re: KUDOS to weather people

Postby Hooray » Sat Sep 04, 2021 11:24 am

We do have http bindings in Nasal available, so the question is what existing services/data sources are compatible with FlightGear (or even just the local use case) - for instance, this is what the AloftWx utility is using:

https://digital-geography.com/cloud-gis ... ther-data/
We are going to use weather forecast data from the Global Forecast System (GFS) of the USA’s National Weather Service (NWS).

GFS is a global numerical weather prediction system that is run 4 times a day on a powerful super computer. The forecast data is released under an OpenData licence and is accessible directly at the following address

http://nomads.ncep.noaa.gov/

However if you navigate to the above link you will be confronted with data files in some difficult to use scientific formats such as GRIB (Gridded Binary). Fortunately the GFS forecast is also available on an ERDDAP server. ERDDAP is Cloud based data broker designed to act as the intermediary between developers/users and scientific data formats (particularly gridded data). Please see my previous post for an introduction to ERDDAP.


For Sea winds, see: https://www.ncei.noaa.gov/products/blended-sea-winds
The Blended Sea Winds dataset synthesizes observations from multiple satellites (up to six satellites since June 2002) to create gridded wind speeds. Blending the data fills in the temporal and spatial data gaps present in each source dataset, and reduces subsampling aliases and random errors. These products were developed in response to the increasing demand for higher resolution global datasets, and can be used to improve the accuracy of ocean and weather conditions forecasts. Please note these are research products, and thus, are experimental in nature.


@V12/lego: Regarding AloftWx, what is the issue - is it installing/shipping/configuring/enabling the tool ? If so, would it help to have another discussion how to turn this into an actual fgfs addon and extend fgfs accordingly, so that the whole thing can be executed by a Nasal addon in a background worker thread to do all the fetching/processing, for a more seamless user experience - analogous to how terrasync and fgcom are integrated these days, rather than being standalone/separate tools ?
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: KUDOS to weather people

Postby wkitty42 » Sat Sep 04, 2021 11:57 am

Hooray wrote in Sat Sep 04, 2021 11:24 am:@V12/lego: Regarding AloftWx, what is the issue - is it installing/shipping/configuring/enabling the tool ? If so, would it help to have another discussion how to turn this into an actual fgfs addon and extend fgfs accordingly, so that the whole thing can be executed by a Nasal addon in a background worker thread to do all the fetching/processing, for a more seamless user experience - analogous to how terrasync and fgcom are integrated these days, rather than being standalone/separate tools ?

ummm... original post on May 2019... last post Nov 2020...
AloftWx v0.9 Beta released

the main problem is building the binary for all platforms... lego was able to do it for windows but apparently has no cross-platform building capabilities so linux and mac are left out unless they can run the external tool in a windows emulator... the tool uses the --generic=socket to connect to FG... if the tool could be integrated inside FG, it would be so so so much better than an external addon of sorts...
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9146
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: KUDOS to weather people

Postby merspieler » Sat Sep 04, 2021 12:18 pm

There's another tool which uses openweather maps as dataprovider: viewtopic.php?f=69&t=37784
Note that you need to register an account to get an API key.

This tool is cossplatform compatible.

Altho crossplatform/integrated AloftWx would be awesome.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: KUDOS to weather people

Postby Hooray » Sat Sep 04, 2021 12:22 pm

External/3rd party APIs (and keys) are somewhat unfortunate for FlightGear to remain/become self-sufficient, i.e. implicit external dependencies, you're basically at the mercy of some 3rd party company/person/service, and the corresponding service could be discontinued immediately (c.f. the mapserver dilemma)

I realize that this is probably a separate discussion, but if lego wrote the whole tool, he can surely tell us what exactly it needs in terms of functionality, and we can work out what's currently possible/missing in FlightGear to either integrate the missing functionality directly, as an addon or add the corresponding C++ hooks to make it possible to implement such tools in scripting space.

I suppose, it primarily needs a way to download data over http/https (for which we have bindings in Nasal space), and a way to extract archived data to a local folder (let's say under $FG_HOME) and post-process such data in background thread, possibly using some 3rd party libs for certain archives/file formats ?

I don't quite understand the comment saying "install wgrib2 (not GPL or compatiable afaik so cannot distribute)" - referring to: https://github.com/erget/wgrib2/blob/ma ... NSE-wgrib2

Specifically:
https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/
Joining the development effort

Source code: knowledge of C and some grib-2
Changes to existing source code should use the same licence as the original code
New code (files) must either be GNU or public domain.

Bug reports are important
Contact wesley.ebisuzaki@noaa.gov


Thus, as far as I can tell, even directly integrating wgrib2 itself into our cmake build system (via 3rd party deps) should be legally safe, unless I am missing a really crucial point ?
And on the build server, it would not even matter if gfortran should be required to build the whole thing - the situation would not be unlike with Qt5 ,which is a no-brainer for people who get the precompiled binaries anyway.

Again, @lego: what exactly is your tool doing in coding terms and what does it need in terms of dependencies (http, threads, archive extraction, wgrib2, other libs) ?
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: KUDOS to weather people

Postby legoboyvdlp » Sat Sep 04, 2021 4:39 pm

In simple terms there's code to calculate what the current active .grb2 file is.

There's code to download the .grb2 file from servers (whatever file the code calculated is the current active one -- it depends on date and time).

There's then code that executes the wgrib2.exe file to parse the .grb2 file and save the output to a new file. Then it runs wgrib2.exe for a second time to extract the data needed from the parsed file.

Finally, it prepares all the extracted data and sends it over to FlightGear via a UDP protocol, as comma separated values -- this is injected into the local weather system via the aforementioned ipoint method.

So as far as I know all it needs is some way to download files from a server; you need some way to launch the wgrib2 executable and run it over the downloaded file, and you need some way to pass the parsed + extracted data to local weather. Not too complicated but I don't intend to do it anytime soon.
User avatar
legoboyvdlp
 
Posts: 7981
Joined: Sat Jul 26, 2014 2:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Re: KUDOS to weather people

Postby Hooray » Sat Sep 04, 2021 4:54 pm

launching 3rd party tools/executables is basically a no-go, but shipping the corresponding library/sources might not be that far-fetched.

FlightGear/Nasal can download stuff over http, and it can also extract files (depending on the file format), but I don't think that UDP/TCP connections can currently be set up dynamically (?), i.e. that would probably need some dedicated fgcommand/Nasal tooling.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: KUDOS to weather people

Postby legoboyvdlp » Sat Sep 04, 2021 5:34 pm

Yep; probably the best way would be to have some data structure or Nasal command so you can access the relevant data via Nasal, obviously it wouldn't make sense to create a UDP connection from flightgear to... flightgear :P
User avatar
legoboyvdlp
 
Posts: 7981
Joined: Sat Jul 26, 2014 2:28 am
Location: Northern Ireland
Callsign: G-LEGO
Version: next
OS: Windows 10 HP

Re: KUDOS to weather people

Postby Hooray » Sat Sep 04, 2021 5:54 pm

right, even though in a multi-threaded environment, it isn't as stupid as it sounds - i.e. in terms of IPC between different threads, imagine a scenario where downloading/extracting/post-processing the data takes longer than a single frame - at that point, you would normally want to use a background thread to handle/split up these tasks.

However, Nasal itself is designed to be threadsafe and does provide built-in synchronization primitives (mutexes/locks and I believe even semaphores via pthreads)

So basically the question is what we're dealing with in terms of downloading (number of files, file size, protocols) and what is needed to extract those files (zip, tgz, bz2, rar)
That part is pretty straightforward actually - adding the wgrib2 sources (or rather getting them added/accepted) seems more daunting, especially if this requires gfortran - at that point, it would make sense to integrate the whole thing as an optional feature via cmake feature toggles.

From an integration standpoint, Nasal cppbind is our best friend: https://wiki.flightgear.org/Nasal/CppBind
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: KUDOS to weather people

Postby V12 » Sun Sep 05, 2021 9:00 am

You are reinventing the wheel. Try to contact authors of the FSXWX / P3DWX freeware (but not GPL2 compatible) utility for cooperation and try to agree. This utility resolves all weather problems.
More than 3 years ago I made quietly attempts with simple external app with downloading and runtime parsing METAR reports from NOAA, all interpolations worked perfectly, but I did not use crucial thing - wind aloft, because METAR does'n contains it. Then I moved to another sim and abandoned development.
This app via UDP sending in 1 minute intervals interpolated fake METAR to FG, in result, this METAR was equal to NOAA METAR only in close vicinity around meteo station, what is not perfect, because METAR reports are discrete IRL. There was another ugly bug - extensive clouds popcorn and abrupt clouds changes. I did not found the way, how to solve it. Commercial WX engines probably sending to the sim clouds coordinates and other needed property and the sim only renders the clouds. I do not know more about it :(
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: KUDOS to weather people

Postby GinGin » Sun Sep 05, 2021 10:40 am

Try to contact authors of the FSXWX / P3DWX freeware (but not GPL2 compatible) utility for cooperation and try to agree


And you can’t contact them by yourself ...?
GinGin
 
Posts: 1580
Joined: Wed Jul 05, 2017 11:41 am
Location: Paris
Callsign: Gingin

Re: KUDOS to weather people

Postby Thorsten » Sun Sep 05, 2021 6:01 pm

More than 3 years ago I made quietly attempts with simple external app with downloading and runtime parsing METAR reports from NOAA, all interpolations worked perfectly, but I did not use crucial thing - wind aloft, because METAR does'n contains it.


That's what we're trying to tell you for a couple of years. So... you claim you figured it out, but you still don't understand the issue? Sounds a bit.. silly ro me.

Commercial WX engines probably sending to the sim clouds coordinates and other needed property and the sim only renders the clouds


Right... you know, you can do that and send FG cloud coordinates just as well. :D

There was another ugly bug - extensive clouds popcorn and abrupt clouds changes.


So you developed something, it had an 'ugly bug' - and somehow that is whose fault? I'm getting confused here...

I do not know more about it


For once we agree.

Then I moved to another sim and abandoned development.


Unfortunately, you didn't actually do that. You keep talking about how your commercial games are better - yet you keep posting your FG flights here, enter SOTM competitions and... keep complaining. So forgive me if I don't really buy the 'other sim' story - that 'other sim' seems to make you very unhappy if you keep putting up with FG since three years.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: KUDOS to weather people

Postby Isaak » Sun Sep 05, 2021 8:01 pm

I sincerely apologise to anyone getting annoyed because I unwillingly ignited yet another V12 vs. FG discussion. That was not my intention.

I'm glad to see some people move to something more constructive.
Want to support medical research with your pc? Start Folding at Home and join team FlightGear!
Isaak
 
Posts: 768
Joined: Sat Jun 04, 2011 3:52 pm
Location: Hamme, Belgium
Pronouns: he, him
Callsign: OO-ISA
Version: next
OS: Windows 10

Re: KUDOS to weather people

Postby wkitty42 » Sun Sep 05, 2021 8:53 pm

you're alright, isaak... stuff happens... sometimes folks just need to get things off their chest ;)
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9146
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: KUDOS to weather people

Postby V12 » Sun Sep 05, 2021 9:51 pm

erik wrote in Sat Sep 04, 2021 9:55 am:I see FSXWX uses GFS Data and GRIB2 file format, GRIB2 File reading code
That might be of interest for FligthGear too.

Erik


I overlooked this post.
Yes, FSXWX uses grib data and some math for interpolation magic. Result is very good, for freeware excellent. But NOAA data or grib decoder are not compatible with GPL2 licence (this information is 3 years old). This was one of the reasons, why I abandoned that project. Without wind aloft, app was unnecessary.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

Re: KUDOS to weather people

Postby V12 » Mon Sep 06, 2021 9:02 pm

merspieler wrote in Sat Sep 04, 2021 12:18 pm:There's another tool which uses openweather maps as dataprovider: viewtopic.php?f=69&t=37784

OpenWeather maps doesn't provide wind aloft data :

lindoro wrote in Sun Aug 09, 2020 1:21 pm:
V12 wrote in Sun Aug 09, 2020 8:16 am:Does OpenWeather provide wind aloft ?

Unfortunately not. I am still looking for a free API that provides this info. In that case, it shouldn't be difficult to add the feature to the code.
Fly high, fly fast - fly Concorde !
V12
 
Posts: 2757
Joined: Thu Jan 12, 2017 5:27 pm
Location: LZIB
Callsign: BAWV12

PreviousNext

Return to Weather

Who is online

Users browsing this forum: No registered users and 5 guests