Board index FlightGear Development Weather

Clouds online

Everything related to weather simulation, visuals should be discussed in the shader subforum.

Re: Clouds online

Postby Hooray » Wed Feb 29, 2012 5:34 pm

stuart wrote:So we really need a single consistent weather state that's pushed out to all the clients via the MP server. A full set of /environment/layer properties, plus cloud positions would probably be adequate.

This probably means a single weather server for the entire MP environment that's pushing out weather state for each of the MP users. We'd either need to proxy this via the MP server, or have clients connect to it separately.

Such a server could also provide a consistent set of AI shipping and other AI aircraft. It would also have the opportunity to do much more interesting weather simulation, free from the constraints of having to run a simulator at the same time.


Like I said previously, that pretty much sums up my own thoughts, too.

Distributed systems are neat and interesting, but there are so many potential shortcomings that cannot be easily addressed.

But, I would certainly be interested in helping come up with a distributed prototype using Anders' "MP channels" to see how far we can get in the meantime.

On the other hand, the multiplayer protocol is severely limited already, there's simply a hard limit that can be reached pretty quickly, this also applies to fgms obviously. So, I have also always thought in terms of a separate "environment server" for such state.

I can't seem to find the related forum thread currently, but personally I would suggest to think in terms of having a METAR-like server, with a higher resolution (i.e. more stations per area) that handles interpolation etc - that server could also have its own Nasal interpreter to run certain code there. Jentron mentioned this 4 years ago: viewtopic.php?f=6&t=2172&p=17200&hilit=metar+server#p17200

Most of this sounds fairly complicated and advanced at the moment, but once we have network access in Nasal (i.e. socket support) and possibly a dedicated server-side property tree, it wouldn't be all that complex to come up with a dedicated Nasal interpreter that may run as a "weather server" in a standalone fashion, and push out property updates to all connected clients, similar to the concept employed by Torsten's FGPanel code: viewtopic.php?f=5&t=15132&p=150341&hilit=#p150341

On the other hand, all of this is touching a number of areas that will be directly affected by the ongoing HLA work.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11573
Joined: Tue Mar 25, 2008 8:40 am

Re: Clouds online

Postby sgofferj » Wed Feb 29, 2012 5:52 pm

Hooray wrote in Wed Feb 29, 2012 5:34 pm:I can't seem to find the related forum thread currently, but personally I would suggest to think in terms of having a METAR-like server, with a higher resolution (i.e. more stations per area) that handles interpolation etc - that server could also have its own Nasal interpreter to run certain code there. Jentron mentioned this 4 years ago: http://flightgear.org/forums/viewtopic. ... ver#p17200

I posted a similar idea some time ago and I could help at least with the handling of the data and provide an experimental data server (http) as my main machine at home anyways pulls GFS data every 6h and does lots of stuff with it for my weather website.
FG 3.1 GIT / Opensuse 12.3 / Phenom II X4 / GForce GTX560
Stefan's little Flightgear corner | The Finnish Weather Center | Wolves in Finland

Working on: EFTP
COM: IAX2:home.gofferje.net/stefan (MO-FR 0700-2000 UTC, SA+SU 0900-2000 UTC)
sgofferj
 
Posts: 790
Joined: Mon Dec 05, 2011 4:13 pm
Location: EFTP
Callsign: OH-SW
Version: 3.1 GIT
OS: Opensuse

Re: Clouds online

Postby Bomber » Wed Feb 29, 2012 7:09 pm

I agree with Stuart...

Consider also the local creation of AAA blooms.

Simon.
"If anyone ever tells you anything about an aeroplane which is so bloody complicated you can't understand it, take it from me - it's all balls" - R J Mitchel
Bomber
 
Posts: 1934
Joined: Fri Dec 14, 2007 7:06 pm
OS: Windows XP and 10

Re: Clouds online

Postby Thorsten » Thu Mar 01, 2012 8:32 am

For an MP environment, we really need more than just the clouds coordinated - the weather as a whole needs to be consistent for all aircraft in the environment. Otherwise you'll have clouds moving against the wind, and people using opposite ends of the same runway (oh, wait, that's KSFO ).

A "first client in the area gets to define the weather" isn't great, as you'll end up with massive weather changes when two users are in adjacent tiles, one flying hard IFR, the other CAVOK VFR.


You can save yourself that trouble. If you're doing MP, you're likely online, if you're online, you have real METAR info, and real METAR info synchronizes the weather state of all clients. I thought that part is fairly trivial and only commented on getting the actual cloud position info synchronized. So the protocol just forces METAR mode whenever the tickbox 'synchronize weather' is selected.

If you create a server, then someone has to decide what the weather on the server is, and since probably no one wants to set this by hand for any place on Earth, the server is going to run METAR anyway.

It would also have the opportunity to do much more interesting weather simulation, free from the constraints of having to run a simulator at the same time.


What comes to your mind? The current constraints for realism are almost exclusively on the rendering side (and on my schedule). If CPU-intense weather simulation is needed, we could always get it out of the main thread, but computing a very differential and realistic rain field isn't going to do much good if I can't render it later on.

Also, making a global weather simulation is a very different beast... For instance, you need to get rid of local cartesian coordinates and such things, I've thought it through a while ago and decided that it's too much pain for little gain to go along this path.

What I don't understand is this: Let's assume the tile is not available, so it has to be built. Let's assume, more than a single aircraft is looking for the same tile, and they all come up with the same decision "hey, it's not there - so let's build it from scratch". How do you deal with this?


If they're trying to build at exactly the same time, then I'll accept an inconsistency (the situation will be so rare that it doesn't matter). If one makes the decision to build, it can forward the building state before it actually builds the tile to others who ask.

So, aircraft A reaches a position a second faster than aircraft B.

A: I want to build a tile at (lat, lon) - B, do you have anything?
B: No.
(A decides to build tile)
B: I want to build a tile at (lat,lon) - A, do you have anything?
A: Yes, please use (lat, lon, <other parameters>).
(B decides to use A's info)
(A builds tile)
(B builds tile)

This goes wrong only if B asks while A is still in the decisionmaking process - so dependent on how fast the info is transmitted there may be a few frames in which A would reply to B's queary 'Don't know' - so B could decide to postpone its decision till A knows.

lol, seems distributed system design is becoming a second nature to you?


No, but since this is the 4th time or so the problem comes up, I've run several scenarios in my head and think I understand the problem quite well by now.

Seriously though: Can you come up with a MINIMAL interface (i.e. prototyping!) that local weather needs to talk to other LW instances?


I think if you use METAR mode and have the same wind models, all you need to transfer is

(lat, lon, random_seed, timeofday+local_offset)

the rest of the info is taken from METAR
Thorsten
 
Posts: 11446
Joined: Mon Nov 02, 2009 8:33 am

Re: Clouds online

Postby Thorsten » Thu Mar 01, 2012 9:09 am

Of course I don't want to keep anyone from building a dedicated MP weather server. My personal conclusion is just that this doesn't gain anything in simulation realism, the challenges are not what I am interested in, my best solution to the problem is different and that therefore I would not participate in such a project.

So if there is a consensus that this should be a server, I'll stop arguing here :D
Thorsten
 
Posts: 11446
Joined: Mon Nov 02, 2009 8:33 am

Re: Clouds online

Postby Hooray » Thu Mar 01, 2012 12:33 pm

Thorsten wrote in Thu Mar 01, 2012 9:09 am:So if there is a consensus that this should be a server, I'll stop arguing here :D


To be honest, at least to me, it just "feels" more natural - but that doesn't mean that I wouldn't be interested in trying out the distributed approach that you described.

However, at the moment, I can easily see more restrictions (and open questions) related to the distributed approach than the central, server, method.

It's just that client/server designs are pretty well understood with well-definee responsibilities - distributed p2p designs on the other hand, can become incredibly tricky fairly soon.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11573
Joined: Tue Mar 25, 2008 8:40 am

Re: Clouds online

Postby Thorsten » Thu Mar 01, 2012 1:22 pm

It's just that client/server designs are pretty well understood with well-definee responsibilities - distributed p2p designs on the other hand, can become incredibly tricky fairly soon.


Well, we have two weather systems that work aircraft-centered, but zero which work server-centered - what's the smaller task - creating a p2p design to transmit a comparatively small amount of info, or write a server-based weather system from scratch?
Thorsten
 
Posts: 11446
Joined: Mon Nov 02, 2009 8:33 am

Re: Clouds online

Postby sgofferj » Thu Mar 01, 2012 1:44 pm

Does it neccessarily have to be written from scratch? I personally was more thinking like replacing METAR as an information source with something more highres, locationwise and with a few more parameters - like GFS/grib2. For this, "only" the METAR inpterpreter would have to be replaced and maybe a few adjustments would have to be made to the existing weather system.
P2P would require either direct communication between the clients, which means, every FG would need to know the IPs of every FG in the vicinity which would undoubtedly come with firewall/router-problems, or the comms would have to go through the server which would mean, fgms would have to be adapted and the traffic on the MP servers would increase, probably dramatically.
FG 3.1 GIT / Opensuse 12.3 / Phenom II X4 / GForce GTX560
Stefan's little Flightgear corner | The Finnish Weather Center | Wolves in Finland

Working on: EFTP
COM: IAX2:home.gofferje.net/stefan (MO-FR 0700-2000 UTC, SA+SU 0900-2000 UTC)
sgofferj
 
Posts: 790
Joined: Mon Dec 05, 2011 4:13 pm
Location: EFTP
Callsign: OH-SW
Version: 3.1 GIT
OS: Opensuse

Re: Clouds online

Postby Hooray » Thu Mar 01, 2012 1:56 pm

Thorsten wrote:Well, we have two weather systems that work aircraft-centered, but zero which work server-centered - what's the smaller task - creating a p2p design to transmit a comparatively small amount of info, or write a server-based weather system from scratch?

Well, I am not going to argue with you, because you are obviously much more familiar with all the code and techniques involved here - but, I wasn't really thinking about writing something completely from scratch. My idea would have been to see how the LW system could be extended to support a client/server approach, not replacing it - just trying to create another operating mode next to the existing modus operandi, and seeing how that could be generalized eventually.

Many of these things (i.e. refactoring) are obviously complicated, because there's often no encapsulation taking place in the code, and all the property tree uses also complicate things when it comes to refactoring code (like I noticed a while ago when I shuffled around some of your code code into wrappers).

But eventually, there'd be the potential that extending the system would become easier, despite Nasal not being the ideal language for large-scale systems development ;-) At the moment, every function and loop shares all the knowledge of system-wide assumptions made (e.g. tile layout, tile dimensions and such) so making a single assumption configurable, inevitably touches MANY places.

P2P would require either direct communication between the clients, which means, every FG would need to know the IPs of every FG in the vicinity which would undoubtedly come with firewall/router-problems

No, read the whole thread: we talked about those multiplayer "channels" already - just imagine it like a chat string (text property) that is propagated to all clients. Clients would then determine how relevant certain messages are.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11573
Joined: Tue Mar 25, 2008 8:40 am

Re: Clouds online

Postby stuart » Thu Mar 01, 2012 4:45 pm

I think if you use METAR mode and have the same wind models, all you need to transfer is (lat, lon, random_seed, timeofday+local_offset)
the rest of the info is taken from METAR


I'm very impressed if that's all that is required - LW goes further up in my estimation. I had assumed we needed to transfer a lot more information.

Presumably you "play forward" the cloud movement/growth from that point to the current time?

My personal preference for a global weather server largely reflect professional preference towards a client-server solution rather than a more client-client approach. My view is that you'll get a more consistent experience if the state is stored centrally rather than being shared by clients. I can imagine some edge-cases where clients fail to share their state, e.g.

Client A generates clouds for a location
Client B joins and generate clouds based on the seed from Client A
Client A leaves
Client C joins, and needs to receive the seed from Client B as Client A is no-longer present.

All of which is solvable, but more a more complicated communication than a simple request/response to a server.

-Stuart
G-MWLX
User avatar
stuart
Moderator
 
Posts: 1485
Joined: Wed Nov 29, 2006 9:56 am
Location: Edinburgh
Callsign: G-MWLX

Re: Clouds online

Postby sgofferj » Thu Mar 01, 2012 4:49 pm

Hooray wrote in Thu Mar 01, 2012 1:56 pm:No, read the whole thread: we talked about those multiplayer "channels" already - just imagine it like a chat string (text property) that is propagated to all clients. Clients would then determine how relevant certain messages are.

I did get this but it wouldn't be possible with the current MP protocol version, would it? Modification of fgms would be required?
FG 3.1 GIT / Opensuse 12.3 / Phenom II X4 / GForce GTX560
Stefan's little Flightgear corner | The Finnish Weather Center | Wolves in Finland

Working on: EFTP
COM: IAX2:home.gofferje.net/stefan (MO-FR 0700-2000 UTC, SA+SU 0900-2000 UTC)
sgofferj
 
Posts: 790
Joined: Mon Dec 05, 2011 4:13 pm
Location: EFTP
Callsign: OH-SW
Version: 3.1 GIT
OS: Opensuse

Re: Clouds online

Postby Hooray » Thu Mar 01, 2012 5:14 pm

sgofferj wrote:I did get this

No, you didn't ...
sgofferj wrote:but it wouldn't be possible with the current MP protocol version, would it? Modification of fgms would be required?


NO:
  • these multiplayer "channels" simply use "generic properties" (=string properties)
  • these generic properties have been supported by FG for years, they are used by MOST MP aircraft already
  • Now, Anders has come up with an abstraction layer to conveniently package Nasal variables in these generic properties, so that you can encode/decode values in custom message types
  • all of this ON TOP of the current multiplayer protocol
  • the concept is pretty much the identical to the stuff we talked about in your "serial port I/O thread", that's why I linked to it here: you would be using a single string property to package other information in it and transmit it.
  • some "addons" already use this, such as the wildfire system or flug's bombable addon
  • the mp_broadcast.nas script is flexible enough to be useful in other contexts, such as this one
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11573
Joined: Tue Mar 25, 2008 8:40 am

Re: Clouds online

Postby Thorsten » Fri Mar 02, 2012 8:12 am

Does it neccessarily have to be written from scratch? I personally was more thinking like replacing METAR as an information source with something more highres, locationwise and with a few more parameters - like GFS/grib2. For this, "only" the METAR inpterpreter would have to be replaced and maybe a few adjustments would have to be made to the existing weather system.


Why would that give you the same cloud configurations?

To state the problem in a nutshell:

An aircraft-centered system doesn't have to care about matching to large-scale weather patterns beyond cloud visibility range. It doesn't have to care that Earth is a sphere. It doesn't have to care if winds lead to severe inconsistencies 3 hours into tile generation, because you'll be long gone from the area by the time the problems become visible. A server-based weather system has to care about all things. There is no space beyond visible range into which you can simply let problems run.

In a different thread, I tried to sketch a balance between 'looseness' (allowing things to deviate from a strict deterministic grid to make them appear more natural) and 'determinism' (keeping the algorithms themselves stable). This balance relies on the system being aircraft centered. The system will simply not run in a server setup because I used the fact that it's aircraft centered in its design.

Basically, you will encounter a completely new set of issues having to do with continuity equations (in an aircraft-centered system, air flows always somewhere out of the simulated region - in a server-based simulation you have to know where it goes).

(Having said that, my algorithm will also not run once the MP server is densely populated, i.e. you'd actually need tiles for hours because there's always one aircraft in the vicinity. My suggestion is actually an n aircraft centered solution which assumes that n is sufficiently small that no non-linear corrections are ever needed. However, since the current MP protocol appears to collapse in dense situations anyway, I'm not seeing that as a problem).

Presumably you "play forward" the cloud movement/growth from that point to the current time?


First, it wouldn't work once dynamical convection is introduced back in. That's just too complicated to synchronize.

Second, winds are a bit of a problem since we switched to your cloud rendering. Previously, I had assigned any whole tile a windspeed for cloud rendering purposes. That would have been easy to consistently forward to any other pilot. Now we're using the windspeed at local aircraft position to render cloud movement, so the actual cloud position you see depends on the altitude profile you have flown since the tile was generated - very bad. The question is - how bad is this in practice?

For multiple display views or shared cockpit, this wouldn't be an issue because they always have the same altitude profile. For two separate planes - how much does it matter if clouds are displaced by, say, 500 m?

A solution would be to go back to a by-tile cloud drift with a speed determined by a set property as before - then you can rewind it exactly.

Client A generates clouds for a location
Client B joins and generate clouds based on the seed from Client A
Client A leaves
Client C joins, and needs to receive the seed from Client B as Client A is no-longer present.


Please give my algorithm some credit... It wouldn't notice this situation either, because it goes through all aircraft in the vicinity, asking if anyone has the tile info. So if A leaves, C wouldn't ask A for obvious reasons, it would ask B and B of course would have the info and be ready to share it.

You're thinking that someone has to be in control at all times - but that's not the case, this is just about sharing information, and once information is shared, nobody is in control of anything.
Thorsten
 
Posts: 11446
Joined: Mon Nov 02, 2009 8:33 am

Re: Clouds online

Postby Bomber » Fri Mar 02, 2012 9:27 pm

Yes it does matter If the clouds are 500m different....that's the whole premise of the thread.

T4T server will have AI AAA gun positions, all aircraft need to experience the same AAA sky pattern.

Simon
"If anyone ever tells you anything about an aeroplane which is so bloody complicated you can't understand it, take it from me - it's all balls" - R J Mitchel
Bomber
 
Posts: 1934
Joined: Fri Dec 14, 2007 7:06 pm
OS: Windows XP and 10

Re: Clouds online

Postby stuart » Fri Mar 02, 2012 11:03 pm

Thorsten wrote in Fri Mar 02, 2012 8:12 am:Please give my algorithm some credit... It wouldn't notice this situation either, because it goes through all aircraft in the vicinity, asking if anyone has the tile info. So if A leaves, C wouldn't ask A for obvious reasons, it would ask B and B of course would have the info and be ready to share it.

You're thinking that someone has to be in control at all times - but that's not the case, this is just about sharing information, and once information is shared, nobody is in control of anything.

OK, I think you've convinced me :). My comments about being impressed that you'd just need to pass a random seed, tile location and time were completely sincere BTW, incase that wasn't obvious.

Bomber wrote in Fri Mar 02, 2012 9:27 pm:Yes it does matter If the clouds are 500m different....that's the whole premise of the thread.

T4T server will have AI AAA gun positions, all aircraft need to experience the same AAA sky pattern.

T4T is of no relevance to this discussion, which concerns FG.

-Stuart
G-MWLX
User avatar
stuart
Moderator
 
Posts: 1485
Joined: Wed Nov 29, 2006 9:56 am
Location: Edinburgh
Callsign: G-MWLX

PreviousNext

Return to Weather

Who is online

Users browsing this forum: No registered users and 1 guest