Board index FlightGear Development Weather

A local weather system (v1.4 available)

Everything related to weather simulation, visuals should be discussed in the shader subforum.

Re: A local weather system

Postby Thorsten » Fri Mar 19, 2010 8:56 am

WooT, thanks for the detailed explanations. I confess I have been too simplistic in my thinking, but we agree on one point:

Still my brain wants to see the thing as moving with the global mass of air, with lower layers slowed down by ground friction.


So, the implication is that the 'curvature' of the banana may change as a function of time if the top is moved with a different speed than the bottom, hence wind and thermal development tie together.

With what function did you create the plot? If you have a C++ code or just the function, I can probably Nasal it without much fuss...

The problem doesn't seem to be to write to the property, but rather to prevent it to be reinitialized at every frame, like it seems to occur now.


So that's the reason... I just noted that this ties well in with a number of other environment parameters which I can't set directly from Nasal. For many, I could set them indirectly by modifying the config version which the weather menu dialog references and then calling

fgcommand("reinit", props.Node.new({subsystem:"environment"}));

- which may not be very clean, but did the trick, but that doesn't work for thermal lift. In any case, I guess that needs a C++ solution to be able to write it.

@Hooray:
Would you consider using an updated binary compiled from a CVS snapshot?
Are you on Linux or Windows?


I'm on Linux, and a compiled binary CVS snapshot with the features we discussed here would definitely be a good solution for me. Ultimately, I'd like to get to compile myself and would appreciate any help I can get, but right now is maybe not the best time - I have a newborn daughter at home, and my time for uninterrupted focused coding or similar work is preciously short, and I have my hands quite full with Nasal already...

As for the code of what I have so far - I'll take a few more days to write some menu functions for a basic user interface to high-level and low-level cloud placement functions, clean the code of unused and obsolete bits, write a few comments and a documentation and then release the package for everyone to look at, optimize and play around, and then go to start implementing the effect volume support, wind system and improve the realism of the algorithms and so on. If anyone else could work out an easy and clean solution for setting weather parameters (thermal lift, visibility, turbulence, wind, ...) from Nasal, that would be absolutely splendid.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system

Postby Thorsten » Sat Mar 20, 2010 12:03 pm

I've now had the time to go through Lürkens' description of the CumulusX setup and through the article on thermals which WooT linked. Since some of you seem to have thought about glider weather condition already in some detail, I'd like to pose two questions:

Lürkens starts by computing thermal energy deposition, and that is in a sense easy enough to obtain - it's proportional to the integral of the energy flux absorbed at a given surface, which primarily a function of sun angle, terrain slope and terrain albedo. Terrain slope is sort of expensive to get, as it would require repeated geoinfo() calls, all others are (relatively) simple, but there's no conceptual problem here.

But that energy is balanced by the thermal activity, and that is something like

(number of thermals) x (typical diameter of a thermal) x (typical strength of a thermal)

So I have the problem of having computed a single number that should determine three other numbers - which is ill defined, and the CumulusX manual describes that as sort of a blackbox. Does anyone have any ideas how the distribution into number, diameter and strength should be done? Right now, I more or less made up rules which give credible visual impressions...

My second question concerns the elevation of thermal clouds - Lürkens offers several options (constant altitude above sea level, constant altitude above terrain and weighted) - which of those would you consider realistic and would you like to see implemented? Technically they are equally easy to do...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system

Postby Hooray » Sat Mar 20, 2010 2:54 pm

Terrain slope is sort of expensive to get, as it would require repeated geoinfo() calls


Yes, true: that would be one of those things that would be ideally done in some sort of worker thread that processes a queue of requested coordinates and populates another queue with the computed slopes for further processing by the main thread.

To make this work reliably, one would probably want to make the geodinfo() extension function in FGNasalSys.cxx thread safe, i.e. add explicit synchronization (lock).

On the other hand, while we are at it, we could just as well add a new Nasal extension function to just compute slopes directly?

I mean, computing slopes was also required for the ridge lift system, and this would be yet another instance, and maybe there are other future uses, too?
Besides, the f_geodinfo() implementation does some more things, apart from just returning elevation for a given point. So repeatedly using geodinfo() for something that it was not conceived for is unnecessarily inefficient in a heavy use scenario with possibly thousands of calls per second.

So just wrapping it in a separate Nasal API like getslope(lat1,lon1,lat2,lon2) might not be such a bad idea?

And it would also be possible to overload the function so that it cannot just be called with one position argument each time, but instead also accept a vector (or hash) of positions to be queried, and similarly return these then directly in one swoop. That would reduce the function call overhead imposed by running a small function repeatedly at a high rate.

While this would require some modifications to the source code, it is really very feasible to do and would mostly involve copy/paste programming by taking the existing f_geodinfo() function and customizing it slightly to turn it into a function that directly computes slopes.

Are there any other features or functions that would be useful to you? I mean in the Nasal interface?
Do you need anything in particular?

Does anyone have any ideas how the distribution into number, diameter and strength should be done? Right now, I more or less made up rules which give credible visual impressions...


To be honest, I don't really have any good ideas for viable distribution heuristics, but last time I checked, there were some scientific papers available on line on this very topic.

I only quickly skimmed through these last year when were talking about how to improve gliding support (the papers were a bit too scientific for my taste). Maybe some of the ideas could indeed be used to come up with a workable approximation for modeling thermals in FlightGear:

Shallow Cumulus Clouds:A study of relations between sub-cloud length scales, cloud size distributions and entrainment profiles.
Size Distributions of Boundary-Layer Clouds
Updraft and Downdraft Statistics of Simulated Tropical and Midlatitude Cumulus Convection
A Multiparcel Model for Shallow Cumulus Convection
Entrainment Theory
An Explicit One-Dimensional Time-Dependent Tilting Cloud Model
Section 2.1 talks about computing velocities for drafts
This might be useful to compute trajectories for downdrafts (specific to thunderstorms)
THERMODYNAMIC AND ENERGY CHARACTERISTICS OF CONVECTIVE CLOUDS ON DIFFERENT STAGE OF THEIR DEVELOPMENT
Life Cycle of Numerically Simulated Shallow Cumulus Clouds.Part I: Transport
Life Cycle of Numerically Simulated Shallow Cumulus Clouds.Part II: Mixing Dynamics


To be honest, I still think most of this is really way over our heads, certainly for use in FlightGear (as long as there is not a dedicated standalone weather system) but maybe just some of the ideas could be borrowed and simplified to come up with simplistic heuristics that cater for the general case. The first two links certainly look interesting to come up with usable distribution heuristics for thermals.

Ultimately, I'd like to get to compile myself and would appreciate any help I can get, but right now is maybe not the best time

Okay, I see feel free to get in touch with me, once your spare time situation is improving (and congrats on the baby!) ;-)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system

Postby WooT » Sat Mar 20, 2010 3:20 pm

With what function did you create the plot?


this is just a fake with animated textures in blender, I made it long ago to try to have a better view of what happens.
It is not realistic in the sense that in my animation, the sinking air is like a skin all around the rising air. This is also the case in the thermal model I put in AIThermal.
While the shape of the lift is relatively easy to conceptualize, the sink is quite more complex. I will try to create a drawing to explain how I see things, and to describe a possible algorithm.

Terrain slope is sort of expensive to get, as it would require repeated geoinfo() calls


Yes, this is a problem. I think anyways we need at least a ground scan a some frequency to check the thermal foot ASL altitude while it drifts in wind.
Then we have two options. The first is available inside FG without modifications, but at the expense of more ground scans: scan an array of points around the foot, get terrain normals for these, and get an average. The second solution would be to generate a normal map, or slope map from the terrain data in the tile before start, and on the fly if we move to another tile. This would free some calculus time once done, but I presume it would be rather complex and heavy when it runs.

Does anyone have any ideas how the distribution into number, diameter and strength should be done?


This is indeed where things get horribly complex... I have been reading a few things about this problem, all I could find when trying to stay close to reality is severe mathematics , and none of these would be applicable with a fast real time process.

You could read this : http://www.drjack.info/BLIP/INFO/parameter_details.html ( from the amazing http://blipmap.walsys.net/NEWSOUTHWALES/index.html )
and this : climate.gsfc.nasa.gov/publications/fulltext/JAS-Chao-Lord-1982.pdf

just two examples to set the type of nightmare that is waiting for us...

I presume we should again go into something that doest try to replicate reality, but rather to just "make it look real".

For this, an interesting book to read is "Meteorology and Flight. A Pilot's guide to weather", by Tom Bradbury. Unfortunately, I don't find an online edition of this on google.

What they describe about thermal distribution, in common sense understandable pilot language, is the following :

( assuming flat country scenario )

On a nice morning end , when thermals begin to appear, the first clouds are usually small, the distribution is uniform, and bases are relatively low. Their life time is short, around 10 minutes , but as soon as a thermal dies, another appears just near.
At noon, when temperature raised a few degrees, bases are a bit higher, and clouds are more widely spaced. The cloud population is less uniform. Some are now bigger, but the amount of small ones is lower. ( bigger ones steal the energy from smaller ones )
At the middle of afternoon, when temperature is at maximum, bases are at their higher level for that day. The amount of clouds is even lower, and the remaining ones are bigger, and the spacing is higher. Between big clouds, large zones of low sink prevent formation of new thermals. Life time is longer too, around 15/20 minutes.
Globally, the average lifetime, size, strength and spacing of thermals is raising with the convective layer thickness.

Another feature that can be easily seen is how sometimes clouds arrange into "street" patterns. Lürkens has an elegant way to create this , using what he calls 'air texture'. this texture can be created by stacking 2 or 3 "zebra stripes" like patterns, and making them align when certain wind conditions are met. If conditions diverge from that ideal point when street are visible, the patterns are rotated, giving a more "voronoi like" pattern, which gives a more uniform distribution.
The spacing of the zebra stripes could be tied to day time / thermal energy so that clouds are more spaced in the late afternoon.

Another approach could be to make the thermal distribution with a Poisson Disk / Jitter Distribution, or anything that lets us decide the size of convective cells size depending on day time/available energy. This has the benefit to prevent overlapping thermals, but doesn't help much for streets. then when a potential thermal is plotted, the algorithm looks at underlying ground to decide if a thermal can live here or not, and its diameter/strength depending on time/energy.

I presume in any case , this will need a lot of testing/tweaking as you said in your initial post. It would be nice to find more people interested in testing and giving opinions about the level of realism achieved.

My second question concerns the elevation of thermal clouds [...]constant altitude above sea level, constant altitude above terrain and weighted


Constant above sea is not an option, I think. as soon as terrain raises enough, everything would happen below ground surface :)
Constant above terrain doesn't sound very right either to me, as cloud bases are defined by the condensation level.
I presume a weighted solution is the best.

I am not sure how we can define the condensation level , and also the top of clouds level ( inversion level ). Despite we have dewpoint, I only find temperature at ground level, and nothing about moisture. Are there ways to know about this for a given altitude ?

I have the same concern about wind speed and direction at various levels. The only solution I see is to interpolate FG winds across the thermal height.
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 5:09 pm

Re: A local weather system

Postby Thorsten » Sat Mar 20, 2010 4:54 pm

Thanks for the variety of links - I'll go through them to see if I find anything interesting!

To be honest, I still think most of this is really way over our heads


I'm doing theoretical physics for a living 8) At work, I code numerical evaluation of quantum field theory problems - so funnily enough, atmospherical dynamics is a lot easier for me to understand than writing a GUI with the xml structures for Flightgear *grin*. But yes, in all likelihood I'm going to boil the problem down to the essential first approximations...

Are there any other features or functions that would be useful to you? I mean in the Nasal interface?
Do you need anything in particular?


I had a few ideas about pre-sampling the terrain elevations to decide if barrier clouds are formed, or to determine the average elevation level of flat terrain patches for Cumulus placement. For the slope, one would need the normal of a 4-point plaquette around a given centerpoint since energy flux from the sun depends on two angles, so sampling along a line is not sufficient. But since the sun position is already known, maybe one can directly get the energy flux ratio to the maximal one?

Any of these ideas would require some rapid method of elevation data retrieval, quite possibly for a whole vector of positions to avoid lags, so that would be something to think about. As for what functions I need specifically, I'm not really sure yet what works... The good thing is that everything interacting with terrain can be done in Nasal, so I can play around with Nasal to see what I actually need, and then we can find a fast implementation.

I am not sure how we can define the condensation level , and also the top of clouds level ( inversion level ). Despite we have dewpoint, I only find temperature at ground level, and nothing about moisture. Are there ways to know about this for a given altitude ?


The pocket formula I learned for the condensation level is (temperature - dew point) * 333 in meters. There are more elaborate formulae which do that dependent on the actual humidity and which would allow to estimate the temperature profile and the condensation level as a function of altitude. The inversion level afaik can't be known a priori from the ground conditions, because that depends on non-local processes. The question is - what atmosphere physics is already implemented and available? For example, the density and pressure as a function of altitude must be available somewhere.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system

Postby WooT » Sat Mar 20, 2010 8:05 pm

I made a few pictures to describe a possible thermal algorithm, which is a slightly modified version of what I made in the current AIThermal.

It has the limitation that the sink acts as a skin around the lift core, when in reality, sink follows a different path. So far I didn't find a way to model it simply.

The idea is the following :

First, we determine the thermal foot position, and the aircraft altitude relative to it.

At this altitude we will consider a thermal "slice", on which we will work :
Image

green is calm air. blue is sink, red is lift.
The shape of the slice is an artifact for representation of Vz, the slice in the model is just flat.

We will first define how the Vz distribution in that disk is at the altitude and time since thermal birth, in a theorical vertical thermal ( no wind at all ).

Lets define some values:
Image

Let D be the distance of the aircraft from the slice center.

Lift and Sink radius are dependent on altitude, and of a max-diameter to be determined at start.

Vz-Max-Avail is the maximum lift we can expect in this thermal.
Vz-Min-Avail is the maximum sink we can expect in this thermal.

The pink curve represents Vz-max(D) and Vz-min(D).
In my implementation in AIThermal, I made it using several portions of sine curves. I am not too good with spline maths, but this would be more elegant ! Also, this is only an approximation based on experience, and certainly doesn't reflect what happens in real.

Vz-Factor is a variable we will use to determine the Vz(D) function from Vz-max(D) and Vz-min(D).

Depending on our altitude, Vz-factor( time ) is multplied by ALT-factor, so that the higher we are in the thermal, the stronger the lift. ALT-factor is certainly not linear, defining how it progesses with altitude must be discussed.

Then a time shift is added,depending on the altitude, to allow the thermal slice creation and death .
This allows to simulate the development of the thermal not only in time, but also in space : the higher the slice is, the later it will fully develop, and die. This time shift could be an approximation, based on experience and empiric observation.

Note that lift and sink must have their own time shift. Lift appears first, until the thermal reaches top. then only some sink appears, and descends slowly until the ground.

So, at a given altitude and time since thermal birth, we can determine two Vz-factors to define Vz(D) in the lift and in the sink, from Vz-max(D).

This gives us Vz depending on the aircraft distance from slice center ( yellow curve )
But where is the slice center ?

An approximation would be to let the slice drift in the wind from a distance given by the sum of wind(alt) and Vz(0) at this altitude:
Image
This is inaccurate because the Vz of this slice varied over time from zero to the Vz-max we defined until our instant. In fact the slice should have traveled more.I don't get very well how to integrate it over time, I guess this is buoyancy mightmare. We could use a factor to correct this.
Of course this has to be added to the foot shift.

So, now we know where the slice center is.
We know where the plane is.

We can then find D, our distance to the center, and thus Vz from Vz(D) !

Lets see what it gives on examples :

First the thermal is not born.
Image

Then lift begins to appear in the bottom part, and progresses to the top :
Image

Lift reached the top,some sink appear at top, note that there is still no sink at the bottom( sink has it own time shift ):
Image

Sink progress to bottom, while lift is now well established all along the height:
Image

Sink reached bottom, but lift begins to decrease at the foot. The thermal begins to die:
Image

Lift disappears in the main body, there is still some at top:
Image

Lift disappears at top, and sink begins to disappear at top too:
Image

No more lift at all, and sink disappearance made its way to bottom. Thermal is dead:
Image

You can see it animated here :
http://www.youtube.com/watch?v=ViRZREvqUiU



Well, this is all very approximative, and I am unhappy that I don't find a way to dissociate the sink flow from the main body, but I think this approach has an interesting quality : it doesn't need to compute the whole thermal to determine Vz. Only one computation is done for the current aircraft position, but still it takes the full development, structure and bending in account.

If someone has an idea how to build the sink flow and mix it smartly with lift flow... it would be nice !

The pocket formula I learned for the condensation level is (temperature - dew point) * 333 in meters.

nice !
Last edited by WooT on Sat Mar 20, 2010 9:09 pm, edited 5 times in total.
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 5:09 pm

Re: A local weather system

Postby Hooray » Sat Mar 20, 2010 8:13 pm

atmospherical dynamics is a lot easier for me to understand than writing a GUI with the xml structures for Flightgear *grin*.


:lol:, you can check your $FG_ROOT/Docs folder for some documentation on creating XML GUI dialogs in FlightGear, there you can find README.gui.

Besides, just looking at some of the simpler dialogs, and starting to modify these is probably easier than starting completely from scratch (I usually copy some boilerplate from other files and then simply customize it).

The question is - what atmosphere physics is already implemented and available? For example, the density and pressure as a function of altitude must be available somewhere.


I don't really know for sure, but I did do some quick research on that and checked the source code and just browsed the git logs, apparently the atmosphere model was recently revised by a developer (in September 2009), apparently the model is now okay up to ~260,000 ft:

Merge branch 'topic/atmos-merge' into next

John Denker's atmosphere changes. Original commit message:
Two-parameter physics-based model of atmosphere up to 262,467 ft i.e.
the top of the mesosphere. Correctly exhibits the HALT phenomenon.


You can take a look at the source code for yourself: http://cvs.flightgear.org/viewvc/source ... iew=markup

The current model still seems simplistic from a functionality point of view, but there are apparently plans discussed to change this.
Also, the current model is currently not yet exposed or used via any form of standardized interface to the rest of FlightGear according to the developer's website:

Structured Interface
There should be one module where most (or perhaps all) of the modeling of the atmosphere should be done.

Specifically, suppose somebody wants to obtain the pressure (for a given height), height (for a given pressure), density, altimeter setting for a particular locale, or the like. They should ask the atmosphere module to provide it. Similarly, if somebody wants to change something -- e.g. to cause a change in the local altimeter setting -- they should ask the atmosphere module to do it.

To say it the other way, it doesn't make much sense for the altimeter code to have one model of the atmosphere, and the ATIS code to have another model, and the gui to make changes based on its own model, et cetera. It doesn't make much sense to be exporting low-level properties such as sea-level-pressure via the native controls interface; such low-level properties should be private to the atmosphere model module, since they might not even mean anything, depending on implementation details.
http://www.av8n.com/fly/fgfs/README.altimetry.html
http://www.av8n.com/physics/altimetry.htm


But looking at the code, it seems pretty doable to provide such a "structured interface" and also make it available to Nasal scripts by adding some new functions to the Nasal API, if you think that should be useful.

Given that the developer of the new atmosphere model seems obviously interested in improving the atmosphere model and adding a structured interface on top of it, we might even be able to get some help implementing such requests ;-)

PS: wow WooT - just saw your posting: very impressive, nice to have you back on board again!!!
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system

Postby Thorsten » Sun Mar 21, 2010 10:41 am

WooT, looks great at first glance, I will go through it in detail soon! Hooray, thanks for the link to atmosphere dynamics in the code. This looks reasonably simple at first glance, I'll look up the equations a bit more to see what would be a good solution for our purposes...

Anyway, I added a link to the current version and a documentation to the first post - hope I didn't miss any important files... Please feel free to play around and let me know what works and what doesn't!
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system

Postby WooT » Sun Mar 21, 2010 11:42 am

Hi !

I've been playing a little with your new version, it looks great, with some problems.

First the interface is really nice, easy to use. Great work for someone who says doesn't know Gui coding :D

Cold front , barrier and convective system give me a severe frame rate drop with the default numbers in the Gui. When flying around, display goes quite jerky. I disabled all other effects, but still there is a framerate issue my side.
Reducing the size of the convective system to 5 km makes things acceptable my side.

Altocumulus and cirrus don't do that. I presume this is a cloud amount question.

The convective system places clouds very nicely. This looks quite realistic for the placement.

Globally, I'm not sure I like the rotating billboard method for Cu and Cb. It gives quite some shading problems, and the planes are very visible in many angles. I think my expectation for "RoI" is higher than what I see.

The view when you have sun behind you looks much better. The view when you are facing sun gives lots of artifacts.

You say in the help file, that eventually it would be possible to use 3D osg system ? How more complicated would this be ? I have been looking a bit into how the osg clouds are done, it seems they are using small billboards in greater amount, place them randomly around the cloud center, then shade them depending on angle/position in the cloud.
Maybe we could try to look into this and try to adapt the current osg clouds , giving them a proper structure / texturing depending on the cloud type ? I guess pushing it to low level rendering may improve framerate too ?
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 5:09 pm

Re: A local weather system

Postby Hooray » Sun Mar 21, 2010 2:27 pm

This is really awesome, this must be one of the biggest and most complex Nasal scripts I have seen in a while ! :-)

Yes, you did a superb job at making things very easy to control!

Actually, it is not just a "script" anymore - it really is a complete subsystem written in Nasal and also evidence to the power of Nasal scripting, the AI Objects system and the property tree to tie everything together: VERY cool indeed! ;-)

It also shows how powerful this combination of techniques is to actually prototype completely new ideas, test logics and algorithms and then turn these into new features for which the underlying systems were never really designed in the first place.

Who would have thought that it is possible to leverage all these features to come up with a working prototype of a weather model, without touching any of the C++ code?

Also, I find it absolutely remarkable how you simply come here, start discussing ideas elaborately, create well thought out presentations and documentation (wiki, homepage, README), and then shortly thereafter simply drop a comprehensive Nasal script implementing these ideas, all within a very short amount of time to actually learn all these new things, hats off!!

I don't currently have the time to look into all of this, but here are some quick notes while looking at your Nasal code:

  • more comments in the source code could be useful, maybe one short comment for each (non trivial) function or loop to describe what it is doing?
  • I think, somewhere in your code you are using "call" as a variable name, "call" is however already a Nasal keyword (function) provided by the standard library: http://plausible.org/nasal/lib.html
  • you are doing lots of manual (and redundant) string building and comparisons for file system paths and property paths using the full path every time in code each line, it would be easier to just store the common parts of a such a path in a variable (var localweather="local-weather/";)and use the ~ operator to concatenate other strings to it: setprop(localweather~"clouds",1);
  • just by using variables for common path elements, you will be able to easily customize the whole script without having to run search/replace in your text editor if you need to change property names or places, also it will save you from typos, redundant typing and help localize problems more easily
  • similarly, you explicitly compare symbols against strings, this would be another thing were instead using variables to set all strings and then refer to them would be easier and help avoid issues due to typos and redundant typing
  • you could also use a separate "toggle" function for the fast/slow loop implementation, or maybe even use a single node "loop-type"?
  • there is lots of explicit use of getprop/setprop, that's ok but you may find that a function (or loop) that makes excessive use of these slows down at some point, in such cases it is more efficient to directly use a property "node" object which you could pre-initialize outside the corresponding function or loop - that is only relevant for improving the performance of code that makes frequent and repeated use of these calles (i.e. only look into this for hotspots)
  • similarly, use of listeners is more efficient than polling (repeated getprop calls) for properties that do not change each frame, once a listener is fired you know for sure that a relevant variable was modified so that you can update it in the script
  • in some places you could simply use hashes to map variables to values, instead of using explicit if/else code later on
  • some of the constants you used are already provided by other Nasal modules
  • if you intend to keep on improving this, you might want to check out the Nasal OOP section on the wiki, your code is currently strictly procedural - but the OOP style supported by Nasal encourages encapsulation and makes it easier to provide self contained modules, which would probably be much easier to work with assuming that the complexity of your code is probably going to increase some more?
  • for the same reasons you might at some point want to look into splitting your code into separate Nasal files, to make it easier to logically split up your modules according to their purpose and responsibilities, so that you don't end up with some giant "monster script" eventually that noone else can understand and maintain ;-)


You say in the help file, that eventually it would be possible to use 3D osg system ? How more complicated would this be ? I have been looking a bit into how the osg clouds are done, it seems they are using small billboards in greater amount, and place them randomly around the cloud center.
Maybe we could try to look into this and try to adapt the current osg clouds , giving them a proper structure / texturing depending on the cloud type ?


That sounds like a cool idea, maybe there is a good way to directly also expose the current 3D clouds system to the property tree and AI object systems, so that native clouds could also be created and fully controlled using the same technique. Maybe this would be even possible for the particle system? That would surely give us many exciting possibilities for prototyping more complex weather scenarios.

Besides, the documentation you have written is really well done - but a section that details useful enhancements to the C++ code that you may need, would be helpful to see how we can help you further improve things.

I think we have already discussed a number of possible ways to improve the Nasal API but also for improving the interface to the 3D clouds and weather systems, so starting to summarize and prioritize these would probably be a good starting point, which would also ensure that there are plans in place to improve the infrastructure as long as it is not yet restricting your work too much.

This also applies to the various restrictions and issues that you have encountered in the meantime (such as properties that are reinitialized each frame and can thus not be controlled from Nasal).

Most of these things are fairly easy to change, and I think most of us agree that removing such restrictions would be a good thing in the long run because it makes FlightGear as a platform more powerful, but such things need to be documented, so that we don't forget about them.

A slow loop with 1 Hz takes care of distant clouds, and only nearby clouds with large anguar motion are handed over to the fast rotation loop with 20 Hz. In tests, acceptable framerates with 700 cloud models in the sky have been observed.


This sounds like it could be very useful to be used as a simple benchmark and stress test for the AI objects and 3D rendering system. Just by monitoring the framerate counter and its trend while increasing and decreasing scene complexity, should help use come up with those places in the source code that could use some optimizations (in both, Nasal and C++).

For the time being, just using systime() to record the time spent in certain functions and loops would still be the easiest thing to see which routines are most complex, and would thus benefit from optimizations.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system

Postby Thorsten » Sun Mar 21, 2010 3:58 pm

@WooT:

Cold front , barrier and convective system give me a severe frame rate drop with the default numbers in the Gui. When flying around, display goes quite jerky. I disabled all other effects, but still there is a framerate issue my side.
Reducing the size of the convective system to 5 km makes things acceptable my side.

Altocumulus and cirrus don't do that. I presume this is a cloud amount question.


Cirrus places static models - in a test run I placed 1500 static clouds into the sky without significant performance loss. So the chokepoint is the rotation of clouds, and that scales with the cloud count. You can check the number of clouds in the property browser under local-weather/clouds/cloud and simply read how many instances of 'cloud' are present and compare that against the framerate count. for comparison, I nominally get something like 70 fps with the ufo in otherwise clear sky, when I run the coldfront I am down to 30 or so. With many clouds, there is also the issue that the slow loop makes a characteristic small stop every second. I know how to fix that (do no more than 50 clouds at a time) - I just haven't implemented it.

The cloud rotation loops as they are are really just to have something. I envisioned early on that they should exist on the C++ level, for example as additional options to the billboard animation (which is completely view-axis based and looks awful with clouds in my opinion). There are more complicated trafos including a version that blends too close layers into alpha which Stuart has been using which would give more realistic results as well at the expense of CPU time - I outlined some in the Wiki. As for Hooray's question what would be useful from the C++ side - support for various cloud trafos would sure be one thing, I suppose that could enhance performance quite a bit. I'll add a more complete list of feature requests to the Wiki :)

As short term fix, you could simply change the timing of the loops. At the end of slow_rotation_loop in local_weather.nas you find a line settimer(slow_rotation_loop, 1.0); - if you increase the number, the loops run slower and eat less performance, although clouds may appear a bit more jerky.

Globally, I'm not sure I like the rotating billboard method for Cu and Cb. It gives quite some shading problems, and the planes are very visible in many angles. I think my expectation for "RoI" is higher than what I see.


I just fiddled around with shading... You can un-shade them by changing the xml wrappers to the emissivity of the Cirrus models. I guess the shading can also be improved by not projecting the texture on plane surfaces but by using curved surfaces instead. That also increases the RoI somewhat, but I haven't tested it yet. The view from above can be improved by simply adding more layers - for which I need more textures 8) I guess basically improving the visual appearance is just a question of man- and CPU power. If you take the comparison with Stuart's 3d clouds, I can't really measure up - but the development time for my Cumulus models was like 2 hours. Spend 20 hours to optimize textures, shading and transformation, and you get way better results. But I guess against the standard thermal cap cloud mine don't look so bad :D

The view when you are facing sun gives lots of artifacts.


Like what? For me, they just seem to whiten out, which is sort of okay.

You say in the help file, that eventually it would be possible to use 3D osg system ? How more complicated would this be ?


I don't know. Supposedly it is possible to simply declare a suitable shader effect for the cloud model, but I know next to nothing about the shader effects and their possibilities (that was one of my 'I need help items' in the first post. It would be quite cool to be able to use that technology.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system

Postby Hooray » Sun Mar 21, 2010 8:35 pm

Another benefit of eventually splitting the script into several files is to start differentiating between stable "library" code and experimental code for developing new features. That should help simplify the development process and also make potential troubleshooting sessions easier, because you know that only certain files and functions were modified. In addition, possible efforts to move certain functions to C++ space would probably also be simplified.

You can check the number of clouds in the property browser under local-weather/clouds/cloud and simply read how many instances of 'cloud' are present and compare that against the framerate count.


We could just as well add a simple "cloud-count" property to the "clouds" node to make this a bit easier and then use it to compute a "percentage of default framerate" to show the relationship between number of clouds and framerate achieved.

for comparison, I nominally get something like 70 fps with the ufo in otherwise clear sky, when I run the coldfront I am down to 30 or so. With many clouds, there is also the issue that the slow loop makes a characteristic small stop every second. I know how to fix that (do no more than 50 clouds at a time) - I just haven't implemented it.


I see, apart from adding additional native cloud transformation types, there are probably not that many ways to realistically optimize the 3d rendering any further without resorting to occlusion culling, which is surely used by the C++ 3D clouds code internally and which would also be directly available there in the form of various SimGear and OSG helpers. So it is questionable if that should be tried at all in scripting space.

A relatively straightforward possibility might be to dynamically determine which clouds are within a configurable range and "field of view" (horizontally and vertically) of the viewer, in order to update only those with higher priority - while the ones that are not currently visible (i.e. "behind" the aircraft or viewer) are updated with lower or default priority, so that clouds ahead of the viewer get prioritized.

Such things are usually done using so called "octrees" (which could also be used for doing indeed occlusion culling).

creation of more cloud textures - if anyone is good with an image manipulation program, the more distinct cloud textures of various types the better


This is probably one of those situations where it is helpful to make things as easy as possible, because those people doing textures are often not necessarily developers, and they may not even use Linux. In other words, packaging the whole thing as a ZIP file instead of a tarball would probably make things easier to users on non nix platforms.

The view from above can be improved by simply adding more layers - for which I need more textures 8) I guess basically improving the visual appearance is just a question of man- and CPU power. If you take the comparison with Stuart's 3d clouds, I can't really measure up - but the development time for my Cumulus models was like 2 hours. Spend 20 hours to optimize textures, shading and transformation, and you get way better results.


I think the lack of textures was also a problem back when the code for 3D clouds was added to FlightGear and the past has shown that these things happen sort of automatically once a new feature gets wider exposure due to being available in a release.

Contributions that are added to CVS also get wider exposure because they are reviewed and used by most developers, also CVS snapshots and pre-release testers will obviously be able to check new features before other users.

In other words: release (or commit) early and often ;-)

Another possible way for getting more people to try out your work is to just post to the new "News" forum or even to the FlightGear mailing lists (devel and users) or just show up on IRC.

In addition, you could add a short notice about your work to the latest newsletter draft on the FlightGear wiki, maybe including a request for help (textures, artwork): http://wiki.flightgear.org/index.php/Fl ... March_2010

That would probably also help to get more people to take a look at your project.

Also, some people may find it complicated to install and configure experimental extensions.
For example, there is code in the Nasal directory available to dynamically add items to the menubar without having to modify or replace the menubar.xml file (the menubar is also just a list of properties that can be written to and then refreshed dynamically).
That would probably be another thing worth looking into to make things somewhat easier.

Besides, if this is especially about getting more textures and cloud models contributed for testing purposes, it might be a good idea to look into providing a corresponding low level placement mode for the script, so that potential contributors can easily use their own textures and 3D models for placing clouds dynamically.

That should be a fairly simple thing to do, because it would only require two paths to be configurable and a way to parametrize the placement code (e.g. custom transformations).
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system

Postby Thorsten » Mon Mar 22, 2010 10:18 am

You're moving too fast for me... if I answer to all things you write in time, I don't get to code anything new :D

Thermals

Okay, I've had a detailed look at the thermal model. Basically, what this is is a factorized approach, i.e. the total lift as a function of distance from center d, altitude alt, and time t can be written in the form

L(d, alt, t) = f(d) * g(alt) * h(t)

or in more detail

L(d, alt, t) = (f1(d) * h1(t) + f2(d) * h2(t) ) * g(alt)

since you want the time dependence of sink and lift separately. Factorized approaches are always neat, as they are simple to compute, behave well in integrations and have a few other nice properties, so I would go with the model unless we realize that it doesn't give the desired results, it looks very nice as it stands.

Atmosphere

Specifically, suppose somebody wants to obtain the pressure (for a given height), height (for a given pressure), density, altimeter setting for a particular locale, or the like. They should ask the atmosphere module to provide it. Similarly, if somebody wants to change something -- e.g. to cause a change in the local altimeter setting -- they should ask the atmosphere module to do it.


That's precisely my idea how things should work. The question to pose to the person who wrote this is in my view as follows: The weather system is usually tied to ground observables. So given that we can supply, say, temperature, dew point (humidity), and pressure at the ground level, can the atmosphere code compute these at all altitudes assuming there are no changes like inversion layers? Or what additional input would be needed? We can certainly also input the (rough) location of inversion layers based on cloud levels - would that help?

I'd like to set up the weather system in a way that it interpolates in space and time between weather parameters supplied at given locations, but the altitude dependence should ideally read a set of properties that the weather interpolation sets and then build the atmosphere in altitude based on the input properties.

So if we could find out what parameters the atmosphere code would need, that would be a good start.

Nasal

Thanks for the words of praise and for the detailed comments to the code - I'm still learning Nasal, and this helps a lot into which direction to think. I will try to improve the quality, some things are easy to do, others are not.

Probably the most difficult is that one:

if you intend to keep on improving this, you might want to check out the Nasal OOP section on the wiki, your code is currently strictly procedural - but the OOP style supported by Nasal encourages encapsulation and makes it easier to provide self contained modules, which would probably be much easier to work with assuming that the complexity of your code is probably going to increase some more?


The problem is that OOP is alien to my thinking - it's not how I usually work and I know next to nothing about it. I realize that outside scientific computing that's more the way to do it and that there are advantages to it, my problem is that I come from scientific computing where most of the time the OOP superstructure just slows things down, so it's not used frequently. I would need to learn the whole philosophy from scratch... so don't expect me to adopt that style of coding quickly, although I see the advantages for the problem at hand.

(Btw., based on your comments, you might enjoy a look at the PYTHIA source code, one of the things I work with - it's 50.000 lines of very sparsely commented FORTRAN code in a single file with brilliant variable naming conventions (pi for example shows as PARU(1) in the code) - so naturally my own perception of my 700+ lines Nasal is that it is very easy to read and well commented...).

Other

In other words, packaging the whole thing as a ZIP file instead of a tarball would probably make things easier to users on non nix platforms.


Really? I thought since scenery files also come as *.tgz it doesn't make a difference. I haven't used Windows for ages, but I thought the decompression programs recognize *.tgz automatically and just decompress the archive...

The Newsletter is a good idea... Let's see if I find the time...

I added a feature request part and the links to the development status of the package to the Wiki (and also a showcase picture) - let's see what happens.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system

Postby Hooray » Mon Mar 22, 2010 2:03 pm

You're moving too fast for me... if I answer to all things you write in time, I don't get to code anything new :D

I am sorry, really!
Don't bother replying to everything that I write, really: don't!
I was just trying to provide some feedback.

I do realize that I provided quite a lot of "feedback" during the weekend.

As you will have noticed, I am pretty interested in all this weather stuff myself. I was probably also one of the most active participants in the ridge lift discussion, if I remember correctly! ;-)

Another reason is, that I will probably be offline for several weeks, so I won't be able to provide any help at all during that time.

Also, I tried to provide as much feedback as possible because in the beginning you didn't get all that much
feedback at all. This is surely also because these things are better discussed with fellow developers, most people here are largely users and you may find that you get better and more reliable feedback on the developers mailing list or at least on IRC at some point.

So I do appreciate your efforts and was only trying to fill that feedback void in the meantime ;-)

That you were getting fairly sparse feedback in the beginning, and now getting indeed too much might also be a sign of success, I guess? :-)

Regarding your questions on the atmosphere model in FlightGear, we should be able to get in touch with the developer who has developed the new model code, either using the mailing list or directly his website.

Besides, I really would not worry too much about my comments regarding OOP. To be honest that would be a also form of "optimization", not necessarily in the sense of performance but instead in the sense of improved readability and maintainability.

While OOP may make some things more intuitive and easier to grasp, it also does add abstraction layers and you are right that OOP is usually understood to have performance penalties associated with it. Even though I am not sure if this should be noticeable in our situation.

Here's a layman's description that I just came up with:
Code: Select all
OOP is all about creating "things" (i.e. a cloud) with "actions" (transform,draw,update) (or "messages").
Where a class (or hash in Nasal) is the "template" for a "thing" containing a number of member fields.
So the class only describes the "layout" or properties of objects that can be created.

These member fields can be variables (e.g. lat, lon, alt) or functions (setAlt, setPos).
And the actual instance (cloud[n] in the property tree) of such a thing is then called an "object".
Functions that work with instance specific state are called "methods", they may refer to instance specific state using a "self reference" (me) in Nasal, that ensures that access to a field or method is using the right instance specific data.

In OOP, internal state is managed by wrapping everything in a class using accessor functions for modifying and getting internal values.
So internal state would in turn only be modified by an abstract interface: class "methods".

For example, instead of doing something like cloud.lat=10.22; cloud.lon=43.22; you would have a method accepting lat/lon variables: cloud.setPos(lat, lon);

That means that the actual variables containing the values for lat/lon are not exposed or used outside the actual object. This is called encapsulation and provides you with a way to manage state and ensure that internal state is valid at all times, because other code may only use the external interface.

This allows you for example to simply rename a class variable, without having to change any of the code that uses the object, because other code only uses class methods.

Another important thing in OOP is separation of concerns, i.e. you don't want to end up with huge bloated "super classes" that manage all sorts of different state, but instead use different classes where appropriate to split code into abstract "modules" with well defined responsibilities.

So, one of the very first steps to convert procedural code to OOP code would be to group your code into a number of logical "classes" (e.g. cloud, cloud field ).

Classes may be composed of other classes, i.e. a "cloud field" class would in turn contain "cloud" classes.
This is then called "composition".

Another way is inheritance, where a type may inherit properties (fields:variables and methods) from a "parent" class. Imagine it like taking a "template" for the class and then saying "make a new class using this template".

Inheritance has the added advantage of providing a means to customize class behavior without having to modify the actual class, because all member fields can be parametrized.

For example, a "cumulus" cloud class could be derived from the "cloud" class, just by parametrizing it (different cloud model, different texture, different transformations), without touching anything in the actual "cloud" class.

This is basically how OOP may be understood: things are classified according to "is a" or "has a" relationship.

Of course, one may still use objects like conventional variables for passing and returning parameters.



All this is not really complicated, you are in fact already doing OOP style programming just by reusing the code from the Nasal directory (i.e. props.node):

Code: Select all
e.getNode("orientation/true-heading-deg",1).setValue(270+angle);


I understand that the procedural style seems more natural to you, and just looking at some of your code functional programming probably feels even more natural to you then, because it is even closer to mathematics.


I didn't really mean to say that your code lacks comments, I didn't have problems skimming through it.
It's just that things are sometimes easier to understand with some short notice, especially in the long term.

And I only pointed out how OOP and separate Nasal files may simplify maintenance of the script in the long run because it is already rather impressive and sizable and these techniques may help isolate those parts of the script that you consider "stable" and those that are under active development.

As previously said, looking at your script it is obvious that you could have saved quite a lot of typing by making more use of variables, so I am only pointing this out here to help you save some time, for example:
Code: Select all
    if (info[1].names[0] == "BuiltUpCover") { p = p+ 0.35;}
    if (info[1].names[0] == "Town") { p = p+ 0.35;}
    if (info[1].names[0] == "Freeway") { p = p+ 0.35;}
    if (info[1].names[0] == "BarrenCover") { p = p+ 0.3;}
    if (info[1].names[0] == "HerbTundraCover") { p = p+ 0.25;}
    if (info[1].names[0] == "GrassCover") { p = p+ 0.2;}
    if (info[1].names[0] == "CropGrassCover") { p = p+ 0.2;}
    if (info[1].names[0] == "Sand") { p = p+ 0.25;}
    if (info[1].names[0] == "Grass") { p = p+ 0.2;}
    if (info[1].names[0] == "Ocean") { p = p+ 0.01;}
    if (info[1].names[0] == "Marsh") { p = p+ 0.05;}
    if (info[1].names[0] == "Lake") { p = p+ 0.01;}
    if (info[1].names[0] == "ShrubCover") { p = p+ 0.15;}
    if (info[1].names[0] == "Landmass") { p = p+ 0.2;}
    if (info[1].names[0] == "CropWoodCover") { p = p+ 0.5;}
    if (info[1].names[0] == "MixedForestCover") { p = p+ 0.1;}
    if (info[1].names[0] == "DryCropPastureCover") { p = p+0.25;}
    if (info[1].names[0] == "MixedCropPastureCover") { p = p+0.2;}
    if (info[1].names[0] == "IrrCropPastureCover") { p = p+0.15;}
    if (info[1].names[0] == "DeciduousBroadCover") { p = p+0.1;}


Just adding a temporary variable "var landcover=info[1].names[0];" would have made this a bit shorter and more self explanatory: ;-)

Code: Select all
var landcover=info[1].names[0];
if (landcover == "BuiltUpCover") { p = p+ 0.35;}
if (landcover == "Town") { p = p+ 0.35;}
if (landcover == "Freeway") { p = p+ 0.35;}   


And what I previously said about using hashes (dictionaries) for this instead of explicit if/else comparisons would boil down to:

Code: Select all
var landcover_map = {BuiltUpCover: 0.35, Town: 0.35, Freeway:0.35, BarrenCover:0.3};


I am saying this here because getting familiar with hashes, is basically the very first step to understanding the OOP stuff.

So you basically create a variable named "landcover_map" here that contains various keys and associated values.

So that you could replace the mega if statement just with something like:
Code: Select all
# only added the first 4 types:
var landcover_map = {BuiltUpCover: 0.35, Town: 0.35, Freeway:0.35, BarrenCover:0.3};
var landcover=info[1].names[0];
p = p + landcover_map[landcover]; # to be sure it's available, you could use the "contains" call to check for key existence


You could also use such a hash to save script specific globals, for example: you have a number of instances of var path="Models/Weather/blank.ac" in your script.
Just having a global hash instead for such common variables would make it easier to change things later on:

Code: Select all
var models = {blank:"/Models/Weather/blank.ac",other:"/Models/Weather/other.ac"};


So that later on you could simply refer to models.blank (or models[blank] - which is equivalent) in your code.
The same technique could be used to map the paths for the various cloud XML files. That would remove some of the complexity due to large blocks of conditionals.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system

Postby Thorsten » Mon Mar 22, 2010 3:49 pm

I've been looking a bit into terrain pre-sampling to assign more detailed meaning to an altitude of the thermal clouds in mountain terrain. Basically, under some conditions (slowly rise of the whole terrain) I assume the clouds would follow a constant elevation above terrain, whereas under different conditions (isolated peaks) the clouds would not follow the peaks, as the air layers can flow around.

The current algorithm for placing the 3d cloud layer seems to employ altitude above the lowest point in the terrain tile. In some cases, I've observed that to fail quite interestingly. For example, in the Grand Canyon scenery, clouds are placed down in the Canyon (I've been there and I know for a fact that this is not what happens). In the Sierra Nevada foothills, cloud layers at constant altitude intersect sometimes with the slowly rising terrain.

3 seconds worth of CPU time (1000 geoinfo() calls) can get the following information about the distribution of terrain altitude (P(elevation) is the probability density of finding terrain at elevation):

Image

Image

In cases like Geneva, it's quite clear where the level terrain is and where elevations are. Other locations (like Bozen or Zell am See) are a bit more tricky to parse and assign above what the Cumulus layer should follow and what is isolated peak. The Grand Canyon is nicely resolved though, and the upward slope of the Sierra Nevada foothills are also quite characteristic. I have a few ideas about selection rules to determine from this info how to determine if the clouds follow the terrain elevation or not in each case. In addition, that info should also be able to determine if it's worth calling a barrier cloud generation call for a tile.

However, all this should probably not go into the main loop but into a worker thread...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

PreviousNext

Return to Weather

Who is online

Users browsing this forum: No registered users and 2 guests