A local weather system (v1.4 available)

[Thorsten]

How embarrassing... I just discovered that the file of the package 0.85 contains the whole compressed 0.81 file and is thus twice as large as it should be Luckily it wasn't on GIT yet.

I've fixed that - the link should now refer to the correctly packaged version, which also has a bug in the 'constant by tile' wind model fixed.

If anyone has installed it yet - please delete local_weather_fgfs2.0.0_v0.81.tgz which should have been mistakenly been dumped into the Flightgear root directory. Also, I'd be happy to get some feedback how the performance improvements work out for other systems.

[HHS]

Hmmm...

win32, GIT from 08/01/2010 (old, I know), DualCore 2.8GHZ, 2 GB RAM, 8800GT

Settings: TNCM, standing rwy 27, Syd's Twinotter DHC6F, Local Weather tile, wind set 0 knt, Single Tile, Framerate-throttle: 30

High-Pressure: unstable fps between 16-30fps, after resetting the whole sim: stable fps of 31
High-Pressure-Core: unstable fps between 11-18fps, after resetting the whole sim: stable fps of 31
High-Pressure-Border: unstable fps between 21-28fps, after resetting the whole sim: stable fps of 31
Low-Pressure-Border: unstable fps between 11-13fps, after resetting the whole sim: stable fps of 31
Low-Pressure: unstable fps between 12-13fps, after resetting the whole sim: stable fps of 31
Low-Pressure-Core: stable fps of 13fps, after resetting the whole sim: stable fps of 31
Cold-Sector: unstable fps between 23-29fps, after resetting the whole sim: stable fps of 31
Warmfront: unstable fps between 14-29fps, after resetting the whole sim: stable fps of 31
Tropical: unstable fps between 6-14fps, after resetting the whole sim: stable fps of 31

Gliders Sky: unstable fps between 15-24fps, after resetting the whole sim: stable fps of 31

No idea which others features beside rain and wind are missing after resetting the sim, at least then I have the clouds without fps-impacts.


Local weather:
mostly stable fps of 30fps!


Noticed Issues:
-when clearing clouds: crashing FGFS- NaN-errors
-when ever a cloud comes into view at Local-Weather-tiles- fps impact!
-Getting error message when using Local weather: "can't find Effects/clouds-test"
-with local weather and clouds barrier: Nasal runtime error: undefined symbol: get_elevation, ...local weather.nas, line 1672, called from line 2552, called from/sim/bindings/gui/binding[10], line1
-ugly edges on some clouds-textures


This time I will win the unexistant contest:


Hope this report is helpfull!

[fredb]

-Getting error message when using Local weather: "can't find Effects/clouds-test"

My bad. I thought it was a leftover.

[Thorsten]

Thanks, that was indeed helpful.

For clarification - is dynamical weather on or off? Also, at the start, there's a lot of terrain sampling going on which should sort itself out after 20-30 seconds - so it's quite normal to have somewhat reduced framerate during pre-flight preparation.

It seems to me that it could be that your reset might be an expensive way to kill the weather dynamics loop - that would indeed leave the clouds (static) in the scenery and lead to a much improved and stable framerate (otherwise the amout of CPU time you burn depends on the number of clouds in view).

Noticed Issues:
-when clearing clouds: crashing FGFS- NaN-errors

Maddening - apparently occurs in some GIT binaries, but neither in my (older) GIT nor in my 2.0.0. I'll have a look again, but I can't spot which part of the Nasal code would be so dangerous.

-when ever a cloud comes into view at Local-Weather-tiles- fps impact!

Yep, smells like the dynamics loop doing what it's supposed to do, i.e. realize that the cloud is in your visual field and move it with 0 kt (so you can't even actually see the motion...) - and doesn't take any resources when the cloud is outside the visual.

-Getting error message when using Local weather: "can't find Effects/clouds-test"

In a sense it's obsolete - it's part of the cloudbox algorithm which is in the local weather menu. I developed it and then in the end decided to go for a different solution, but since I had it, I decided to publish it in case someone else wants to play with it. Maybe I should rename the files though...

-with local weather and clouds barrier: Nasal runtime error: undefined symbol: get_elevation, ...local weather.nas, line 1672, called from line 2552, called from/sim/bindings/gui/binding[10], line1

Haven't used that one for a long time - quite possible that I messed something up - will have a look.

-ugly edges on some clouds-textures

Well... yes. Texture cleaning is on the agenda. Just not so high up... *sigh*

Okay - so basically we can get you to 31 fps with a single tile and default visual range - that's not so bad. Multiple tiles *should* work with a similar number due to the buffering.

Btw.: Funny you would pick the edge of the Stratocumulus layer for your screenshot. I am actually unhappy about that edge modelling, because it has isolated single textures which doesn't look so good, and I've been thinking on how to improve that.

[HHS]

Hi,

yep, dynamical weather is on.

With dynamical weather off fps are much better and mostly at or near 30fps.

I waited until the console showed that clouds arrangement is finished.
When reset the clouds are still billboarding.

Yep, smells like the dynamics loop doing what it's supposed to do, i.e. realize that the cloud is in your visual field and move it with 0 kt (so you can't even actually see the motion...) - and doesn't take any resources when the cloud is outside the visual.

Problem- with the movements of the aircrafts there are always coming new clouds in view, so there are always impacts.

So indeed fps has been really improved to be useable unless you are using dynamical weather.

If we had moving clouds with the same fps- that would be great!

Regarding edges: we had the same problem with the first clouds set. A russian artist solved that- no idea how.

[Thorsten]

When reset the clouds are still billboarding

Yep - reset kills Nasal loops, not shader operation- billboarding is done by the shader, movement (currently) by Nasal (would probably better done by C++ - talking to the scenery through the property tree doesn't seem to efficient).

Regarding edges: we had the same problem with the first clouds set. A russian artist solved that- no idea how.

gimp, eraser tool and patience... It's no problem in principle, it just takes time to do. Some of the cloud textures are rather old - in the meantime, I am way better in extracting and have better images, so eventually I'll redo them.

[Hooray]

High-Pressure: unstable fps between 16-30fps, after resetting the whole sim: stable fps of 31
High-Pressure-Core: unstable fps between 11-18fps, after resetting the whole sim: stable fps of 31
High-Pressure-Border: unstable fps between 21-28fps, after resetting the whole sim: stable fps of 31
Low-Pressure-Border: unstable fps between 11-13fps, after resetting the whole sim: stable fps of 31
Low-Pressure: unstable fps between 12-13fps, after resetting the whole sim: stable fps of 31
Low-Pressure-Core: stable fps of 13fps, after resetting the whole sim: stable fps of 31
Cold-Sector: unstable fps between 23-29fps, after resetting the whole sim: stable fps of 31
Warmfront: unstable fps between 14-29fps, after resetting the whole sim: stable fps of 31
Tropical: unstable fps between 6-14fps, after resetting the whole sim: stable fps of 31

Gliders Sky: unstable fps between 15-24fps, after resetting the whole sim: stable fps of 31

That's actually a pretty good report with useful metrics, maybe we should think about using a Nasal script to directly create such a report, write it to the property tree and dump everything to an XML file using the helpers in io.nas? That way, we could really get useful benchmarks from many people.


PS: We really need to add those fantastic screen shots to the wiki, so that they can be reused in the newsletter and in the FG 2.0 gallery.

[VaLeo]

gimp, eraser tool and patience...

With that approach, you'll get non-natural feather edges.
You need clean photo of single cloud at blue sky background. And then turn blue color to transparency. That way I've done our present cloudset.
Btw, I began to make textures for your project since first version, but I can't keep up for you )

[Thorsten]

You need clean photo of single cloud at blue sky background. And then turn blue color to transparency. That way I've done our present cloudset.

You can do this up to a point (and in fact this is how I did it with the recent set of Cumulus cloud textures - although WooT claims I shouldn't do it and there is an even better technique based on overlaying the image on itself as a layer mask - which is completely beyond my abilities) - but there's no way you'll ever be able to pull this off with Stratus textures, because you don't ever get crisp Stratus cloudlets on blue sky background. And layered clouds are where my real problems and my worst textures are...

Btw, I began to make textures for your project since first version, but I can't keep up for you )

If you have any - by all means please send them - new cloud textures are easily brought into the system.

[Thorsten]

Here's the complete code of the function clear_all(). If anyone could look over it and tell me where it has the potential to crash Flightgear, I'd be obliged. It deletes a few property nodes (but only those it created before), it sets arrays to zero size and it sets other properties - this *should* be harmless enough, and indeed works fine for everything I tested - and yet it crashes HHSs system.

Without being able to reproduce the crash, it's beyond my skill to fix - so if anyone could help me out here, I'd be grateful.

Code: Select all

###########################################################
# clear all clouds and effects
###########################################################

var clear_all = func {

# clear the clouds and models

var cloudNode = props.globals.getNode(lw~"clouds", 1);
cloudNode.removeChildren("tile");

var modelNode = props.globals.getNode("models", 1).getChildren("model");

foreach (var m; modelNode)
   {
   var l = m.getNode("tile-index",1).getValue();
   if (l != nil)
      {
      m.remove();
      }
   }

cloudNode.getNode("cloud-number",1).setValue(0);

# clear effect volumes

props.globals.getNode("local-weather/effect-volumes", 1).removeChildren("effect-volume");

# clear weather stations

props.globals.getNode("local-weather/interpolation", 1).removeChildren("station");

# clear winds

props.globals.getNode("local-weather/interpolation", 1).removeChildren("wind");
setprop(lwi~"ipoint-number",0);

# reset pressure continuity

weather_tiles.last_pressure = 0.0;

# stop all loops make sure thermal generation is off

setprop(lw~"effect-loop-flag",0);
setprop(lw~"interpolation-loop-flag",0);
setprop(lw~"tile-loop-flag",0);
setprop(lw~"lift-loop-flag",0);
setprop(lw~"dynamics-loop-flag",0);
setprop(lw~"timing-loop-flag",0);
setprop(lw~"buffer-loop-flag",0);
setprop(lw~"housekeeping-loop-flag",0);
setprop(lw~"tmp/generate-thermal-lift-flag",0);

# also remove rain and snow effects

compat_layer.setRain(0.0);
compat_layer.setSnow(0.0);

# set placement indices to zero

setprop(lw~"clouds/placement-index",0);
setprop(lw~"clouds/model-placement-index",0);
setprop(lw~"effect-volumes/effect-placement-index",0);
setprop(lw~"effect-volumes/number",0);
setprop(lw~"tiles/tile-counter",0);


# remove any quadtrees and arrays

settimer ( func { setsize(weather_dynamics.cloudQuadtrees,0);},0.1); # to avoid error generation in this frame
setsize(effectVolumeArray,0);
n_effectVolumeArray = 0;

settimer ( func {
   setsize(weather_tile_management.modelArrays,0);
   setsize(weather_dynamics.tile_wind_direction,0);
   setsize(weather_dynamics.tile_wind_speed,0);
   setsize(weather_tile_management.cloudBufferArray,0);
   setsize(weather_tile_management.cloudSceneryArray,0);
   setprop(lw~"clouds/buffer-count",0);
   setprop(lw~"clouds/cloud-scenery-count",0);
   },1.1);

}


[WooT]

I have been pulled away from flightgear development for the last monthes, due to both family and work, and I first want to express my sincere apologies to you Thorsten for having let all the work by the side without notice. At some point I felt so guilty I didn't dare to give news.. I came across you last development news in this thread, and despite I didn't try it yet, it all seems to progress amazingly, and I hope I can find time soon to hop in the train again...

You need clean photo of single cloud at blue sky background. And then turn blue color to transparency. That way I've done our present cloudset.

You can do this up to a point (and in fact this is how I did it with the recent set of Cumulus cloud textures - although WooT claims I shouldn't do it and there is an even better technique based on overlaying the image on itself as a layer mask - which is completely beyond my abilities)

This being said, I would like to publish here what I sent to you in private, as an answer to the above quotes, regarding cumulus cloud texture extraction, if anyone is interested to try it on their source images.
The method is described for GIMP, but the principle is the same for photoshop, only tools names would change.

Please also note that these are only guidelines, based on the idea of working the pictures in a non destructive way, with as little hand work as possible to limit artificial stokes and "handwork print". I didn't have time enough to investigate the technique far enough for production quality results, but I feel this approach goes in the good direction when we need to extract information from such a diffuse and noisy signal that a cloud picture is, without bringing in hand made artifacts.

A typical cumulus cloud source image contains a lot of blue, either in the sky that we want totally removed , and in the cloud itself. We want to get rid of this, to create a black and white semi transparent texture.

Simply turning blue to transparency will loose a lot of fine detailing, specially in the thin feathery boundaries. Then main idea is to filter this blue component in order to get a very contrasted black and white solid picture, that we can then use as a transparency mask over a grey version of the cloud picture.

Here is how I worked so far :

1 ) denoise . This is done with the wavelet denoise plugin available
here : http://registry.gimp.org/node/4235
One interesting feature of this filter, is that you can denoise
the Luminance, Blue Chroma and Red Chroma separately, so with fine
tuning on each channel and using small threshold values, you can denoise
the picture with minimal detail loss.

2 ) Color removal, by Channel mixing. This step has two goals : make a
black and white image, and filter as much blue as possible to darken the
sky.
This filter is found in "Colors", "Components","Channel Mixer" .
You have to check the "monochromatic" checkbox, and use R, G and B
values to simulate a yellow/orange filter. I found that R 72 , V 28, B 0
works great.

( note : The best would be to use a physical red or orange filter when taking the picture - I ordered such a filter for my camera, second hand, back in the begining of the summer, but this was never delivered...joys of buying 2nd hand on the net... Channel mixing simulates this quite well, but we are loosing light in the process )

3 ) Compression of dark tones, this is done with the "Curves" tool. I
shift the black point to where the histogram begins to raise, and even a
bit more, so that the gray part of the sky becomes black, and I push
light grays and white a little to compensate. If you are not comfortable
with the curves tool , you can think of it as remapping the values over
the image, what we want here it that the gray pixels of the sky become
black, and that the lighter pixels in the cloud remain as they are.
Input is on the horizontal axis, output on the vertical. By shifting the
curve's origin to the right, we get rid of the darkest grays, and by
making the curve a bit convex, we readjust dynamic to preserve lighter
tones. This sounds tricky but its easy to play with this, as you can see
the preview in real time.

4 ) repeating 3 with other values on new layers if the background is not
really uniform. Often the blue of the sky ranges from dark at the top,
to lighter at the bottom. If I do step 3 based on the lighter blue zone,
this "eats" many details in the darkest zone. So I make step 3 on
several layers, cutting the image in bands from top to bottom, and I
re-assemble these layers with masks, so that I get a pitch black sky and
good cloud detail all over the image.

5 ) Some manual airbrush work may be needed to erase / correct unwanted
cloud parts. I always use the softest brush available, with very light
opacity. Here a tablet and pen are almost mandatory for clean work.

6 ) At this point we have a good black and white image. We can now use this black and white
image as a layer mask over a greyscale version of the the original image.We can also play on the mask levels , or even paint in the mask , to fine tune
transparency.


One of the problems I met with this approach, is that it leads to unwanted sheer areas in the cloud, where we would like to have opaque colors.
Painting on the mask is easy, and allows to quickly get rid of these sheer parts. The key is to use a very feathery brush, and avoid by all means to touch any boundaries, so we don't destroy all the hard work we made to preserve fine detailing on feather details.

I will try to post pictures to illustrate the process when I find time, and ... will try to work more on this at some point.


Thank you for the amazing work, and please accept my apologies again

[Thorsten]

And... the next milestone reached: I'm now proud owner of a piece of code in which the convective layer 'flows' over obstacles - the cloudbase rises to some degree when the drifting clouds reach a barrier, and then it falls off behind the barrier, and you can watch the individual clouds do so (if you have a fast enough computer).

Basically, this is a dynamical version of the convective altitude determination algorithm - periodically the code checks what the elevation beneath the cloud is, computes at which altitude it should be, given that elevation and sends it drifting there, at only small extra cost in computing time.

That's nice, because now the structure I need to cover the whole life-cycle of convective clouds is almost in place - the step that remains is to continuously monitor the terrain type beneath the cloud and to let it evolve, decay and respawn according to that.

[Hooray]

And... the next milestone reached: I'm now proud owner of a piece of code in which the convective layer 'flows' over obstacles - the cloudbase rises to some degree when the drifting clouds reach a barrier, and then it falls off behind the barrier, and you can watch the individual clouds do so (if you have a fast enough computer).

That sounds great, it seems this could also be useful for eventually visualizing and modeling wave lift or even foehn winds (LOWI).

[Thorsten]

That sounds great, it seems this could also be useful for eventually visualizing and modeling wave lift or even foehn winds (LOWI).

Yes and no.

The clouds of course 'know' only the terrain directly below. As time goes by, they would sample what you need for a wave, i.e. the terrain elevation along the wind direction. But that's too slow. So one could spend another 50-100 geodinfo(); calls to do some sampling along lines at startup to determine if a wave can be formed. Then find a model of how lift in the wave works and use that in a suitable placed effect volume. Then define a class of clouds and effect volumes which do *not* drift in the wind even if all others do (they could be assigned to a tile zero and sorted into a different array altogether which is probably fastest, or they could get a different flag).

In principle I guess I know how to do all of that. I'm just not sure if I will give it any priority in coding at the moment.

I've now tested soaring with the data moved to Nasal hashes. Thermals are also now integrated into the normal weather tiles, so no glider-specific tiles any more. It's a nice touch - I had started in relatively okay weather today, but it turned out I couldn't do my flight because I hit bad weather and incoming layered clouds - so I had to land. With default settings (dynamical weather on) I had 50+ fps in the ASK-13. Should make even HHS happy...

In addition, I've traced two small bugs in the thermal model - it's upper and lower cutoff is now smoothed as it should be (previously, the thermal model had a nice smooth fading out of the lift above the cloud base - but I cut that out because the effect volume cut right just above the cloud base - now fixed. The main problem now is that the temptation to go flying rather than coding is substantial...

[grtux]

Hi,

I don't know if it is the right place to tell it:

I have just tried the new menu with the Global weather.
I notice we have lost the real weather working.
Trying to get it working with the menu is right now IMPOSSIBLE, only the other scenarii are working.
OR only one way:
request it within FG loading --enable-real-weather-fetch (which is working without crash with, only, YASim Aircrafts, but that is an other story) .

Thanks

Gérard

BTW: i am getting a strange message on the console:
Warning: invalid line segment passed to IntersectVisitor::addLineSegment(..)
-nan -nan -nan -nan -nan -nan segment ignored..
geodinfo found
10000
Warning: invalid line segment passed to IntersectVisitor::addLineSegment(..)
-nan -nan -nan -nan -nan -nan segment ignored..
-geodinfo:Vector support unavailable
---> A patch is available at viewtopic.php?f=5&t=7358&st=0&sk=t&sd=a&start=90#p82805
features.geodinfo_supports_vectors=0
##########################################

I don't know is there is a relationship, with the issue