Board index FlightGear Development Weather

A local weather system (v1.4 available)

Everything related to weather simulation, visuals should be discussed in the shader subforum.

Re: A local weather system

Postby Hooray » Tue Mar 23, 2010 10:06 am

Okay, so that's about 330 terrain samples per second run in the main thread, right?
Did you notice any signiifcant effect on the framerate? (for testing purposes, we might just as well disable most other subsystems or at least reduce scenery complexity to allow further development)

The graphs are neat, we might also want to take a look at using LOWI (Innsbruck airport) as a testbed for terrain sampling because it has interesting (read complex) terrain and is also known to have interesting weather patterns in real life as well (turbulence in the form of foehn winds, thermals, wave lift, ridge lift) and is also pretty nicely modeled in FlightGear by Tuxklok: viewtopic.php?f=5&t=5350

Several people here are using it for soaring scenarios and multiplayer gliding, so it is frequently discussed here, which might also make it easier to get more feedback.

And here are a couple of questions and considerations regarding the Nasal extensions required:

However, all this should probably not go into the main loop but into a worker thread...


That's true, and that would actually be another thing to keep in mind for the C++ code: ensure that the required Nasal/FG APIs are fully threadsafe, so that they can actually be called from worker threads once the need arises (there is some code in available in SimGear to help with this sort of stuff: http://simgear.org/doxygen/classSGMutex.html http://simgear.org/doxygen/classSGGuard.html )

terrain elevation: For thermal convection and barrier clouds, rapid sampling of terrain elevation is needed. Can there be a way to rapidly receive terrain elevation info only in Nasal for a whole set of pre-specified coordinates which performs better than geoinfo() calls?
http://wiki.flightgear.org/index.php/A_ ... 2B.2B_side


Depending on what it is exactly that is required, I think the solution that we discussed a couple of days ago (passing a vector with positions and getting one with normals) might be the easiest and fastest thing to do: viewtopic.php?f=5&p=70999#p70628

The question is how to sanely provide this info, i.e. from a format point of view: would it be sufficient to just pass in a vector of latitude/longitude pairs and get in turn just a vector that contains the computed normal for each tuple of GPS coordinates?

Like I said, this is the easiest approach and probably also the fastest. If on the other hand, we need to be able to lookup the normals by using the lat/lon tuple as a key for a map, we would need to return a nested hash instead.

Just using a Nasal vector would boil down to size(positions)/4 == size(normals) (with all normals in order).

This is straightforward to do, the code for the geoinfo API in Nasal (see NasalSys.cxx) looks like this (the Nasal internals are documented at http://wiki.flightgear.org/index.php/Ho ... tend_Nasal):
Code: Select all
// For given geodetic point return array with elevation, and a material data
// hash, or nil if there's no information available (tile not loaded). If
// information about the material isn't available, then nil is returned instead
// of the hash.
static naRef f_geodinfo(naContext c, naRef me, int argc, naRef* args)
{
#define HASHSET(s,l,n) naHash_set(matdata, naStr_fromdata(naNewString(c),s,l),n)
    if(argc < 2 || argc > 3)
        naRuntimeError(c, "geodinfo() expects 2 or 3 arguments: lat, lon [, maxalt]");
    double lat = naNumValue(args[0]).num;
    double lon = naNumValue(args[1]).num;
    double elev = argc == 3 ? naNumValue(args[2]).num : 10000;
    const SGMaterial *mat;
    SGGeod geod = SGGeod::fromDegM(lon, lat, elev);
    if(!globals->get_scenery()->get_elevation_m(geod, elev, &mat))
        return naNil();
    naRef vec = naNewVector(c);
    naVec_append(vec, naNum(elev));
    naRef matdata = naNil();
    if(mat) {
        matdata = naNewHash(c);
        naRef names = naNewVector(c);
        const vector<string> n = mat->get_names();
        for(unsigned int i=0; i<n.size(); i++)
            naVec_append(names, naStr_fromdata(naNewString(c),
                    const_cast<char*>(n[i].c_str()), n[i].size()));
        HASHSET("names", 5, names);
        HASHSET("solid", 5, naNum(mat->get_solid()));
        HASHSET("friction_factor", 15, naNum(mat->get_friction_factor()));
        HASHSET("rolling_friction", 16, naNum(mat->get_rolling_friction()));
        HASHSET("load_resistance", 15, naNum(mat->get_load_resistance()));
        HASHSET("bumpiness", 9, naNum(mat->get_bumpiness()));
        HASHSET("light_coverage", 14, naNum(mat->get_light_coverage()));
    }
    naVec_append(vec, matdata);
    return vec;
#undef HASHSET
}


So this is obviously doing some more things than just getting the terrain elevation and that would be another possible option for optimizing things, because most of this would not even be required for just computing normals.

Apart from that, it's just calling
Code: Select all
if(!globals->get_scenery()->get_elevation_m(geod, elev, &mat))


Which is defined in scenery.cxx:

Code: Select all
bool
FGScenery::get_elevation_m(const SGGeod& geod, double& alt,
                           const SGMaterial** material,
                           const osg::Node* butNotFrom)
{
  SGVec3d start = SGVec3d::fromGeod(geod);

  SGGeod geodEnd = geod;
  geodEnd.setElevationM(SGMiscd::min(geod.getElevationM() - 10, -10000));
  SGVec3d end = SGVec3d::fromGeod(geodEnd);
 
  osgUtil::IntersectVisitor intersectVisitor;
  intersectVisitor.setTraversalMask(SG_NODEMASK_TERRAIN_BIT);
  osg::ref_ptr<osg::LineSegment> lineSegment;
  lineSegment = new osg::LineSegment(toOsg(start), toOsg(end));
  intersectVisitor.addLineSegment(lineSegment.get());
  get_scene_graph()->accept(intersectVisitor);
  bool hits = false;
  if (intersectVisitor.hits()) {
    int nHits = intersectVisitor.getNumHits(lineSegment.get());
    alt = -SGLimitsd::max();
    for (int i = 0; i < nHits; ++i) {
      const osgUtil::Hit& hit
        = intersectVisitor.getHitList(lineSegment.get())[i];
      if (butNotFrom &&
          std::find(hit.getNodePath().begin(), hit.getNodePath().end(),
                    butNotFrom) != hit.getNodePath().end())
          continue;

      // We might need the double variant of the intersection point.
      // Thus we cannot use the float variant delivered by
      // hit.getWorldIntersectPoint() but we have to redo that with osg::Vec3d.
      osg::Vec3d point = hit.getLocalIntersectPoint();
      if (hit.getMatrix())
        point = point*(*hit.getMatrix());
      SGGeod geod = SGGeod::fromCart(toSG(point));
      double elevation = geod.getElevationM();
      if (alt < elevation) {
        alt = elevation;
        hits = true;
        if (material)
          *material = SGMaterialLib::findMaterial(hit.getGeode());
      }
    }
  }

  return hits;
}


This code could be further simplified, because we only need to compute normals (no material sampling required for the normals, right??) - to reduce the function call overhead in C++ space, we could just as well add another method that directly takes a pointer to lat/lon tuples:

double *
FGScenery::get_elevations_m_vector(size_t size, const SGGeod* geod, double* alt);

(I think, we should not gain anything by using the STL (std::vector) here because everything will be directly converted to Nasal structures which are C only?)

And returns a pointer to a list of computed normals, in the Nasal function this could then be converted to a Nasal vector and returned to the script. That way, we end up with only very few calls, even for large lists of positions.

If you think that the original functionality of the full geoinfo API should be retained, we could at least make it optional so that material info is only checked and provided when requested by the caller, that would keep things fast for situations where no material information is required.

PS: maybe we should quit discussing everything that comes to mind in one monster thread? There are now different things covered here already (weather modeling, textures, Nasal scripting, required extension functions), it might be better to ask a moderator to split this topic.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11267
Joined: Tue Mar 25, 2008 8:40 am

Re: A local weather system

Postby Thorsten » Wed Mar 24, 2010 10:47 am

Okay, so that's about 330 terrain samples per second run in the main thread, right?
Did you notice any signiifcant effect on the framerate? (for testing purposes, we might just as well disable most other subsystems or at least reduce scenery complexity to allow further development)


As of now, all tile setup calls are in for or while loops - which means that they block. The effect on the framerate is that when I do the terrain sampling, Flightgear does nothing but 3 seconds of sampling in a single frame, then runs normally afterwards. Which is why it should be accelerated dramatically and/or divorced from the main loop as soon as tile management is implemented :) Having 3 second delay on startup is quite okay, in flight it is not.

The graphs are neat, we might also want to take a look at using LOWI (Innsbruck airport) as a testbed for terrain sampling because it has interesting (read complex) terrain and is also known to have interesting weather patterns in real life as well (turbulence in the form of foehn winds, thermals, wave lift, ridge lift) and is also pretty nicely modeled in FlightGear by Tuxklok


Hm, that's mean... even more tricky than Zell am See - it's a very good testcase then. Has anyone ever flown in Innsbruck in reality and can tell me how a glider's day cloud distribution looks like? I guess a few photographs would do as well - I remember Innsbruck mostly with overcast skies or in winter...

The question is how to sanely provide this info, i.e. from a format point of view: would it be sufficient to just pass in a vector of latitude/longitude pairs and get in turn just a vector that contains the computed normal for each tuple of GPS coordinates?


First, I'd need one function for elevation only (some problems require that) and a distinct function for the surface normal. I would not modify the original geoinfo() function, but maybe introduce new geoelevation() and geonormal() calls.

For the geoelevation(), I'd imagine something like

elev = geoelevation(n, coords)

where n is the number of points to be sampled and coords is a vector of length 2n which is structured like coords = lat1, lon1, lat2, lon2, ...lat n, lon n) and elev is then (elev1, elev2, ...elev n).

For geonormal(), I'd thinl of

normal = geonormal(n, coords, scale)

where n and coords are as above, but scale is the size scale of the surface of which the normal is computed (obviously, calling with larger scales averages over small terrain features). So the function would generate 3 points spaced in a triangle around (lat1, lon1) with each distance scale (I think the surface is not well defined when one takes 4 points, so I have to correct myself), compute the normal of that surface and return a vector of length 3n like normal = (norm1x, norm1y,norm1z,...).

PS: maybe we should quit discussing everything that comes to mind in one monster thread? There are now different things covered here already (weather modeling, textures, Nasal scripting, required extension functions), it might be better to ask a moderator to split this topic.


Please do so - but I don't envy the mod for trying to pick everything apart 8)

In other matters - I have implemented and tested the effect volume system yesterday - it should support overlapping volumes and it does support nested volumes. I've just tested it for visibility and rain, I'll add turbulence and thermal lift today and combine it with a cloud model for a nice demo. As of now, I have to use workarounds to set the weather, for example, I have to set the cloud base of the first layer very high, because rain is by default only present below that layer. However, for effect volume definitions, no such prescription is needed, the volume definition should constrain where rain is found, so I'd need a 'dumb' switch to add rain and snow whenever I tell it instead of the 'smart' version /environment/metar/rain-norm I have to use right now. It should also switch on and off fast and without much transition. I'll add that to the Wiki request list.

So, the next release will set thermal lift already, although without a change in the C++ code it won't do anything...
Thorsten
 
Posts: 10102
Joined: Mon Nov 02, 2009 8:33 am

Re: A local weather system

Postby Thorsten » Fri Mar 26, 2010 7:17 am

Another update - I've managed to use a modified version of the 3dcloud shader to rotate a cloud model yesterday by declaring a suitable shader effect for the model. Since it's essentially the same shader that also does the 3d clouds, the model inherits a few nice features, such as the transparency of close layers or the fading with distance of the standard 3d clouds.

I'll make a large-scale performance test with many cloud models soonish - but if that works, I guess it might be a good step towards more clouds in the sky in the local weather system 8)

It's clear that the trafo isn't applicable to all weather phenomena, for example precipitation layers cannot be rolled upward, only rotated along the vertical axis.

I've said it early on, but if someone who understands shaders is reading this: I don't really know how that technology works, how coordinate systems are defined and what must be declared and what not, and most of what I do is simply try and error. If anyone can spare the time to explain to me a few basics, this would be *extremely* valuable.
Thorsten
 
Posts: 10102
Joined: Mon Nov 02, 2009 8:33 am

Re: A local weather system

Postby Mike4Linux » Fri Mar 26, 2010 4:47 pm

Thorsten wrote:Another update - I've managed to use a modified version of the 3dcloud shader to rotate a cloud model yesterday by declaring a suitable shader effect for the model. Since it's essentially the same shader that also does the 3d clouds, the model inherits a few nice features, such as the transparency of close layers or the fading with distance of the standard 3d clouds.

I'll make a large-scale performance test with many cloud models soonish - but if that works, I guess it might be a good step towards more clouds in the sky in the local weather system 8)

It's clear that the trafo isn't applicable to all weather phenomena, for example precipitation layers cannot be rolled upward, only rotated along the vertical axis.

I've said it early on, but if someone who understands shaders is reading this: I don't really know how that technology works, how coordinate systems are defined and what must be declared and what not, and most of what I do is simply try and error. If anyone can spare the time to explain to me a few basics, this would be *extremely* valuable.



Wow the 3D clouds look gorgeous, so this weather will simply be phantastic. Great
Mike4Linux
 
Posts: 80
Joined: Sat Mar 13, 2010 8:48 am

Re: A local weather system

Postby Thorsten » Sat Mar 27, 2010 4:05 pm

And... here we go - works nicely. Letting the shader rotate the clouds gives about an order of magnitude in performance - only beyond 4000 clouds did I get any serious deterioration of performance. Here are some pics from the tests:

A sky dotted with ~3500 Altocumulus in 5 distinct layers (not very realistic, but looks impressive) - seen from below

Image

and from between the layers:

Image

Having the possibility to mass-place cloudlets also allows to arrange them in layers - finally a viable solution for Nimbostratus clouds! Here are some first test results - a Nimbostratus layer seen from below

Image

and from above:

Image

This needs to be refined however - the problem is that a layer composed of cloudlets can realistically cover a 20 km x20 km area or so. That's not an issue from below the layer because the visibility is low anyway - but in principle from above, a Nimbostratus layer can be seen for potentially hundreds of km from cruise altitude of an airliner - so eventually there needs to be a solution that uses individual cloudlets in low altitude above the layer, fading to a single textured sheet from far above. I'm also not too happy with the shader trafo as it is implemented now - I will try to see if I understand enough to finally get the blend of position vector for large distance and view direction for short distance I prefer...
Thorsten
 
Posts: 10102
Joined: Mon Nov 02, 2009 8:33 am

Re: A local weather system

Postby erik » Wed Mar 31, 2010 7:53 am

To be honest I was a bit sceptical about weather effects using the AI subsystem but this is starting to work out fantastically. Nice work!
erik
 
Posts: 1505
Joined: Thu Nov 01, 2007 1:41 pm

Re: A local weather system

Postby Thorsten » Wed Mar 31, 2010 8:07 am

To be honest, it has left the AI subsystem a while ago 8) - most of the layer placement and environment management is now Nasal (the current count is 1500 lines of code...), and the CPU demanding low level cloud transformations run as shader effects. Still, all the cloud models could still be used in standard AI scenario definitions...
Thorsten
 
Posts: 10102
Joined: Mon Nov 02, 2009 8:33 am

Re: A local weather system

Postby erik » Wed Mar 31, 2010 8:59 am

If it's in Nasal now it would probably be better to move it to plain C++ somewhere in the future. That aside, it's pretty impressive.

Erik
erik
 
Posts: 1505
Joined: Thu Nov 01, 2007 1:41 pm

Re: A local weather system

Postby VaLeo » Wed Mar 31, 2010 7:58 pm

Nimbostratus from above - very good :!:
VaLeo
 
Posts: 186
Joined: Wed Nov 29, 2006 10:00 am
Location: Ukraine, Dnipropetrovsk
Version: GIT
OS: Debian 7

Re: A local weather system

Postby WooT » Fri Apr 02, 2010 1:37 am

oh wow, I've been away a few days, and been working on a glider for FG too ( viewtopic.php?f=4&t=7531 ) , and when I come back, I see you have been amazingly productive again ! Your nimbostratus layers look stunning !

So, the next release will set thermal lift already, although without a change in the C++ code it won't do anything...


What exactly would be needed ? Honnestly I can't wait to see your convective system connected to the fdm , and try it !

Something is sure : /environment/wind-from-down is the sum of ridge_lift and thermal_lift and this is the property we would need to send something to eat

in environment.cxx
Code: Select all
void
FGEnvironment::_recalc_updraft ()
{
  wind_from_down_fps = thermal_lift_fps + ridge_lift_fps ;
}


thermal_lift_fps is returned by the current AIthermal, and nothing else is using it.

How about if we modify environment.cxx so that AIthermal has its own property to write to, say AI_thermal_lift_fps ( so that if someone wants to use the old AI demo, its stays available ),
and your system is given a nice property to play with, say, local_thermal_lift_fps, we make sure that this property is accessible from nasal, and we make it so that the three values are summed into wind_from_down ?

I presume this wouldn't be a lot of work.
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 4:09 pm

Re: A local weather system

Postby Thorsten » Fri Apr 02, 2010 8:03 am

After some hard work, a new release of the system is due next week (essentially the code is finished, I just need to write proper documentation). The new release will have no weather dynamics yet (no time evolution of Cumulus clouds, no wind displacement...), and there are no long-range tile management routines, so everything works fine in a 40x40 km square, but beyond you have to erase and manually rebuild a weather tile. However, inside the tile most functionality is there - weather changes continuously between values at specified stations as you fly around, and effect volumes allow to control visibility, rain and snow. I haven't touched turbulence yet, as I believe that requires some kind of altitude model, and I'd like to discuss by what system that should be treated.

There is _in principle_ support for defining thermal lift in effect volumes, and there will be an example how a rain effect volume is associated with a cloud after which thermals can be coined. However, _in practice_ I can't set wind-from-below from Nasal as it get overwritten immediately - so as of now, the system just writes a property - which some C++ developer can read out. Since I can't test the functionality properly, as of now I haven't provided examples of a thermal.

With the release come several weather scenario tiles (all about 40x40km) - here is a picture gallery:

Fair weather, which is in essence a call of the convective system:

Image

Altocumulus sky - a few layers at 15.000 ft

Image

An overcast stratus sky (funnily enough, I learned how to do good layered clouds by a mistake - I unintentionally applied a shader designed for Nimbostratus to the Altocumulus models... - the results were magnificent).

Image

Unlike current layers, this one has an edge:

Image

An incoming rainfront with some rough weather and low visibility

Image

Some broken cloud layers:

Image

And finally, the Cirrus sky is back in action:

Image

Probably, thunderstorms will not yet be available - I seriously have to redo the Cb models, they look rather crappy at the moment.

Since the shaders do now most of the cloud management, the bad news is that you need shader effects to see anything. The good news is that performance is dramatically increased - despite placing O(1000-2000) clouds, I could fly all scenarios in the KSFO area with more than 30 fps (in fact, I could use as much as 5000 cloud models, but I've taken some care not to overdo it in the scenarios).

The whole system is more like an artists' toolkit - the weather tiles are in a sense just paintings made with this kit, but the real strength of the system comes when people use the toolkit to create their own tiles - and we collect those in some library. I will provide instructions and examples how it is done - essentially, it boils down to writing a few Nasal calls of high-level cloud placement routies (If anyone is a GUI wizard, maybe we can even get a GUI for that, but I simply don't see how to do it...).

By the way - if anyone knows how to make a menu option appear from Nasal, please let me know - I just tried directly setting the properties in /sim/menubar/default/ , but that did not seem to create anything, visible, although I can see the properties in the tree...
Thorsten
 
Posts: 10102
Joined: Mon Nov 02, 2009 8:33 am

Re: A local weather system

Postby ot-666 » Fri Apr 02, 2010 8:31 pm

stunning! great work.

cheers, oliver
Callsign: ot-666
Working on LOWI and other stuff - Custom Scenery Overlay Repo: http://gitorious.org/fgfs-custom-scenery/custom-scenery-overlay/
VMX22 - Osprey... sometimes in 2014
ot-666
 
Posts: 746
Joined: Sun Nov 08, 2009 5:14 pm
Location: Germany, Konstanz
Callsign: ot-666
IRC name: ot666
Version: GIT
OS: win7 64bit

Re: A local weather system

Postby f-ojac » Sat Apr 03, 2010 8:41 am

Very impressive. Congratulations, and keep it on. Definitely, the cloud visuals are one of the most eye-candy important feature for a flight sim.
--
If you want to support my Terrasync server, hosted on a private server, you can donate here: http://ns334561.ip-5-196-65.eu/WS2.0/WS ... 2.0.1.html
f-ojac
 
Posts: 1272
Joined: Fri Mar 07, 2008 9:50 am
Version: GIT
OS: GNU/Linux

Re: A local weather system (v0.5 available)

Postby Thorsten » Sat Apr 03, 2010 12:53 pm

New version available for download see first post - please let me know if

* all files are distributed (I'm having an increasingly hard time sorting the standard Flightgear files from obsolete test files and actually used files...)

* the package works correctly

* any other feedback

WooT, as far as your suggestion goes: The C++ code can for example read local-weather/current/thermal-lift - this is the place where the system stores all current weather info. That would make thermals (at least with constant lift) work immediately. If that works, I can Nasal-code you a complicated thermal with functional dependence of the lift on position for further testing. Or someone can let me have a CVS binary and I'll work from there... Or so - it shouldn't be difficult to support gliders from here, and then the glider community can fine-tune the convective system a bit.

PS: Love having the ASK-13 - that's the second Flightgear plane I've actually flown myself 8)
Thorsten
 
Posts: 10102
Joined: Mon Nov 02, 2009 8:33 am

Re: A local weather system (v0.5 available)

Postby WooT » Sat Apr 03, 2010 1:29 pm

A-MA-ZING !

I have just tried the new version, it is really good !
I don't see any significant frame rate drop anymore, everything keeps above very good numbers, whatever the tile I choose.

The various clouds look very good and convincing, with sometimes a minor downfall that can be easily fixed : some of the texture have some hard edges, making the texture planes appear. With your permission, Ican try if I can make them smoother / more feather like with gimp.
Both the cold front and stratus tiles look really good, either from above or from under, and the cirrus and altocumulus are just stunning !

This is a giant step in visual realism in my opinion, but also in flight dynamics, as your effect volume allows to simulate turbulence, sink, lift, whatever happens depending on the atmosphere configuration.

To answer your questions :

I have all the files in the archive

Everything works great my side, excepted a detail : when I first enable a rainy scenario, the precipitations do not show. I have to manually enable then disable " 3d clouds" to make it appear. Then, when clearing the tile, or changing to a non rainy situation, the precipitations remain, and I didn't find a way to stop them.

Another little thing could be to do so that when enabling a tile, FG 2D clouds layers are automatically cleared.

As for lift integration, I will try to make environment read your property right now. ( so far local-weather/current/thermal-lift doesn't exist, right ? )
This makes me think, there is lift in thermals, and we discussed how to simulate that based on gliding experience / documentation. But, I presume there is also a massive sink under rain conditions ? I have no experience in flying in such a situation, but it would be great to take advantage of your effect volume system to simulate that too. I see a possibility too, to simulate the horizontal winds that occur all around the base of a rain column under cumulonimbuses.
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 4:09 pm

PreviousNext

Return to Weather

Who is online

Users browsing this forum: No registered users and 3 guests