Board index FlightGear Development Weather

A local weather system (v1.4 available)

Everything related to weather simulation, visuals should be discussed in the shader subforum.

Re: A local weather system (v0.51 available)

Postby WooT » Tue Apr 06, 2010 12:34 am

the standard 3d clouds don't, because they use the same cloudlets everywhere, so they look always like developing clouds


I absolutely agree with that. They look great, but always like developing

So maybe we get multi-cloudlet models for soaring only and keep that a bit separate from the normal tile system and model library...


Do you think it would be possible to gradually fade into a multi-cloudlet model when getting very close, but keep the low-poly models for more distant clouds ?

I tried a 300 km triangle tonight, it was quite fun, unfortunately FG crashed at my 200 km...

Anyways, I noticed something weird with thermal-lift. Sometimes ( I'm not sure the pattern ), lift would not reset to zero, even long after I left the cloud area. I had to set it to zero again from property browser, then everything went back to normal. I didn't identify a way to reproduce it, but it happened like 3 or 4 times during the 200 km trip
( I cheated to have cumulus sky on such a distance, just by reseting clouds and calling another tile when I arrived at the border )
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 5:09 pm

Re: A local weather system (v0.51 available)

Postby Thorsten » Tue Apr 06, 2010 9:27 am

Do you think it would be possible to gradually fade into a multi-cloudlet model when getting very close, but keep the low-poly models for more distant clouds ?


You'd have to take some care to avoid that the cloud suddenly changes shape as you approach. One could use the same photographs though - from afar projected onto 4 layers, from close at higher resolution projected onto 20 cloudlets, suitably arranged. As you approach the cloud, it would not change much, but since the individual cloudlets would rotate around their center, as you circle the cloud it would look different from each angle and the madly spinning cloud problem would be gone. When you leave it, it would suddenly change shape when switching models - but since that happens in the rear, it wouldn't be a big issue. But that's a lot of work to get it right...

Anyways, I noticed something weird with thermal-lift. Sometimes ( I'm not sure the pattern ), lift would not reset to zero, even long after I left the cloud area. I had to set it to zero again from property browser, then everything went back to normal. I didn't identify a way to reproduce it, but it happened like 3 or 4 times during the 200 km trip


:( I really thought I had fixed that. I suppose that's caused by overlapping volumes somehow, but I don't quite see how, and I haven't seen it for a while.

If it occurs again, can you check the value of local-weather/effect-volumes/number-active-lift in the property tree? If it's zero, the system has realized it isn't in the thermal any more, but hasn't restored outside values properly. If it is nonzero, the system hasn't realized it left the thermal volume - that may help tracking it down.

Btw. - should the outside air (when not in a thermal) get some small fluctuation of lift, i.e. a small random positive or negative number? The interpolation loop runs every three seconds anyway, and can put such a small fluctuation in for no big price.

Also, could you send me a bit of (C++?) code for how to make the thermal more detailed (I can Nasal it from there)? For various complications, I'll deal with long-range tile management for the next release before tackling time evolution of weather, but I'd like to have a more detailed thermal model in anyway.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system (v0.51 available)

Postby WooT » Tue Apr 06, 2010 11:26 am

I'll try to check the active number.

should the outside air (when not in a thermal) get some small fluctuation of lift, i.e. a small random positive or negative number?


I would say that there are three situations when we could see that :

We could from time to time pass through a rising bubble, which is not associated to a thermal. On very active days, you would feel a lot of these, but maybe this would be better simulated by some short turbulence peaks.

Sometimes between thermals you can meet a large area of slowly sinking air. The sink speed would be very low in most cases, as the total energy is sprayed across a big surface.

Sometimes despite the air is charged with moisture, there can be a thermal without a cloud cap. But then this is different, there is actually a thermal, just no visual clues.


could you send me a bit of (C++?) code for how to make the thermal more detailed


I'll try with pseudo code. I'm afraid I will make too many errors in c++
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 5:09 pm

Re: A local weather system (v0.51 available)

Postby Thorsten » Tue Apr 06, 2010 11:44 am

That's a multi-cloudlet cloud of 36 cloudlets (looking familiar)?

Image

You can play with those yourself without any additional coding, the functionality of 0.51 can generate them easily.

* use the layer-placement call from the menu
* select small dimensions for the volume, say 0.4 km
* select edge = 1.0 to only place small cloudlets
* select high density, 100-150 or so
* stack several volumes of different size as needed

In this case, it's Altocumulus cloudlets. Visuals going circles below the cloud or through the cloud are nice, but it's almost 10 times more cloudlets than the normal Cumulus model.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system (v0.51 available)

Postby Torsten » Tue Apr 06, 2010 1:00 pm

Thorsten wrote:I guess pressure, temperature and humidity should be specified at the ground, and the atmosphere computed as a function of altitude from these (is that sufficient info?) - condensation level could be computed from those - that would help placing cloud layers automatically.

I guess for wind and turbulence it makes little sense to interpolate their high-altitude values from something like a METAR, so I'd compute them from a wind model inside the tile (in some situations, say a thunderstorm, the wind would be very different from the ground value) and would like to set them locally wherever the aircraft is.

I think visibility should be likewise be set wherever the aircraft is, and the tile visibility model specifies how it looks as a function of position and altitude.

Thermal lift seems to be solved nicely be WooT (see the script above) - I haven't tested it myself. If you could look into the others, I think that would be very helpful!


Here is a little trick to stop the environment controller from updating the environment settings. Call this once in the initialization phase of your script.
Code: Select all
props.globals.getNode("/environment/config/aloft").removeChildren();
props.globals.getNode("/environment/config/boundary").removeChildren();
fgcommand("reinit", props.Node.new({subsystem:"environment"}));

With empty aloft and boundary tables, the update method of the environment controller does nothing. The environment itself recalculates itself after setting a property so all values should be in a reasonable state after writing.
Just make sure, you don't open the weather-conditions dialog - weird things happen...
This is just a workaround for a better environment controller to come.

Torsten
User avatar
Torsten
 
Posts: 648
Joined: Fri Feb 01, 2008 10:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: A local weather system (v0.51 available)

Postby Thorsten » Wed Apr 07, 2010 9:48 am

I'm not sure I understand correctly:

Say visibility - if I try to set environment/visibility-m from the property browser or Nasal, it doesn't do because something overrides it, so I have to set it in config and call environment.reinit().

Are you telling me that the thing which prevents me from writing visibility is the update of environment, and if I block that by wiping its config table, I can set environment/visibility-m directly? That'd be neat... although I probably shouldn't distribute a version of the weather system that makes use of such drastic workarounds - user usually don't read instructions like 'but don't open the config menu dialog'...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system (v0.51 available)

Postby Torsten » Wed Apr 07, 2010 10:26 am

Are you telling me that the thing which prevents me from writing visibility is the update of environment, and if I block that by wiping its config table, I can set environment/visibility-m directly?

Exactly. Just make sure you leave the nodes aloft and boundary in place. Just wipe out everything below these nodes.
That'd be neat... although I probably shouldn't distribute a version of the weather system that makes use of such drastic workarounds - user usually don't read instructions like 'but don't open the config menu dialog'...

That's certainly true. You could disable the menuitem for the dialog, of course - but it's still an ugly hack, unusable for a production system.
If you are satisfied with the system one day, it might be a good idea to integrate it into the C++ code of the environment system.

Torsten
User avatar
Torsten
 
Posts: 648
Joined: Fri Feb 01, 2008 10:22 pm
Location: near Hamburg, Germany
Callsign: offline
Version: next
OS: Linux

Re: A local weather system (v0.51 available)

Postby WooT » Tue Apr 13, 2010 9:00 am

posting here some interesting links about how to -potentially- improve the graphics quality :

http://www.ofb.net/~niniane/clouds/
http://www.markmark.net/PDFs/RTClouds_HarrisEG2001.pdf
WooT
 
Posts: 92
Joined: Tue Mar 17, 2009 5:09 pm

Re: A local weather system (v0.51 available)

Postby HHS » Tue Apr 13, 2010 9:39 am

WooT wrote:posting here some interesting links about how to -potentially- improve the graphics quality :

http://www.ofb.net/~niniane/clouds/
http://www.markmark.net/PDFs/RTClouds_HarrisEG2001.pdf


Harris code was the start of realistic 3d-clouds in Games and sims. And FGFS was one of the first which used it! :D

Niniane clouds are based on Haris code, but was extented for MSFS.

Our current 3d-cloud system is a futher developement of all this codes and indeed more or less based on.
You might want to ask Stuart for more informations of current 3d-clouds!
Up, up and away
User avatar
HHS
 
Posts: 3625
Joined: Thu Jul 19, 2007 9:09 am
Version: GIT

Re: A local weather system (v0.51 available)

Postby Thorsten » Wed Apr 14, 2010 5:23 am

Guys, my main problem right now is that I hit a definite performance ceiling with layered clouds - I need to be able to load 4 tiles (to guarantee a decent visibility of clouds) and be above 20 fps. My problem at this point is not that I would run out of ideas how to improve a single cloud 8)

I spent the day going transatlantic (real, not in Flightgear), and looking out of the window was just an exercise in humility - I frequently saw four layers of different cloud types stacked in various combinations, visibe to the horizon at probably more than 300 km distance, looking absolutely fabulous, at the end even with great shading effects at sunset. I wish I could design clouds that well...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system (v0.51 available)

Postby Hooray » Wed Apr 21, 2010 10:25 pm

Wow, there's really lots to catch up with here, you have again been highly productive!
And it seems your code is now also available in CVS - congratulations! (also to WooT!!) :-)

Thorsten wrote:To be honest, it has left the AI subsystem a while ago 8) - most of the layer placement and environment management is now Nasal (the current count is 1500 lines of code...), and the CPU demanding low level cloud transformations run as shader effects. Still, all the cloud models could still be used in standard AI scenario definitions...


Using shaders to handle the cloud transformations was indeed a nice and clever move - this possibility should have occurred to us much earlier!! ;-)

If it's in Nasal now it would probably be better to move it to plain C++ somewhere in the future. That aside, it's pretty impressive.


While I do of course agree that certain things would probably be better done in C++ space, I also think that it's important to keep in mind how all this became possible in the first place: It is due to the flexibility of the Nasal and AI systems that all this could be done to this extent without touching any C++ at all.

So, it would be a pity to directly move everything into C++ space, because it is this very flexibility that also makes everything so configurable now. Porting everything to C++ space would also mean that we would have to sacrifice some of the flexibility and customizability that is provided by the use of Nasal/XML, so that changes/updates to this system would then need to be committed to CVS and FlightGear would need to be rebuilt in order to make changes take effect.

That would probably slow down development quite a bit, especially given that code is not easily discussed, reviewed and committed/maintained in FlightGear CVS on a regular basis.

That is something that most users and contributors here have noticed, which is probably also the reason for websites such as unitedfreeworld and others that provide a place for people to make their contributions available without necessarily requiring access to CVS.

Having on the other hand a system that does not necessarily depend on being maintained in CVS is a good thing because it also means less work for the developers.

Personally, I think it would be better to look into which parts of the script would benefit most from being run in C++ space, and then see how these could be added there while retaining the full flexibility of the Nasal based approach for the time being.

Several ideas to do this were already suggested a couple of weeks ago.
In the beginning, having dedicated Nasal helper functions that run in C++ space would be the most obvious way of doing this, without sacrificing any flexibility.

Porting the features over to C++ should definitely remain an important consideration, but not a top priority during the prototyping phase. I mentioned this already, but I think splitting the script into separate modules might make sense. That would help organize everything and provide a way to group functions logically.

For example, some stuff should probably be best run in C++ space, other code could be run in a dedicated Nasal worker thread, while yet other code is simply "library" code with helper routines that does not need to be updated as often as some of the other modules, and which could thus be considered "stable".

To be honest, I have already looked into the latest script, and even started re-organizing some chunks of code - however I didn't want to interfere with any of the latest developments (which would actually be another argument in support of modularizing the script, so that people could agree to work only on certain parts/components).

I will provide instructions and examples how it is done - essentially, it boils down to writing a few Nasal calls of high-level cloud placement routies (If anyone is a GUI wizard, maybe we can even get a GUI for that, but I simply don't see how to do it...).


If you can provide some details about the workflow that you envision, we might be able to come up with a working prototype for a GUI.

By the way - if anyone knows how to make a menu option appear from Nasal, please let me know - I just tried directly setting the properties in /sim/menubar/default/ , but that did not seem to create anything, visible, although I can see the properties in the tree...


I just looked into this, it seems the menubar properties are not using listeners but are instead read in once during initialization, all of this is to be found in menubar.cxx ( see FGMenuBar::make_menubar(void) ).

Which means that even if your properties are 100% valid, they will not be used because the corresponding code is only executed once during initialization (before you set these properties). Have you tried running reinit for the GUI (similar to your environment workaround)?

If that does not work, there are basically two ways to proceed here: 1) use listeners to allow menubar entries to be changed dynamically, or 2) use a new dedicated fgcommand to re-build the menubar on demand (seems much easier to implement to me).

As a crude workaround, you could just try to add the corresponding properties to the property tree BEFORE the GUI is built, that way your menu items will be present without needing a reinit.


The freeze for a certain period of time is (sort of) normal - the setup calls are all done in a single frame - which may take forever to execute. In tests, I had times ~minute freeze to set up 500 barrier clouds in a tricky situation. I'll soonish work on moving these calls to worker threads outside the simulation frames, I believe this will solve the freeze problem.


When using threads, keep in mind what I wrote a couple of weeks ago about NOT using any of the FlightGear APIs in a Nasal worker thread, you will not want to access any of the functions that modify state which is maintained in the fgfs main thread, because this requires explicit synchronization/locking or fgfs may simply segfault/crash due to race conditions (i.e. multiple threads invalidating their state without sync).

While making these APIs thread safe is not really complicated, it is simply not yet done so far - so that only those APIs that do not modify state are implicitly thread safe. This would probably be another requirement to keep in mind for the C++ changes required.

When I was playing with threads in FlightGear/Nasal a year or two ago, I had to recompile the binary to add some locking because we were looking into the update rates that can be realistically achieved with threads in Nasal.

Guys, my main problem right now is that I hit a definite performance ceiling with layered clouds - I need to be able to load 4 tiles (to guarantee a decent visibility of clouds) and be above 20 fps. My problem at this point is not that I would run out of ideas how to improve a single cloud 8)


So the performance problem is not due to use of Nasal now, but instead due to the amount of clouds that you have to render (i.e. the system is the bottleneck and not the approach)?

Can you load and display 4 weather tiles without updating them (i.e. just shown statically)?

If it is just due to the sheer volume/amount of clouds that you have to render, then we are back at looking into the data structures and algorithms that are used to determine which clouds are to be drawn. I assume currently, you simply update all objects in a loop?

I mentioned this already some time ago, but normally you'd do certain optimizations to prioritize the drawing of certain objects or even disable drawing altogether if they are simply not visible but this would normally require use of corresponding data structures (like octrees: http://en.wikipedia.org/wiki/Octree ) in the first place (e.g. occlusion culling for hidden surface removal: viewtopic.php?f=5&t=7358&p=70903#p70903 ).

This would basically mean to use spatial data structures in order to determine which objects are to be drawn (lots of fancy 3d maths), instead of drawing everything regardless of its visibility status.

So that is a very real option to optimize the performance of the whole thing. On the other hand, I am not sure if one should really look into doing such fancy things in Nasal space. While it is certainly possible, all the infrastructure for this is already provided by FlightGear/SimGear or OpenSceneGraph and it would seem sort of redundant to start coding spatial data structures in Nasal.

From our point of view, it would be far better (and easier) to simply tell FlightGear "update all objects" and it would internally decide what to update in C++ space.

An easier thing to do would be to start looking into using LOD animations for cloud objects, so that distant weather objects (clouds) use lower levels of detail, or where multiple separate objects are replaced by one simple object (i.e. cloudlets far away simply become fog/particles).
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system (v0.51 available)

Postby Thorsten » Thu Apr 22, 2010 4:42 pm

I've been travelling the last two weeks, so I didn't work too much on the clouds - hopefully (keep fingers crossed...) I'll be back home next week.

Done already for the next release:

* routines for automatic tile loading/unloading - realized in split loops (usually processing 25 clouds/frame), which means that performance is mostly okay unless cloud number per tile exceeds 1300 or so - since I've gone away from thread.newthread() structures, the implementation should be stable, and I did not see any crashes, but for very fast aircraft (=ufo) one can get a freeze when the system tries to load and unload tiles at the same time into the same array

* mid-level routines to draw 2/8, 4/8 and 6/8 layered clouds as cloud banks

* a few more cloud types

Curently being done:

* WooT cooked up a detailed model for a thermal - I'm halfway through implementing that

* more randomization of tiles and a different classification - currently I'm entertaining the idea of classifying by airmass ('low pressure system fringe', 'low pressure system core'...) and creating a random arrangement of cloud layers every time such a tile is called

* basic tile selection rules to allow transitions from one airmass to the other

All other ideas I currently have will probably have to go into the release after 0.6.

So the performance problem is not due to use of Nasal now, but instead due to the amount of clouds that you have to render (i.e. the system is the bottleneck and not the approach)?


I don't know, but I think so. The way the system creates clouds is by writing from Nasal into models/model[i] in the property tree. The observation is that if there are O(5000) clouds in that array, the system slows down substantially.

Initially I thought that this was due to the rotation matrix computations which the shader code has to do. However, now I don't think this can be completely true, because my understanding is that the shader doesn't actually run unless the model is in the field of view - thus performance should depend strongly on if the clouds are before me and visible or if I left them behind. But (by accident) I tested that leaving 7000 clouds 50 km behind where they are completely invisible still slows the system down.

I don't know if that is peculiar for having the models up in the models/model[i] space - after all, large numbers of trees for example seem to render well - I haven't started counting, but I guess at some point I had increased tree density to have a large number in the field of view.

So that is the main bottleneck. Currently that translates into a design guideline that none of my tiles should have much more than 1200 clouds and that the range to load tiles should be chosen such that no more than 4 tiles can be active at the same time. It would be interesting to see performance data from other computers though.

There is a secondary one having to do with removing models from the models/model[i] array - essentially I have to search the whole array in order to do that, because I must assume that other processes also want to write into there (the tanker.nas script and the ufo do for example). It would be neat to be able to spawn in subdirectories like models/tile[0]/model[i] - so one could load and unload models much easier. This actually works fine for loading the models into the scenery and removing the associated property nodes later from the tree when unloading the tile - the problem is that this doesn't delete the models themselves from the visible scenery. If that could be addressed from the C++ side, transitions for tile loading/unloading could be smoothed out further.

I guess that's a summary of the current status.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system (v0.51 available)

Postby Hooray » Thu Apr 22, 2010 6:08 pm

since I've gone away from thread.newthread() structures, the implementation should be stable, and I did not see any crashes


Actually, Nasal threads do work pretty well, even in FlightGear.
For the time being, we just have to make sure not to mix them with any of the FlightGear APIs that are not yet thread safe.

So I really don't think that Nasal threads are bad in general. I do think however that their use together with data structures that are stored/maintained in the property tree (fgfs main thread) is pretty unfortunate at best, because there is currently no synchronization taking place so one has to be extra careful to run only certain code in a worker thread, and e.g. only do property tree access in the main thread. Still, internal Nasal data structures should be perfectly fine to be used in a worker thread.

And like I said already, it may become questionable whether the property tree is a suitable "container" for storing spatial data structures. Native C++ (or even Nasal) structures may be more suitable for this.

I don't know, but I think so. The way the system creates clouds is by writing from Nasal into models/model[i] in the property tree. The observation is that if there are O(5000) clouds in that array, the system slows down substantially.


I see, another thing worth looking into might be the AI system itself, which may simply have become the bottleneck because it now has to deal with 5000+ objects? One could try to use static objects (without any transformations applied) that are NOT constantly updated and see if the AI system can handle those. If it does, then the AI system is not (yet) the problem. One could also see if not drawing the objects (but still updating) speeds up FlightGear, which would mean that drawing is becoming the bottleneck (or just completely disable the AI system while still running the local weather system, that would directly show if there is anything to be gained by optimizing the AI code).

Initially I thought that this was due to the rotation matrix computations which the shader code has to do. However, now I don't think this can be completely true, because my understanding is that the shader doesn't actually run unless the model is in the field of view - thus performance should depend strongly on if the clouds are before me and visible or if I left them behind. But (by accident) I tested that leaving 7000 clouds 50 km behind where they are completely invisible still slows the system down.


Do you also experience this slowdown when not applying any shaders/transformations at all?

I don't know if that is peculiar for having the models up in the models/model[i] space - after all, large numbers of trees for example seem to render well - I haven't started counting, but I guess at some point I had increased tree density to have a large number in the field of view.


I don't think that the trees are placed and updated using the property tree, also their position obviously is not updated at a high rate?

So another thing worth keeping in mind is that the property tree itself was probably never intended to be used for such purposes, i.e. updating 5000+ objects (each having a number of 10-20 sub nodes, I assume?) at a high rate.

To see if the property tree contributes to this bottleneck, we could just as well use a simple Nasal script and add 5000+ properties with a bunch of child nodes (plain property updating, nothing else) and see if/when it starts choking (framerate) or not, this would also give us some performance numbers to see what update rates can be realistically achieved with this approach of updating each object using the property tree.

Also, I mentioned this already last month: you are doing lots of explicit and redundant string building in your script, this is fine -and simple- for prototyping but all this string building in itself can cause quite a slowdown, too. You can check for yourself, just by doing a conventional setprop call with a static property path, and one that is dynamically assembled at runtime in an expensive loop, e.g.

Code: Select all
for (var i=0;i<=50000;i=i+1)
  setprop("/f"~"o"~"o",i);

versus:
Code: Select all
for (var i=0;i<=50000;i=i+1)
  setprop("/foo",i);


(Using systime() timestamps to benchmark the whole thing)

With regards to setprop calls in general, this is another thing that may slow down the simulator when run repeatedly, it is better to use a props node object instead which is sort of "pre-allocated". While setprop is more high level it is also more expensive than directly using a props node object. Re-using initialized props node objects is like using a "pointer cache", so all those hash lookups don't have to be done constantly. This is another thing that can be easily verified by using a small Nasal script to see if it makes any significant difference or not.

There is a secondary one having to do with removing models from the models/model[i] array


Do you mean the property tree or a Nasal array/vector here?

- essentially I have to search the whole array in order to do that, because I must assume that other processes also want to write into there (the tanker.nas script and the ufo do for example).


Maybe I am misunderstanding this, but I would imagine that doing some bookkeeping could help here, so that you know which properties were created by you, and which ones weren't - so you could keep track of those properties in a Nasal vector and only remove those properties that are in your vector??

It would be neat to be able to spawn in subdirectories like models/tile[0]/model[i] - so one could load and unload models much easier. This actually works fine for loading the models into the scenery and removing the associated property nodes later from the tree when unloading the tile - the problem is that this doesn't delete the models themselves from the visible scenery. If that could be addressed from the C++ side, transitions for tile loading/unloading could be smoothed out further.


Maybe you could clarify this? Because I'm probably really misunderstanding this now, as I think this problem would not exist by keeping track of those properties that you created?

In general, I think that it this issue of updating 5000+ cloud objects is pretty much one of those "embarrassingly parallel" problems that would be ideally handled in parallel because updating several thousand objects that do not have any mutual dependencies, should really not be done in a sequential fashion.

Thanks for the update!
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: A local weather system (v0.51 available)

Postby Thorsten » Thu Apr 22, 2010 8:04 pm

Still, internal Nasal data structures should be perfectly fine to be used in a worker thread.


That was my impression as well - but there's precious little which needs actual performance that I need to compute in Nasal only. The computationally expensive operations inside the Nasal script are

* repeated geoinfo() calls to get terrain elevation and coverage info
-> I can do those in split loops 25 a frame so they run without performance loss for the main loop - my impression is that they would be safe in worker threads, I had no problems when I tested them in dedicated threads, but I was warned that they may be unsafe

* updates of a large number model properties inside the property tree
-> those I can't do right now in worker threads, because the operation is unstable

One could try to use static objects (without any transformations applied) that are NOT constantly updated and see if the AI system can handle those.


That seems to be a misunderstanding of sorts. Let me try to explain this in detail:

Clouds are placed using the following bit of code:

Code: Select all
###########################################################
# place a single cloud
###########################################################

var create_cloud = func(type, path, lat, long, alt, heading, rain_flag) {

var tile_counter = getprop(lw~"tiles/tile-counter");

var n = props.globals.getNode("local-weather/clouds", 1);
var c = n.getChild("tile",tile_counter,1);

var cloud_number = n.getNode("placement-index").getValue();
      for (var i = cloud_number; 1; i += 1)
         if (c.getChild("cloud", i, 0) == nil)
            break;
   cl = c.getChild("cloud", i, 1);
   n.getNode("placement-index").setValue(i);


# var m = props.globals.getNode("models", 1).getChild("tile",tile_counter,1);
var model_number = n.getNode("model-placement-index").getValue();
var m = props.globals.getNode("models", 1);
      for (var i = model_number; 1; i += 1)
         if (m.getChild("model", i, 0) == nil)
            break;
   model = m.getChild("model", i, 1);
   n.getNode("model-placement-index").setValue(i);   


cl.getNode("type", 1).setValue(type);
var latN = cl.getNode("position/latitude-deg", 1); latN.setValue(lat);
var lonN = cl.getNode("position/longitude-deg", 1); lonN.setValue(long);
var altN = cl.getNode("position/altitude-ft", 1); altN.setValue(alt);
var hdgN = cl.getNode("orientation/true-heading-deg", 1); hdgN.setValue(heading);
var pitchN = cl.getNode("orientation/pitch-deg", 1); pitchN.setValue(0.0);
var rollN = cl.getNode("orientation/roll-deg", 1);rollN.setValue(0.0);

cl.getNode("tile-index",1).setValue(tile_counter);

model.getNode("path", 1).setValue(path);
model.getNode("legend", 1).setValue("Cloud");
model.getNode("latitude-deg-prop", 1).setValue(latN.getPath());
model.getNode("longitude-deg-prop", 1).setValue(lonN.getPath());
model.getNode("elevation-ft-prop", 1).setValue(altN.getPath());
model.getNode("heading-deg-prop", 1).setValue(hdgN.getPath());
model.getNode("pitch-deg-prop", 1).setValue(pitchN.getPath());
model.getNode("roll-deg-prop", 1).setValue(rollN.getPath());
model.getNode("tile-index",1).setValue(tile_counter);
model.getNode("load", 1).remove();

n.getNode("cloud-number").setValue(n.getNode("cloud-number").getValue()+1);

}


As you can see, this creates data structures in local-weather/clouds/tile[index]/cloud[n] which contain type, position and orientation of the cloud and structures in models/model[n] which are in my understanding necessary to make the model actually appear in the scenery, most of which are linked to the corresponding properties in local-weather/clouds/tile[index]/cloud[n].

The idea is that the system performs the following operations on them:

* spawn them to generate a weather tile (with index index)
* update their position according to wind drift (currently that is not done!)
* delete them to unload a weather tile

Once spawned, for the purpose of the AI system, the models are static - there is no update of any of their properties performed, simply because the rotation of the model to always face the viewer is done in the shader, so model pitch, roll and heading are never to be changed (which means that probably I needn't keep them in, but I didn't know that when I wrote that bit of code).

The position update is eventually to run on the local-weather/clouds/tile[index]/cloud[n] structure, because that can explicitly be referenced by tile index. Unloading this part of the structure is no problem, because it can be done for the whole directory by tile index. My expectation is that also updating cloud position poses no problem, as it can be done by a very slow loop over all clouds in the current tile.

The problem is spawning and unloading, because models/model[n] does not support subdirectory structures, and other processes can create in there as well, but I need to write precisely that vector to make any model visible.

What I do when spawning is to search for the next free position in the array and place the next cloud there. Each model gets the tile index as a property - so when removing what I do is the following:

Code: Select all
###################################
# tile removal call
###################################


var remove_tile_loop = func (index) {

var n = 25;

var flag_mod = 0;

var mvec = props.globals.getNode("models", 1).getChildren("model");
var msize = size(mvec);

var status = getprop(lw~"tmp/thread-status");

if ((status == "computing") or (status == "placing")) # the array is blocked
   {
   settimer( func {remove_tile_loop(index); },0); # try again next frame
   return;
   }
else if (status == "idle") # we initialize the loop
   {
   setprop(lw~"tmp/last-reading-pos-mod", msize);
   setprop(lw~"tmp/thread-status", "removing");
   }

var lastpos = getprop(lw~"tmp/last-reading-pos-mod");

# print("msize: ", msize, "lastpos ", lastpos);

if (lastpos < (msize-1)) {var istart = lastpos;} else {var istart = (msize-1);}

if (istart<0) {istart=0;}

var i_min = istart - n;
if (i_min < -1) {i_min =-1;}

for (var i = istart; i > i_min; i = i- 1)
      {
      m = mvec[i];
      if (m.getNode("legend").getValue() == "Cloud")
         {
         if (m.getNode("tile-index").getValue() == index)
            {
            m.remove();
            }
         }
      }

if (i<0) {flag_mod = 1;}


if (flag_mod == 0) {setprop(lw~"tmp/last-reading-pos-mod",i); }

if (flag_mod == 0) # we still have work to do
   {settimer( func {remove_tile_loop(index); },0);}
else
   {
   print("Tile deletion loop finished!");
   setprop(lw~"tmp/thread-status", "idle");
   setprop(lw~"clouds/placement-index",0);
   setprop(lw~"clouds/model-placement-index",0);
   }

}


Essentially I loop through the array, and remove the model for matching tile index. This creates a sparse array, so there may be gaps after removal, but the next spawning of new models will automatically fill any gaps.

It's immediately clear that the workload would be much reduced if I could use subdirectories in there like for the cloud position info, i.e. if I could remove simply by subdir - then I'd never have to search the whole array and not have to do any comparisons at all. But that doesn't work, as removing the corresponding nodes from the property tree does not delete models from the simulation.

I could of course maintain a Nasal list of all indexes into which I wrote the models for a given tile and remove by index - the problem is that sparse property tree arrays (i.e. when m[0], m[2], m[4] in the property tree with m[1] being nil) are mapped into dense Nasal arrays (so the above sparse array would become m[0],m[1],m[2] in Nasal) - so keeping lists doesn't actually work as well as one would like. And ultimately I need to write that property tree structure, because that's the way to make models visible in the scenery.

Do you also experience this slowdown when not applying any shaders/transformations at all?


Yes - to gauge the relative magnitude:

In empty skies with the ufo over the ocean I get 350 fps. Placing 3500 static cirrus models gives me 10 fps when the bulk of the models is in view and 50 fps when the clouds are not in the view field. So there is a sizeable performance loss associated with objects that are not in the field of view, and a different effect on top of that when the actual rendering needs to be done.

So another thing worth keeping in mind is that the property tree itself was probably never intended to be used for such purposes, i.e. updating 5000+ objects (each having a number of 10-20 sub nodes, I assume?) at a high rate.


In the current version, there is no high-rate updating of any properties in the tree happening. Unless a tile is loading or unloading, the Nasal is completely idle apart from monitoring loops every few seconds. So the Nasal structures cannot be responsible for what I'm observing, and optimizing the Nasal will not lead to better performance in flight, only to smoother loading/unloading of a tile.

Do you mean the property tree or a Nasal array/vector here?


That's the property tree structure.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: A local weather system (v0.51 available)

Postby Thorsten » Sun May 02, 2010 5:20 pm

... and WooT's thermal model is (without time dependence) coded - lacking a binary with the patch implemented, I can't *really* test-fly it, but at least when I measured it with the UFO, it seems fine.

Long-range weather changes also work fine - I had a very nice 1200 km trip with the UFO across the ocean, starting with overcast skies and rain, changing slowly to clear skies and Altocumuli, then worsening again, all the while with smooth changes in visibility, pressure and temperature going on - nice!

Now, just a little more work automatic generation of cloud banks and a few more weather tiles (the fun part) and the new release is ready!
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

PreviousNext

Return to Weather

Who is online

Users browsing this forum: No registered users and 1 guest