Board index FlightGear Development Nasal

From io.include to io.import

Nasal is the scripting language of FlightGear.

From io.include to io.import

Postby galvedro » Fri Nov 28, 2014 6:14 pm

Hi!

A few months ago, Necolattis bumped into a limitation of io.include. io.include is completely dumb, it just loads and executes a script into the calling namespace. It takes care of not loading the same script twice in the same namespace, but nothing else. If the same file is included from independent namespaces, the file content is loaded twice, executed twice, and its symbols are instantiated twice.

This was the case when Necolattis started working on the GUI for the failure manager. As it happens, the FailureMgr "includes" the trigger library, and Necolattis' GUI did the same. When he tried to compare the triggers he "included" with the triggers that where already loaded by the system into the FailureMgr, they would never match. Why? Because they came from two independent instances of the trigger lib. I hope this makes sense :?.

I think this limitation can be confusing enough to justify some work in this area, so I have spent some time and come up with hopefully a better proposal: io.import.

There are only some small differences with respect to io.include, namely:
    - The target script is loaded into a "hidden" namespace, and then mirrored into the calling namespace.
    - Subsequent imports of the same module will detect that the module is already loaded, and will just mirror it where requested.
    - It accepts variadic arguments for specifying a smaller set of symbols to import (api borrowed from Andy R's own import).
    - The entire module hash is returned.

These changes allow for a variety of use cases:

Code: Select all
io.import("mymodule.nas", "*");

This is equivalent to io.include(), but addresses the instancing problem I described above.

Code: Select all
io.import("mymodule.nas", "one_symbol", "another_symbol");

This loads the module (if not already loaded), but makes only "one_symbol" and "another_symbol" visible in the calling namespace.

Code: Select all
var lib = io.import("mymodule.nas");

In this case, no symbol is imported into the local namespace. Instead, the module is accessible through "lib". Using io.import in this manner, the same behavior offered by io.load_nasal() can be emulated, for example, like this:

Code: Select all
globals.module = io.import("mymodule.nas");


It seems to work, what do you think?

(You can have a look at the code in my repo, here: https://gitorious.org/fg/galvedros-fgdata/source/2bb8d0fce6ddb3261c60361775fd08ab5fc5ed03:Nasal/io.nas)
galvedro
 
Posts: 145
Joined: Mon Oct 07, 2013 12:55 pm
Location: Sweden

Re: From io.include to io.import

Postby Hooray » Fri Nov 28, 2014 6:48 pm

I have spent some time and come up with hopefully a better proposal: io.import

FYI: The developer who came up with the Nasal engine originally (Andy Ross) also maintains a copy of his old Nasal code base on github, which includes a number of extension functions/bindings, and Nasal modules, that are not part of FlightGear, i.e. used/useful in unrelated projects (such as AlgoScore), but which includes a custom, and pretty nifty, import() implementation: https://github.com/andyross/nasal/blob/ ... r.nas#L163

Do keep in mind thought, that apart from FlightGear, Nasal isn't being maintained elsewhere, and even its original developer, meanwhile considers the FG/SG repository to be the primary development repositories - and if he should ever decide to continue Nasal hacking, he'll work off the FG/SG repo.

Besides, anything involving a proper io.import() implementation in the context of FlightGear is obviously overlapping with dependency resolution (currently, we're using a hard-coded workaround and the notion of "Nasal submodules" for this), as well as scripted bootstrapping - which is something a few of us are working towards, to get rid of hard-coded initialization restrictions, as well as making the simulator better configurable, and more Nasal modules optional (for better debugging/regression testing).


Subject: Modular Nasal bootstrapping (again)
Philosopher wrote:Continuing from http://flightgear.org/forums/viewtopic.php?f=30&t=19487, this isn't "fixed" but rather "dynamic" loading of modules.

I've been thinking that the best way to handle "modules" (loadable and unloadable bits of code that may have inter-module depencies) is to use the property tree. What this really allows for is requesting loading/unloading using the functionality of listeners. Loading is simply loading file(s) into a namespace, unloading potentially triggers listeners set during loading, and reloading - well, I have to decide on that one. There's a tree structure setup in property-tree://nasal/; /nasal/base-dir[0..n] are standard locations to search for Nasal files (currently $FG_ROOT/Nasal and $FG_HOME/Nasal, probably $FG_AIRCRAFT/Nasal, and maybe also $FG_NASAL and such). /nasal/base-dir has "file" and "dir" nodes which have "loaded", "load-requested", "unload-requested" nodes (bool), "path" (relative to parent), "absolute-path", and "namespace" nodes (string). dir can also contain dir and file nodes, recursively. Loading a dir involves loading all of its children; first files then directories (outside-in loading order, because I determined that makes most sense).

One thing this does is allow for circular dependencies that resolve really well. For instance, one can specify a list of symbols for the require() directive that must be present; if they aren't, the loading fails.

It also should allow for runlevels, early Nasal initialization (parsing command line arguments!! :D), and C++ code can simply set a property to request "all of the rest of Nasal be loaded".

I have an initial sketch of a "nasal_bootstrap.nas" file (untested, I don't have much access to FG these days):
Code: Select all
# Runs the caller in the desired namespace; returns 1 if it had
# to be called again, 0 if it was already in the namespace.
#
# Defining the namespace:
#   Method 1: names
#     run_in_namespace can accept multiple scalars specifying
#     a path from the global namespace, e.g. "io" (globals.io)
#     or "canvas", "svg" (globals.canvas.svg). If the namespace
#     does not exist, it is created.
#  Method 2: hash
#     It can also accept a direct hash for the namespace.
#
# Common usage:
#   run_this_in_namespace(...) and return;
#
#   if (run_this_in_namespace) return;
var run_this_in_namespace = func(ns...) {
   if (size(ns) == 1 and typeof(ns[0]) == 'hash') {
      ns = ns[0];
   } else {
      var namespace = globals;
      foreach (var n; ns) {
         if (typeof(namespace) == 'hash' and !contains(namespace, n))
            namespace = (namespace[n] = {});
         else namespace = namespace[n];
      }
      ns = namespace;
   }
   var c = caller(1);
   if (c[0] == ns) return 0;
   else {
      call(c[1], c[0]["arg"], c[0]["me"], ns);
      return 1;
   }
}

# Load a sub-module or file if not already loaded.
# @param module_path A sufficiently resolvable path pointing
#                    to a *.nas file or a directory of *.nas
#                    files and/or subdirectories.
# @param symbols A list of symbols, as per _require_symbols(),
#                to require the module to have once loaded or,
#                in the case of a circular dependency, once
#                recursively required again. An empty list just
#                requires that the whole file is loaded before
#                continuing. Nil says "I don't care".
var require = func(module, symbols=nil) {
   resource = find_resource(module_path);
   if (resource.getValue("loaded")) {
      if (symbols == nil) return;
      var namespace = globals[var ns = resource.getValue("namespace")];
      var sym = _require_symbols(namespace, symbol);
      if (sym != nil)
         die(sprintf("symbolic dependency failed: symbol '%s' not"
                     " found in module '%s' once loaded", sym, ns));
      return;
   }
   if (resource.getValue("load-requested")) {
      if (symbols == nil) return;
      var namespace = globals[var ns = resource.getValue("namespace")];
      if (typeof(symbols) == 'scalar')
         symbols = [symbols];
      elsif (!size(symbols))
         die("circular dependency failed: resource wasn't loaded");
      var sym = _require_symbols(namespace, symbol);
      if (sym != nil)
         die(sprintf("circular dependency failed: symbol '%s' not"
                     " found in module '%s'", sym, ns));
      return;
   }
   resource.setValue("load-requested", 1);
}

# Private; for each symbol in symbols, return it if it doesn't
# exist. Each symbol can be a scalar (first-level symbol) or
# a list of scalars (multi-level symbol, e.g. props.Node.getValues).
var _require_symbols = func(namespace, symbols) {
   if (typeof(symbols) == 'scalar')
      symbols = [symbols];
   foreach (var sym; symbols) {
      if (typeof(sym == 'scalar')
         if (!contains(namespace, sym))
            return sym;
      else {
         var current = namespace;
         foreach (var subsym; sym)
            if (!contains(current, subsym))
               return sym~"."~subsym;
            else current = current[subsym];
      )
   }
   return nil;
}

var nasal_dirs = nil;
var make_resources = func() {
   nasal_dirs = [ getprop("/sim/fg-root") ~ "/Nasal",
                  getprop("/sim/fg-home") ~ "/Nasal"
                ] ~ props.globals.getNode("sim").getChildren("fg-nasal");
   var NasalNode = props.globals.getNode("nasal");
   #NasalNode.removeChildren();
   var load_listener = func(n) {
      if (!n.getValue()) return;
      var parent = n.getParent();
      if (parent.getValue("loaded"))
         parent.setValue("unload-requested", 1);
      var name = parent.getName();
      if (name == "file")
         io.load_nasal(parent.getValue("absolute-path"),
                       parent.getValue("namespace"));
      else {
         if (name == "nasal")
            # Load "standard" directories before extra files:
            var resources = parent.getChildren("base-dir")
                           ~parent.getChildren("file");
         else
            # Outside-in loading order otherwise:
            var resources = parent.getChildren("file")
                           ~parent.getChildren("dir");
         foreach (var resource; resources)
            resource.setValue("load-requested", 1);
      }
      parent.setValue("loaded", 1);
      n.setValue(0);
   }
   var unload_listener = func(n) {
      if (!n.getValue()) return;
      var parent = n.getParent();
      if (!parent.getValue("loaded"))
      { n.setValue(0); return }
      parent.setValue("loaded", 0);
      settimer(func n.setValue(0), 0); #next frame, after all listeners have run, FIXME
   }
   var get_abs_path = func(node) {
      node.getParent().getValue("absolute-path")
      ~node.getValue("path");
   }
   var get_module_name = func(node) {
      var current = node.getParent();
      while (current.getParent().getName() != "base-dir")
         current = current.getParent();
      return current.getValue("path");
   )
   var init_module_part = func(node, path) {
      node.setStringValue("path", path);
      node.initNode("loaded", 0, "BOOL");
      setlistener(
         node.initNode("load-requested", 0, "BOOL"),
         load_listener);
      setlistener(
         node.initNode("unload-requested", 0, "BOOL"),
         unload_listener);
      if (node.getName() != "base-dir" and node.getName() != "nasal")
         node.setStringValue("absolute-path", get_abs_path(node));
      if (node.getName() != "nasal")
         node.setStringValue("namespace", get_module_name(node));
      return node;
   }
   var make_basedir = func(path) {
      if (path[0] != `/`)
         die("paths must be absolute");
      path = string.normpath(path);
      var dir = init_module_path(NasalNode.addChild("base-dir"), path);
      var subpaths = directory(path);
      foreach (var subp; subpaths) {
         make_child(dir, subp);
      }
   }
   var make_child = func(node, path) {
      var subpaths = directory(node.getValue("path") ~ path);
      if (subpaths == nil) {
         if (split(".", path)[-1] == "nas")
            init_module_part(node.addChild("file"), path);
      } elsif (size(subpaths)) {
         var dir = init_module_part(node.addChild("dir"), path);
         foreach (var subp; subpaths) {
            make_child(dir, subp);
         }
      }
   }

   init_module_part(NasalNode);
   foreach (var abspath; nasal_dirs)
      make_base_dir(abspath);
}

# A few rules on top of ordinary resolving (see http://wiki.flightgear.org/
# Resolving_Paths). Tries prepending Nasal/ and appending .nas.
#
# It is an error in the filesystem to have both ${resolvable_path}
# and ${resolvable_path}.nas.
var resolvenasal = func(path) {
   var p = path;
   var try = resolvepath(path);
   if (try != "") return try;
   if (split("/", path)[0] != "Nasal") {
      path = "Nasal/"~path;
      var try = resolvepath(path);
      if (try != "") return try;
   }
   if (split(".", path)[-1] != "nas") {
      path = path~".nas";
      var try = resolvepath(path);
      return try;
   }
}

var find_resource = func(path) {
   var p = path;
   path = resolvenasal(path);
   if (path == "") die("nasal path could not be resolved: "~p);
   var NasalNode = props.globals.getNode("nasal");
   var startswith = func(a,b) substr(a,0,size(b)) == b;
   var _find_resource = func(node) {
      foreach (var resource; node.getChildren("dir")~node.getChildren("file")) {
         if ((var abs = resource.getValue("absolute-path")) == path) return resource;
         elsif (startswith(abs, path)) {
            var res = _find_resource(resource);
            if (res != nil) return res;
         }
      }
      nil;
   }
   foreach (var basedir; NasalNode.getChildren("base-dir"))
      if (startswith(basedir.getValue("path"), path)) {
         var res = _find_resource(basedir);
         if (res != nil) return res;
      }
   var res = _find_resource(NasalNode);
   if (res != nil) return res;
   printlog("info", "Creating new nasal resource for path "~p);
   var resource = NasalNode.addChild("file");
   resource.setStringValue("path", p);
   resource.setStringValue("absolute-path", path);
   resource.initNode("loaded", 0, "BOOL");
   setlistener(
      node.initNode("load-requested", 0, "BOOL"),
      func(n) {
         if (!n.getValue()) return;
         var parent = n.getParent();
         if (parent.getValue("loaded"))
            parent.setValue("unload-requested", 1);
         io.load_nasal(parent.getValue("absolute-path"),
                       parent.getValue("namespace"));
         parent.setValue("loaded", 1);
         n.setValue(0);
      });
   node.initNode("unload-requested", 0, "BOOL");
   node.setStringValue("namespace", split(split(path, "/")[0], ".")[0]);
   return resource;
}
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11783
Joined: Tue Mar 25, 2008 8:40 am

Re: From io.include to io.import

Postby galvedro » Fri Nov 28, 2014 7:12 pm

Hi Horray,

I am not sure how to read your answer. Did you understand from my post that change I am describing is implemented here?: https://gitorious.org/fg/galvedros-fgdata/source/2bb8d0fce6ddb3261c60361775fd08ab5fc5ed03:Nasal/io.nas
galvedro
 
Posts: 145
Joined: Mon Oct 07, 2013 12:55 pm
Location: Sweden

Re: From io.include to io.import

Postby galvedro » Mon Dec 01, 2014 3:37 pm

I haven't provided much rationale behind why I think this change is important, besides "it fixes a problem in io.include", so I will try to elaborate a bit more.

There is no point in spending effort writing reusable code if the target language does not provide a reasonable mechanism for accessing shared functionality, and by reasonable I mean: well defined, clean, efficient and convenient. In my opinion, Nasal, as integrated by FG, is lacking such a mechanism.

There are two api functions covering this ground, as far as I am aware: io.load_nasal and my own io.include

io.load_nasal loads a Nasal file into a target namespace, if no namespace is provided, it will be inferred from the filename. It will only accepts namespaces hanging directly from the globals namespace. This has some problems:
    - It pollutes the globals namespace (not clean).
    - The same file can be loaded in different namespaces, resulting in several instances of the same script (not efficient).
    - There is no way to get direct access to the imported functionality in the local namespace (somewhat inconvenient).
    - Two modules with the same filename will clash if the target namespaces are not manually (and correctly) specified.
io.include loads a Nasal file into the calling namespace. Multiple inclusion and circular dependencies are managed, but:
    - There is no control over which symbols are loaded where: it's everything or nothing, which is bad for symbol shadowing.
    - A certain file can be loaded from different namespaces, resulting in several instances of the same script (not efficient and confusing).
The io.import function that I am proposing here addresses these problems:
    - The target file is loaded into a private namespace that is "hidden". This private namespace uses the file PATH as key, so it supports multiple files with equal names. By hiding this private namespaces, it doesn't pollute the globals namespace.
    - Because every file is loaded into a defined private namespace, they are only loaded once, regardless of how many times they are "imported" or from where.
    - Its interface gives the user options to specify which symbols are accessible where, but these symbols will be aliases of those contained in the private namespace. This gives the user a convenient way to access the shared functionality.
    - Circular dependencies are supported, as long as dependent scripts do not use each other before they are completely loaded (same as io.include and io.load_nasal I think).
Now Hooray, you mentioned "dependency management" and pointed to Philosopher's work in this area. I read that thread in the past and was aware of that work, even before I wrote io.include, actually. If I have to be honest, I don't fully understand the meat of that discussion, but my impression is that two different problems are being blended together:

- One is the problem I discussed above: how can Nasal handle access to shared functionality at a language level.
- The second one is how can a complex system like FG manage its own internal runtime dependencies. This is a completely different problem.

Part of the reason why I think these concepts are mixed is because in general, in FG, Nasal libraries are treated as runtime systems. A clear indication of this is that we don't have a defined location for Nasal libraries that are NOT to be loaded by default.

I hope this creates a bit more context around why I have spent some time on the problem, and hopefuly also makes some sense :D
galvedro
 
Posts: 145
Joined: Mon Oct 07, 2013 12:55 pm
Location: Sweden

Re: From io.include to io.import

Postby Hooray » Mon Dec 01, 2014 5:59 pm

thanks for the additional background/rationale info.
Like you said, these are two different things - but I do believe that they should be developed in conjuction, i.e. dependency resolution should be using io.import() and run-time dependencies should probably use some kind of listener-based interface. So the only thing I was suggesting is to keep related efforts in mind to ensure that the 2-3 "new" APIs will be compatible with each other, so that we can model more complex functionality on top of your io.import() helper.
Another increasingly important consideration, based on recent forum discussions, is having a way to gracefully suspend/restart a script, as well having a way to reload scripts from disk - e.g. for RAD purposes, so that FlightGear doesn't necessarily need to be exited/restarted in order to test a modified script.
If we could come up with a handful of APIs/frameworks to support these use-cases, the benefits would be self-evident, especially when compared to the current, chaotic, situation.
We're seeing an increasing amount of subsystems/features implemented in scripting space. but that's often done in a very roundabout fashion - and even if not, then, all the code will still be 100% custom - i.e. not generic.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11783
Joined: Tue Mar 25, 2008 8:40 am


Return to Nasal

Who is online

Users browsing this forum: No registered users and 0 guests