Board index FlightGear Development Google Summer of Code

Proposal: a Nasal JIT LLVM front-end

Project proposals, student applications and all related discussions.

Proposal: a Nasal JIT LLVM front-end

Postby erik » Sat Feb 17, 2018 10:04 am

Yesterday I was reading this article about Intel using the LLVM framework for their graphics stack:
https://www.phoronix.com/scan.php?page= ... te-Runtime

This made me wondered how difficult it would be to create a Nasal front-end for LLVM so we could benefit from all the work by compiler professionals and still be able to use Nasal for out project (we're the only one I know of who uses Nasal).

This example seems to be a major step in creating a Nasal JIT without the need to do massive compiler backend work:
https://llvm.org/docs/tutorial/OCamlLangImpl1.html

Erik
Current: Parachutist, Paraglider, Pterosaur, Pilatus PC-9M and variants, ERCO Ercoupe, Fokker Dr.1, Fokker 50, Fokker 100
Less active: Cessna T-37, T-38, Santa Claus. Previous: General Dynamics F-16. Worked on: Wright Flyer
erik
 
Posts: 2244
Joined: Thu Nov 01, 2007 2:41 pm

Re: Proposal: a Nasal JIT LLVM front-end

Postby Hooray » Sat Feb 17, 2018 11:19 am

that would be possible, there was previously talk about this on the forum (as well as on the original Nasal mailing list).
A few years ago, Philosopher considered doing something along those lines, after writing the "Nasal internals" documentation: https://sourceforge.net/p/flightgear/fg ... format=raw

Realistically, something like this would probably be best bootstrapped by using a subet of the existing JavaScript LLVM front-end, and customize that as needed (func => function)

For FlightGear itself, a LLVM front-end is unlikely to matter much, because the main bottleneck when it comes to Nasal use in FG is not its maintenance status or lack of documentation, but the way it is integrated in the FlightGear main loop, and how its code is executed via timers and listeners (also running in the main loop) - which is why its GC overhead (garbage collection) does have an impact on frame rate/spacing.

A LLVM front-end would be primarily useful to build Nasal executable, i.e. run Nasal modules out of process - i.e. without running in the fgfs process/address space.

When it comes to Nasal performance and its impact on fgfs, the lowest hanging fruit is re-implementing props.nas using the Nasal cppbind framework - because property I/O is known to be the most significant factor affecting runtime performance. Next, it would make sense to revisit some of the GC related patches that ThorstenB and AndersG posted, and maybe explore integrating an existing incremental or concurrent GC library (a handful standalone GCs would not seem to be very difficult to integrate).

That being said, Nasal related additions have not been getting much attention from core developers unfortunately - even Philoshopher's changes took many months until they wer reviewed and committed.

Furthermore, even the patches that AndersG posted repeatedly were seeing a rather trepid response unfortunately - even though they could have certainly been made available as an experimental option that is configured during build time/startup

Given how things are unfolding in the addon department, it seems likely that, over time, more and more Nasal "modules" will become addons, and once those have well-established dependencies (inter-module, but also subsystem-level), it would not be that difficult to run those scripts in a Nasal background thread and let them interface with the rest of fgfs via some kind of messaging API (analogous to Richard's Emesary framework)


Subject: Nasal vs. Forth (offtopic)
Hooray wrote:In fact, back in the days of the Nasal mailing list, there used to be talk about doing something along those lines by targeting LLVM, so that Nasal code could be compiled into native binaries.



Subject: Intern'ing
Hooray wrote:
Philosopher wrote:Haha, my brother keeps asking if it's possible to compile Nasal...

The Nasal mailing list has been offline for several years and I can't seem to find an archive of it, but yes, back then, there's been talk about compiling Nasal IR via LLVM, Andy really liked the idea for some reason that eludes me ... I guess mfranz could elaborate on the details.

However, I find Nasal often painful to use for stuff that isn't trivial, especially when it comes to refactoring and shuffling code around, all the duck-typing makes it far too easy to introduce issues that go unnoticed for several iterations, until it's too late/too difficult to determine when a certain error got introduced.

As an aside, I also didn't particularly like Nasal when I got first in touch with it in FlightGear, and in fact I ended up implementing AngelScript next to it to convince myself just how bad Nasal really was. But I guess it's one of these "programmer vs. software engineer" debates - a programmer will always stick with what he knows, while a true software engineer will be open-minded and also explore new/niche stuff that isn't otherwise common/used. Obviously, Nasal knowledge isn't as "useful" as Perl, Python or even Lua knowledge - I think that's part of the reason why we are seeing so few C++ developers actually making use of it, it's a niche language and not really useful outside FG.

To be honest, fast forward a decade later, I have come to realize that I was certainly biased, and my thinking was colored by wanting to use popular/established tools/languages, so I didn't really understand very much about Nasal internals and technically Nasal in FG could even be considered superior in comparison to Python or Lua - i.e. due to an inherent thread-safe design right from the start. Unfortunately it's one of these things where Nasal is more future-proof than FG itself, so that these strengths cannot currently be easily explored, i.e. multi-threaded scripting in FlightGear still is a niche, too.

Making Nasal compile to native code would not be such an impossible undertaking, even though I highly doubt that it would be particularly useful :lol:

For scripting purposes in FlightGear, it has obviously served us well - arguably, a more popular/established language probably would have worked, too.
On the other hand, if it had been Python, Perl or Lua, that would have probably caused completely new issues, due to the sheer amount of bindings/libs available there - i.e. FG scripting would have probably gone overboard quickly, and people would have used all sorts of bindings and native code modules/libraries, and most things would no longer work out-of-the-box or be compatible in the first place.

So there are some advantages to having an "obscure niche language" like Nasal. Technically, it has all the right features that you would expect in a modern language, so I cannot find any reasons not to use it.

Now, scripting a flight simulator is obviously a challenge in and of itself, and over time I have really come to the conclusion that while having a dynamically-typed GC-managed language significantly lowers the barrier to entry for non-coders, it's causing lots issues sooner or later due to the sheer amount of code written by people who have a BASIC-approach to programming, which is unfortunate because it's reflecting badly upon Nasal and also FlightGear, due to performance issues (framerate, latency).

Thus, I believe that using a purely functional language with support for static and strong typing would have served the simulator better. This may seem weird, but we've sort of reached a critical mass here now, because core development is stalling in comparison to what's going on in the base package (see ohloh.net), so we're seeing more and more stuff being contributed in scripting space, where it's far too easy to bring the simulator down to its knees currently - without the Nasal engine itself being prepared to handle such things, not just due to GC issues, but also due to the sequential nature of 99% of our code.

We only need to look at the init mess, where declarative stuff is mixed with callbacks started via listeners and timers, or scripts running next to critical code that should ideally be running in a separate context. Overall, it's a challenge similar to the browser/JavaScript issue, where rogue JS code could halt a browser.

Not being able to inspect the Nasal VM at runtime is another issue, i.e. some way to identify routines that are causing (contributing to) performance issues would be great obviously.
Originally, Lua also didn't provide a solution to most of these problems, but the Lua community is just so huge that there's tons new stuff available each month.

Most of these concerns should no longer be showstoppers once the HLA effort materializes a little more, but based on lurking (and playing around with it a little), that seems like another 2-3 years away currently.

Then again, we do have a few people who are familiar with Nasal internals already, and adding new features (not just APIs) is no longer an obscure challenge. So it would just be a matter of coming up with a list of desirable features for "FG Scripting for 2014+" :lol:
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Proposal: a Nasal JIT LLVM front-end

Postby erik » Sat Feb 17, 2018 11:53 am

The main reason I think moving Nasal execution to LLVM would be a good thing is not to speeds things up (but JIT support is a great bonus) but because the Nasal code is basically unmaintained (and nobody dares to touch it). Creating it as a front-end to LLVM would be a relatively easy way to remove that burden from the project as far as I can judge.
Current: Parachutist, Paraglider, Pterosaur, Pilatus PC-9M and variants, ERCO Ercoupe, Fokker Dr.1, Fokker 50, Fokker 100
Less active: Cessna T-37, T-38, Santa Claus. Previous: General Dynamics F-16. Worked on: Wright Flyer
erik
 
Posts: 2244
Joined: Thu Nov 01, 2007 2:41 pm

Re: Proposal: a Nasal JIT LLVM front-end

Postby Hooray » Sat Feb 17, 2018 12:15 pm

In theory, yes. In practice, I doubt that Nasal is going to be particularly interesting compared to other LLVM front-ends - I mean, let's be real here: let's suppose I am looking for a scripting language with a LLVM back-end, why would I want to use Nasal over JavaScript/v8 (NodeJS) ?

Like you said, Nasal is a highly-niche language - it's really only used by a handful of projects, with FlightGear being the primary ones obviously.

Delegating maintenance work to a 3rd party sounds great in theory (which brings us back to the usual Python/Lua/JavaScript is better than Nasal" debate.
But my impression here is that this is the usual "adopt 3rd party library XYZ and benefit from that" approach (which seems to have worked fine for OSG, but less so for Qt5, and especially HLA/RTI). Maybe this is due to the realization that rewriting all of Nasal in fg isn't feasible ?

Speaking in general, it's not that nobody dares to touch it - over the years, a handful of folks have posted patches improving the Nasal VM/codegen - Philosopher came up with a bunch of nice additions, take a look at some of his branches at: https://gitorious.org/fg/philosophers-s ... nded-nasal

But it seems that core developers are rather hesitant to review/integrate and commit such patches, which obviously isn't uncommon at all (just look at YASim or other unmaintained parts of the codebase).

Then again, both James and TheTom have made a number of Nasal core changes over the years, especially during the time when the Canvas system got integrated.

So it's not that we're really lacking the expertise to do such things

I am not convinced that adding a LLVM frontend for a "toy-language" like Nasal is automatically delegating responsibility to another project - especially because Nasal has hardly anything to offer given what else is available in the LLVM department.

And from our standpoint, the primary benefit would be the ability to run code without having to recompile it, and without having to run it in-process of the fgfs binary.
Thus, the LLVM is only likely to matter to efforts wanting to deal with an existing Nasal codebase that they want to run outside the fgfs process - e.g. for people wanting to use existing FG systems implemented in Nasal (think Bombable addon, advanced weather etc) - possibly in conjunction with other flight simulators like X-Plane or FSX, or maybe even to create an integrated multiplayer server component that has combat support while providing an environmental simulation.

Anyway, with Nasal being open source, and its lexer being generally available, some of the core devs have been contemplating to write a Nasal2FOO translator, e.g. the idea of rewriting Nasal code to convert it to valid Lua/Python or JavaScript syntax. That would obviously be possible, and a quick way to get rid of Nasal.

However, personally I am not convinced that the outcome will be very popular from maintenance standpoint - especially when it comes to Lua and Python which both have a very different syntax. A semi-automated JavaScript "rewrite" would seem more likely to succeed given that both languages are based on ECMAScripts, i.e. Nasal code could be relatively easily converted to valid JavaScript without making a complete mess of it.

But for any of that to succeed, we'd need a rather thorough testing suite - especially because JavaScript's execution model is different compared to Nasal, so while the code may be syntactically correct, its behavior may still cause issues - especially because of the mainloop-based constraints imposed on the way such code is currently executed in fgfs.

And then there's bugman's idea of bringing FGPythonSys to FlightGear - personally, I am convinced that for any of these efforts to succeed, the first step is dealing with Nasal by providing an option to disable it entirely: http://wiki.flightgear.org/Howto:Disable_Nasal_entirely

And the next step would be incrementally initializing only some bits of it: http://wiki.flightgear.org/Initializing_Nasal_early

At that point, you'd end up with a fgfs binary that can run without Nasal, but also with only a tiny subset of Nasal code (think GUI/Canvas stuff), so that it would become much easier to experiment with alternate scripting solutions, without having to deal with a totally crippled fgfs binary.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Proposal: a Nasal JIT LLVM front-end

Postby erik » Sat Feb 17, 2018 12:44 pm

Hooray wrote in Sat Feb 17, 2018 12:15 pm:In theory, yes. In practice, I doubt that Nasal is going to be particularly interesting compared to other LLVM front-ends - I mean, let's be real here: let's suppose I am looking for a scripting language with a LLVM back-end, why would I want to use Nasal over JavaScript/v8 (NodeJS) ?

Because your project has already used it for arouund ten years or more and hundreds of sub-projects rely on in now.

Erik
Current: Parachutist, Paraglider, Pterosaur, Pilatus PC-9M and variants, ERCO Ercoupe, Fokker Dr.1, Fokker 50, Fokker 100
Less active: Cessna T-37, T-38, Santa Claus. Previous: General Dynamics F-16. Worked on: Wright Flyer
erik
 
Posts: 2244
Joined: Thu Nov 01, 2007 2:41 pm

Re: Proposal: a Nasal JIT LLVM front-end

Postby Hooray » Sat Feb 17, 2018 12:50 pm

That's a point in favor of using LLVM, but not one that will make much of a difference when being faced with two options like JavaScript vs. Nasal when starting a new project.
It would seem rather far-fetched that people would look for a LLVM-frontend like Nasal when they could use JavaScript, Lua or Python

Like you say, the existing legacy code base is an important factor - but not one that will have much of an impact on people already using fgfs.
We do have a "nasal-standalone" interpretere provided by Andy (github), and Philosopher came up with an improved version of that which links to SimGear (headless), which includes all the property tree stuff.

Thus, the real question is what the goal is here - our existing fgfs scripts can really only be executed out of the fgfs main loop if they become thread-safe, but they'd still need access to a ton of fgfs specific APIs - at least an interfacing mechanism like Emesary to hook up an external process to the fgfs process and call fgcommand to get/set properties etc.

PS: Andy actually liked the idea of Jonathan tinkering with a corresponding llvm front-end originally ...
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Proposal: a Nasal JIT LLVM front-end

Postby Richard » Sat Feb 17, 2018 11:06 pm

for me javascript falls into the "really should never have got out of the prototype phase"; there's so much wrong with the language. Nasal is a cleaner implementation of a basic script language, as is LUA. Before I got used to Nasal's foibles it was irritating me, but now that I've got used to it I realise that it actually does the job quite well. There are known 'bad' areas within Nasal such as GC, but switching to any other language is not going to fix this and may actually make it worse.

But to return to the original point; using LLVM might help, but it's not going to solve the main problem with Nasal; which is badly thought out logic that simply does to much in each frame. Nasal can be clean and efficient, just as any language can, but equally duff code can bring a system to its knees...
Richard
 
Posts: 810
Joined: Sun Nov 02, 2014 11:17 pm
Version: Git
OS: Win10

Re: Proposal: a Nasal JIT LLVM front-end

Postby Hooray » Sat Feb 17, 2018 11:36 pm

Point well taken, but I still think that given these two choices (Nasal vs. JavaScript LLVM front-end), you'd pick the one that doesn't sound like you need to get an ENT doctor involved to do any maintenance :D (and no, I am certainly not opposed to Nasal)

Actually, I believe that it was even Erik himself who originally came up with an in-sim implementation integrating JavaScript during the early PUI/PSL days (early 2000s), but then Andy/Nasal came around, ~2003 ...

Like you say, Nasal is doing a good job given its purpose - its primary issues are not language related, and concerning the GC it's actually a rather tiny, and self-contained, module. So it would not seem like much of a stretch to generalize the GC interface (a handful of C functions only) to provide a mechanism for tinkering with alternate GC integrations: http://wiki.flightgear.org/How_the_Nasal_GC_works

That being said, I don't fully subscribe to the notion that the GC is the actual problem here - many/most scripts don't even need to run in the main loop at all, so that the GC mechanism should not even matter if the whole interpreter/context runs out of the main loop, i.e. if Nasal scripts were to talk to the rest of the sim (main/rendering loop) via message passing primarily.

I totally agree that switching to another language, especially a more popular/established one, is likely going to make such issues even more prominent. Python, Lua and JavaScript are obviously great for a plethora of reasons, but the degree of 3rd party tooling, libs and documentation would also mean that the shortcomings in FlightGear's architecture would become very visible within a few months time.

The whole FGNasal/SGSubsystem-based integration on top of listeners/timers is literally begging for trouble - which is certainly also the main reason for most core developers encouraging people to re-implement their Nasal algorithms via property rules, or rewrite their Nasal code in C++ directly.

We did have the same discussion in the context of bugman's FGPythonSys experiments.

I don't quite see how LLVM is supposed to help with the real issues we're seeing in the FlightGear scripting department - if anything, it means faster Nasal code execution in the VM - but that won't solve the real problem. Like Richard said, dumb code can be written in any language - thus, a Nasal frontend for LLVM would at best create faster Nasal code, that would only hide the real problem better - which is bringing us back to the incidents surrounding the creation of the so called "Effects Factory", due to Torsten's findings that we had native (C++) code doing really dumb things at frame rate.

I am still not quite sure what the goals for a Nasal/LLVM frontend are - but I don't think it's going to have much of a positive impact on FlightGear scripting at all.
Let's just imagine, there'd be a working Nasal/LLVM front-end right now (should be easy enough to prototype), what would be the next step ? What do you think is going to happen realistically ?

Compared to the work that would be involved in fixing the GC or providing support for alternate GC schemes, it seems rather excessive to tinker with a new LLVM frontend.
Don't get me wrong, I do still believe it's an interesting discussion/idea - but I also think it's time for a pain/gain check. For instance, the Nasal GC module is roughly 400 LOC last I checked, and it's really self-contained too. There are meanwhile many 3rd party GC libs available that could be integrated for testing purposes.

Anyway, even that kind of work could be redundant if NEW Nasal scripts could be delegated to dedicated worker threads - which primarily means coming up with a safe versions for accessing simulator internals, via some kind of messaging API (not even talking about HLA/RTI).

My thinking here is that there is an increasing tendency among some folks to go all on on "totally rewrite X", without realizing how much work that comprises (working features vs potential regressions).

From a pain/gain standpoint, coming up with a Nasal/LLVM front-end is certainly going to take much time than integrating other GC schemes, or fixing the built-in one, even if that just means running the GC in a background thread (as per AndersG's patches). Likewise, coming up with a down-stripped version of FGNasalSys that runs its scripts in a background thread that only ever talks to the rest of fgfs via some MPI-like mechanism is going to provide more bang for the buck.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Proposal: a Nasal JIT LLVM front-end

Postby erik » Sun Feb 18, 2018 9:48 am

The main goal I would see is to separate the Nasal Lexer from the rest of the code so the scripting language would basically become: LLVM + a tiny bit of code to support the Nasal syntax.

By the way:
Chapter #8: Conclusion and other useful LLVM tidbits - This chapter wraps up the series by talking about potential ways to extend the language, but also includes a bunch of pointers to info about “special topics” like adding garbage collection support, exceptions, debugging, support for “spaghetti stacks”, and a bunch of other tips and tricks.

Erik
Current: Parachutist, Paraglider, Pterosaur, Pilatus PC-9M and variants, ERCO Ercoupe, Fokker Dr.1, Fokker 50, Fokker 100
Less active: Cessna T-37, T-38, Santa Claus. Previous: General Dynamics F-16. Worked on: Wright Flyer
erik
 
Posts: 2244
Joined: Thu Nov 01, 2007 2:41 pm

Re: Proposal: a Nasal JIT LLVM front-end

Postby Hooray » Sun Feb 18, 2018 12:14 pm

As far as I know, there is no GC support available in llvm at all - other than an interface for integrating an external GC implementation.
I also don't know how exactly LLVM would be integrated inside an existing application like fgfs - let's keep in mind that Nasal is used as an extension mechanism - thus, it would be interesting to look at examples/docs covering how the llvm codegen/JIT and VM can be added to an existing app like fgfs

I am not opposed to any of this, just wanting to make sure that we're on the same page regarding the pros & cons of this and our expectations ...

However, my impression still is that fixing up the existing gc. file (e.g. by integrating/improving AndersG's groundwork), or providing a new (additional) FGNasalSys subsystem that can run NEW scripts (e.g. certain addons) outside the main loop, would provide a better pain/gain ratio, because it would certainly be less work ...

Regarding gc.c in specifically, it's under 500 LOC, and it's plain C, too - i.e. no C++ or fancy STL containers. In other words, someone familiar with C can get up to scratch with the gc stuff relatively easily - and given the syntactic similarity between Nasal and C, that may even include folks who are primarily familiar with Nasal. Given the number of Nasal related patches we've seen over the years that were not added, my impression is that people are hesitant to get involved in efforts that are unlikely to end up being reviewed/integrated, and that may certainly also include a llvm based Nasal back-end, since it would obviously touch much more code than just coming up with a separate, and optional, gc implementation
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Proposal: a Nasal JIT LLVM front-end

Postby geed » Sun Feb 18, 2018 5:25 pm

Sound to me more like optimizing what Nasal is doing per frame and implementing what else could be done in a frame. (As per Richard)

Also where is the list of wanted/needed feature for the scripting host?

If features like that would be implemented what would be the acceptance criteria for adding them to the code base (given that a lot of additions where no accepted in the past)
geed
 
Posts: 89
Joined: Fri Apr 18, 2014 1:53 pm
Location: in between
Callsign: G-EED
Version: 2017.3.1
OS: OSX, Win8.1

Re: Proposal: a Nasal JIT LLVM front-end

Postby Hooray » Sun Feb 18, 2018 5:54 pm

Let's keep in mind that Nasal itself can already start a background thread and run outside the main loop - as long as you are only using Nasal library functions, i.e. none of the fgfs specific extension functions, which are known for not being threadsafe (they can hardly be made thread-safe in a reasonable/effective way given the current chaotic way of doing all sorts of weird stuff without formalizing and establishing runtime/data dependencies)

In other words, a new FGNasalSys interpreter (or context) could be set up, without all the fgfs specific stuff, and it would "just work" - obviously, it would not be particularly useful without being able to talk to the rest of fg via properties and fgcommands, or dedicated Nasal bindings (think Canvas).

setting properties or calling fgcommands or other APIs could obviously be handled by a simple IPC mechanism - heck, even Phi would work here (Nasal does already have http bindings which are also async), i.e. you could get/set properties and invoke fgcommands via Phi - and everything would "just work".

Subsystems with dedicated Nasal bindings (cppbind) are a different beast - I believe AndersG mentioned on several ocassions that his patch caused issues when dealing with Nasal C++ objects/handles (so called "ghosts") that were referencing Canvas pointers.

However, given the nature of the Canvas system, it would not be much of a stretch to use a dedicated private SGPropertNode (=property tree) per Canvas - that way, you could also do all sorts of property access in a background thread - at the mere cost of running the Nasal interpreter manipulating that property tree in the same SGThread, or by simply using a messaging passing approach again.

You'd end up with a Canvas subsystem holding its own private property tree and running its own Nasal interpreter in a background thread to update/animate it, without being constrained by the main loop - which would also map nicely to the way threaded OSG rendering works.

Realistically, it'd be a ton of work to port existing stuff over to such a new system - on the other hand, the new addon framework could certainly support certain addons designed to be executed in a background thread, without providing access to any of the standard extension functions (e.g. getprop/setprop & fgcommands)

Like I said previously, being able to disable all Nasal really is the first step, the next step is being able to incrementally bits of it, to only use a portion of the existing stuff, so that such schemes can be explored.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU


Return to Google Summer of Code

Who is online

Users browsing this forum: No registered users and 0 guests