Board index FlightGear Development Scenery

Photoscenery generator for Julia compiler

Questions and discussion about enhancing and populating the FlightGear world.

Photoscenery generator for Julia compiler

Postby abassign » Mon Apr 19, 2021 2:46 pm

Image

Many users are asking me for clarification on how to create a photoscenery with simplicity. The procedures currently proposed are rather complicated and work well only on Linux machines, they use various types of programs that less experienced users are unlikely to understand. This is why I wanted to create a scenario generation procedure that was as automatic as possible and that would allow you to change the resolution of scenarios in specific areas (for example around an airport) according to the wishes of the users.
In these posts I insert the information necessary to use the procedure that is now available on this GIT under GPL2 license:

https://github.com/abassign/Photoscenary

Criteria for implementing the program

Analyzing the various problems of FGFS users who want to get a good photoscenary to be able to fly on a photo-realistic territory, I noticed that some were very recurring so much that I could list some features to be solved:

1. Use a single programming environment that is compatible for Linux, Windows and MAC.
2. Do not use script language that only work on certain types of shells, which is hardly acceptable to a user without high programming skills.
3. Be very light on the CPU level to be able to run on PCs that are doing other things, such as working or flying with FGFS.
4. Being able to resume an image download session without having to start over.
5. Handle the images that have not been properly uploaded.
6. Being able to change the resolution of images in certain areas (for example around an airport) to reduce the RAM required to take advantage of this feature.
7. Being able to define a certain area only with a pair of coordinates (for example an airport or a city) and a radius within which to cover with the photoscenary.
8. Ability to perform multiple downloads that increase the download speed of the scenarios from 3 to 10 times (depending on the actual speed of the network connection).

Use of the Julia programming environment

This is not the place to determine the use of a specific programming language, but the introduction of Julia among the languages present in the constellation of FGFS applications can be a good opportunity to rethink what can be done in 2021. Julia, like Java, is a completely platform independent language, this fact is essential to simplify the life of those who program, but also of those who use it. Julia is a language that has an executive permance equal to C ++ despite its syntax being more similar to that of a script-language such as Python. In Julia it is not necessary to define variable types (although recommended in projects that require maximum performance and guarantee a more precise debug), which is very useful when you need to produce prototypes or programs that are easy to modify, as is the case in this case.
Julia has other very interesting features, it is an imperative language, but also functional (in this case similar to NASAL), it allows to interface directly to almost all programming languages(C, C ++, Java, Java Script, Python etc) and produces a compiled very compact and "hidden" from the eyes of those who use it.
Finally ... Julia can directly use the GPU for some applications and OpenGL etc ...
In short, it can be a good investment for those who want to program and, observing its constant increase in popularity, they are very likely to think so.

Prerequisites

For all three environments (Linux, Windows and MAC) it is necessary to satisfy two prerequisites that can be easily downloaded at these two links:

ImageMagick https://imagemagick.org/index.php
It is absolutely necessary to convert the downloaded png file to dds type files which are used by FGFS for scenarios. It is also possible to use just png files, but if these are large, FGFS performance drops miserably and often overflows memory. For this reason I did not want to insert the option to use only png files, but I oblige the user to accept the automatic conversion of files from png (which are then deleted) to dds.
Imagemagick allows you to convert files transparently to the user very quickly.
In Linux distributions, Imagemagick is often already present, to find out just type the command "convert --version" and see if something responds. In windows it must be downloaded with an installer that can be downloaded here: https://imagemagick.org/script/download.php#windows
Imagemagick allows you to convert files transparently to the user very quickly.

Julia https://julialang.org/downloads/
The download link is this: https://julialang.org/downloads/
I strongly advise you to use the Julia 1.6 version as the compiler is much faster to compile than the 1.5 which is often present in Linux distributions. However, 1.5 works well with this program as there are no particularly advanced instructions.
Also for Julia download the program file and install, it is very simple and takes a few minutes of time.

The executable program
The executable program is called photoscenary.jl (jl is the typical extension of programs in Julia) and can be obtained at this link:
https://github.com/abassign/Photoscenary
Download the zip file with [code] (green button you see at the top right) and open it in any folder.

Execution of the program

The first time you have to run the setup.jl program is the program that loads all the Julia modules needed to run this application. The modules, once loaded, are placed in a hidden directory of Julia. If you uninstall Julia it is likely that you will need to rerun setup.jl. Maybe in the future I will make this procedure automatic, but for now I ask you to be patient. Once done it is not necessary to rerun it as the modules are stored on the hard disk.

Now everything is fine and you can run the program ...

Image

The program, for example, you can run it with this command in the terminal o command (in the windows system):

During the first execution of the program you will notice that there will be a phase of loading the system modules necessary for the execution of the program. This phase is automatic and can be repeated if the program undergoes a version change.

Then you can run the download program:

julia -t 10 photoscenary.jl --lat 45.66 --lon 9.7 -r 15 -s 3

where:

julia -t 10

--lat : the latitude of the central location of the area
--lon : the longitude of the central location of the area
-r : the radius of the area to be covered with the photoscenary in nm
-s : the resolution of the individual tiles (0 = 512x256 1 = 1024x512 2 = 2048x1024 3 = 4096x2048) Two new resolutions are planned in the future: 4 = 8192.4096 and 5 = 16384x8192
-p : the realtive path of photoscenary ()
That is the path after the home of your system, you must know that the program knows where the home is and therefore uses it as a starting base, if you do not enter the -p option the program will automatically use the default path which is for Linux (and MAC)
/home/user/fgfs-scenary/photoscenary
in Windows
c:\users\user\fgfs-scenary\photoscenary
if path is -p myfolder\scenary
in windows, for example will be:
c:\users\user\myfolder\scenary
If instead in Linux or MAC the path begins with the character / (slash) the path used will be only the one entered with this parameter. For windows the same rule applies, only in this case the path begins either with a letter followed by : or with the character \ (backslash).

The program has other options:

--latll --lonll : latitude and longitude of lower left corner of the area
--latur --lonur : latitude and longitude of upper right corner of the area
--over : Forces overwriting of scenario files. with a numerical value greater than or equal to 1. This option allows you to build variable size photoscenaries of the tiles, for example it is possible to create a low resolution photo scene in a large area, for example 100 nm, but then make a photo scene around an airport with a high resolution 10 nm radius
-d : with a numerical value greater than or equal to 1 the debug log is activate
--version : The program version is displayed with the next program stop. This option is handy for knowing if the entire Julia ecosystem has been properly loaded. Obviously the version number is essential information to request technical assistance.

Various notes

The program is currently evolving, and therefore I recommend checking if the GIT contains a new version as thanks to the notes that are coming to me I could immediately make some corrections necessary to improve the product.

The program currently uses only one source to download images: http://services.arcgisonline.com/
In the future I plan to add other sources, so I hope for everyone's help.

I hope these explanations are enough to get you started, if you have any problems or ask for clarification, write here that I try to answer. Obviously I hope to have others to give me a hand in the work, there would be many things still to do, such as directly integrating the program with FGFS.
Last edited by abassign on Tue Apr 20, 2021 9:59 pm, edited 4 times in total.
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenary generator for Julia compiler

Postby montagdude » Mon Apr 19, 2021 3:19 pm

Thanks for sharing. So what is the benefit of this tool compared to creator.py by Nathaniel Warner? And is "photoscenery" purposely misspelled?
montagdude
 
Posts: 255
Joined: Tue Dec 31, 2019 6:04 am

Re: Photoscenary generator for Julia compiler

Postby abassign » Mon Apr 19, 2021 10:47 pm

montagdude wrote in Mon Apr 19, 2021 3:19 pm:Thanks for sharing. So what is the benefit of this tool compared to creator.py by Nathaniel Warner? And is "photoscenery" purposely misspelled?


Thank you for your compliments, :D
However, the name of the program came about thanks to a typographical error that for me, who are not English speakers, was not intercepted and corrected. But honestly I like it that way, I think that this imperfection can bring luck to the program and therefore I gladly leave it as a perennial reminder that it is not necessary to write perfectly in English to write a good program. A lot of commercial product or program names are born this way, perhaps because in a certain language, certain names, they sound better.

For the benefits I listed them in the previous post, but if you missed them, I'll rewrite them:

1. Use a single programming environment that is compatible for Linux, Windows and MAC.
2. Do not use script language that only work on certain types of shells, which is hardly acceptable to a user without high programming skills.
3. Be very light on the CPU level to be able to run on PCs that are doing other things, such as working or flying with FGFS.
4. Being able to resume an image download session without having to start over.
5. Handle the images that have not been properly uploaded.
6. Being able to change the resolution of images in certain areas (for example around an airport) to reduce the RAM required to take advantage of this feature.
7. Being able to define a certain area only with a pair of coordinates (for example an airport or a city) and a radius within which to cover with the photoscenary.
8. Ability to perform multiple downloads that increase the download speed of the scenarios from 3 to 10 times (depending on the actual speed of the network connection).

From the requests, comments and thanks I received on Discord ... I would say that it has already been appreciated by many and I hope from you in the future, indeed when you use it let me know how it looks to you and if there are any additions or changes to be made.

Greetings
Adriano
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby montagdude » Tue Apr 20, 2021 2:51 am

Okay, sounds good. I'm sure it is a good tool. Personally, I don't understand your points #1 and #2 though, as Python meets those as well. But I don't mean to criticize; as with any open source tool, if it's useful and works well, it is worth using.
montagdude
 
Posts: 255
Joined: Tue Dec 31, 2019 6:04 am

Re: Photoscenery generator for Julia compiler

Postby portreekid » Tue Apr 20, 2021 5:28 am

I’m an architect and I have an addition to the list. Maintainability. As a developer you want to leave a legacy. Therefore bringing in a language not used in Flightgear is a hindrance. Take TerraMaster or OSM2City they have been around for ages and are not maintained by the original developer. Just a thought.
portreekid
 
Posts: 516
Joined: Tue Jan 14, 2014 3:36 pm
Location: Leipzig
Callsign: PORTREE
Version: 2020.2.1
OS: Windows 10

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 6:19 am

montagdude wrote in Tue Apr 20, 2021 2:51 am:Okay, sounds good. I'm sure it is a good tool. Personally, I don't understand your points #1 and #2 though, as Python meets those as well. But I don't mean to criticize; as with any open source tool, if it's useful and works well, it is worth using.


Ciao, I think I understand your perplexity, just as it is right that it is in the face of a novelty that modifies something that we previously thought immutable. The current script, to download a group of tiles, requires three different programming languages, Python 3,Perl, Bash shell, which makes it difficult to approach those who have Windows.
I noticed this by following for weeks various users of the Discord forum who wanted to get the image files.
For those who have windows the only solution is to install the Bash shell through the installation of a Linux distro that Microsoft allows through its Windows ecosystem.
Unfortunately, those who did the script did so in their own image and likeness, without thinking about usability by less experienced users.
Nothing bad, they are personal choices that I accept and consider stimulating within a complex ecosystem such as flightgear.
I think there's no problem proposing an alternative that simplifies some aspects, and for me, in my programming experience (I've been programming for 40 years) Julia was the most convenient way to get what I was interested in. So much for a user to install Python 3 or julia is exactly the same thing.
The difference of a name in return for getting a certain goal.
I could write the program in C++ but it had to be compiled for three different environments and those who use it cannot be sure that it is safe, an executable can contain everything, especially if written by a mysterious programmer..
So for this type of program the best solution is to use languages to compile on the fly, there is Go for example, but it is strongly web oriented, Julia Instead it is highly computational oriented and is making great progress with an absolutely excellent library of applications.
I could rewrite in Python 3 sure, I programmed for years in Python, but I find this script language absolutely confusing and difficult to maintain once you write code.
I could use Java, Java Script etc...
But in the end I used Julia as it turned out to be absolutely suitable for this task, produced a clear and compact code, and it was very easy to find libraries that solved the problems opened by my little adventure.
Finally, I also programmed a lot in NASAL and in the end I noticed that Julia is NASAL have many points in common, they are both imperative languages, but also functional, only that Julia is managed by a very large community of developers and allows you to operate easily in realtime through excellent solutions. In short, it could have good potential in Flightgear...
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 6:53 am

portreekid wrote in Tue Apr 20, 2021 5:28 am:I’m an architect and I have an addition to the list. Maintainability. As a developer you want to leave a legacy. Therefore bringing in a language not used in Flightgear is a hindrance. Take TerraMaster or OSM2City they have been around for ages and are not maintained by the original developer. Just a thought.


I understand your consideration, but here we are talking about a program that replaces three different programs made with three different programming languages to solve a problem that we do not know for how long it will be a problem.
In this case it is normal to use the fastest solution, so much in a year someone finds another better solution. This is how complex systems evolve and Flightgear is a system complex enough to evolve.
Anyway, I just wanted to draw an alternative in the programming model. The two programs you mentioned, one is written in Python 3 and the other in Java, neither language is officially present in Flightgear, so using a third language doesn't seem strange to me.
Julia, however, has a substantial difference, it is an extremely modern language that has a boundless amount of libraries and is followed by a large community of developers in university, science and engineering.
The performances are then incredible, the speed is on average similar to that of C++, but with the advantages of scripting languages, my program is multithreaded, that is, it allows you to run multiple download sessions simultaneously. The workload on my PC, with 15 sessions simultaneously downloading and converting to DSS, is on average 3% on a single CPU...
So let's see how it goes, in the meantime I invite you to try photoscenary is really great in FGFS.
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby portreekid » Tue Apr 20, 2021 7:12 am

I already use Photoscenery. Since I fly in Scotland 100% of the time I used the fguk lazy solution. ;-) Don't get me wrong regarding Julia. It's of course good to reduce the number of different languages used for one module. It would be sad if work like yours would get lost due to a language barrier. Keep up your work and share information far and wide.
portreekid
 
Posts: 516
Joined: Tue Jan 14, 2014 3:36 pm
Location: Leipzig
Callsign: PORTREE
Version: 2020.2.1
OS: Windows 10

Re: Photoscenery generator for Julia compiler

Postby StuartC » Tue Apr 20, 2021 7:35 am

good work, but working towards a GUI based scenery generator would be best, and probably easiest solution for many. An Interface like Terramaster, where you can simply click on the tiles you want etc.
StuartC
 
Posts: 2949
Joined: Fri Jun 18, 2010 8:18 pm
Location: Arse end of the Universe
Callsign: WF01
Version: 2019.1
OS: W10 64 bit

Re: Photoscenery generator for Julia compiler

Postby benih » Tue Apr 20, 2021 8:04 am

Or a simple map where you draw rectangles for the area of interest...
User avatar
benih
 
Posts: 563
Joined: Tue Aug 15, 2017 9:34 am
Callsign: D-EBHX

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 8:14 am

StuartC wrote in Tue Apr 20, 2021 7:35 am:good work, but working towards a GUI based scenery generator would be best, and probably easiest solution for many. An Interface like Terramaster, where you can simply click on the tiles you want etc.



In fact, the program already allows you to insert only the images that have not been inserted before, this allows you to easily pass to phase two of the project, that of integration in Fligtgear. My idea is to add the port and possibly the IP of the FGFS program and analyze some information of the property tree, such as the direction of flight, speed and altitude, to automatically define which tiles to download and transform into DDS.
My wish is that the program is transparent to the user, for example the parameters can be inserted in a persistent file (with Julia it is very easy to manage persistence) so it would avoid that every time the user has to tell the program what to do.
But there is an underlying problem that I ask you (and I also ask others who know the Orthophotos option):
In your opinion, when does FGFS see if there is an image tile to upload in a certain area? Does it do this on the fly or does it only do it at the start of loading or when it enters a certain area?
It would be interesting to understand this, otherwise the user loses the perception of what is happening.
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby wkitty42 » Tue Apr 20, 2021 5:57 pm

abassign wrote in Mon Apr 19, 2021 2:46 pm:Many users are asking me for clarification on how to create a photoscenery with simplicity.

you really should put this on the wiki if it isn't there already ;)
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 7094
Joined: Fri Feb 20, 2015 3:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 10:04 pm

wkitty42 wrote in Tue Apr 20, 2021 5:57 pm:
abassign wrote in Mon Apr 19, 2021 2:46 pm:Many users are asking me for clarification on how to create a photoscenery with simplicity.

you really should put this on the wiki if it isn't there already ;)


As soon as the main debug is finished I put it on the wiki, today for example I have inserted some changes that some users have requested, such as the absolute path and not linked to the user's home, this is particularly useful for those who use external storage units. There is always something to learn when putting a program on the net!
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Wed Apr 21, 2021 12:32 pm

A new version 0.1.1 of the program has been included in the GIT

Code: Select all
Photoscenery generator by Julia compilator,
Program for uploading Orthophotos files

<versioning>
    <version>0.1.1</version>
    <autor>Adriano Bassignana</autor>
    <year>2021</year>
    <licence>GPL 2</licence>
  </versioning>

Start the elaboration for 35 tiles the Area deg is latLL: 45.800 lonLL: 12.100 latUR: 46.400 lonUR: 13.000
The images path is: /home/abassign/fgfs-scenary/photoscenery/Orthophotos

Time elab:   7.6 Residual time to finish:  237.6 Elab. tiles: 35 residual tiles: 34 Thread: 5
...


A new version of the program has been included in the GIT, it has been updated by removing various defects, such as a problem with the --lat and --lon coordinates (those used with the -r value that is greater than 0.0 nm) they have values not multiples of 0.125 and this causes the reference to move with respect to the real one.

A second change was to manage the loading of julia system modules directly from the program itself, it is no longer necessary, the first time, to run setup.jl, the program will automatically manage the updating of Julia's system modules if necessary .

A third change was to allow the use of absolute paths and not just relative ones with respect to the home of the user running the program. Therefore a path becomes absolute when the last type: -p / home / username / xyz ... or if in windows: x: \ hjhk \ hkjhkj or \ jkj \ jkjk \ jkjk This seems to me a very convenient way to decide where to put the downloaded files.

A recommendation!
Do not use Julia version lower than 1.5.4 ...
Often the --over 1 option may be useful
This option allows you to rewrite the previously downloaded tiles, perhaps because you want to increase the resolution of the tiles (or reduce it if it is too high for your system), perhaps only in certain areas, avoiding having to reload everything.

One last recommendation, don't increase the julia -t xy parameter too much ... when you run the program. Running too many sessions could be a problem for the server that releases the image files to you and lead it to drop connections with a consequent lengthening of download times. In my opinion 4-6 is a very good value. For now I don't insert the block of the maximum number of sessions because I have to figure out which is the best limit for everyone. However, with an internet bandwidth of 40-60 mbit / s 6 is a very good value.

photoscenary.jl GIT
https://github.com/abassign/Photoscenary
abassign
 
Posts: 871
Joined: Mon Feb 27, 2012 5:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10


Return to Scenery

Who is online

Users browsing this forum: No registered users and 4 guests