Board index FlightGear Development Scenery

Photoscenery generator for Julia compiler

Questions and discussion about enhancing and populating the FlightGear world.

Photoscenery generator for Julia compiler

Postby abassign » Mon Apr 19, 2021 3:46 pm

Image

Many users are asking me for clarification on how to create a photoscenery with simplicity. The procedures currently proposed are rather complicated and work well only on Linux machines, they use various types of programs that less experienced users are unlikely to understand. This is why I wanted to create a scenario generation procedure that was as automatic as possible and that would allow you to change the resolution of scenarios in specific areas (for example around an airport) according to the wishes of the users.
In these posts I insert the information necessary to use the procedure that is now available on this GIT under GPL2 license:

https://github.com/abassign/Photoscenary

Criteria for implementing the program

Analyzing the various problems of FGFS users who want to get a good photoscenary to be able to fly on a photo-realistic territory, I noticed that some were very recurring so much that I could list some features to be solved:

1. Use a single programming environment that is compatible for Linux, Windows and MAC.
2. Do not use script language that only work on certain types of shells, which is hardly acceptable to a user without high programming skills.
3. Be very light on the CPU level to be able to run on PCs that are doing other things, such as working or flying with FGFS.
4. Being able to resume an image download session without having to start over.
5. Handle the images that have not been properly uploaded.
6. Being able to change the resolution of images in certain areas (for example around an airport) to reduce the RAM required to take advantage of this feature.
7. Being able to define a certain area only with a pair of coordinates (for example an airport or a city) and a radius within which to cover with the photoscenary.
8. Ability to perform multiple downloads that increase the download speed of the scenarios from 3 to 10 times (depending on the actual speed of the network connection).

Use of the Julia programming environment

This is not the place to determine the use of a specific programming language, but the introduction of Julia among the languages present in the constellation of FGFS applications can be a good opportunity to rethink what can be done in 2021. Julia, like Java, is a completely platform independent language, this fact is essential to simplify the life of those who program, but also of those who use it. Julia is a language that has an executive permance equal to C ++ despite its syntax being more similar to that of a script-language such as Python. In Julia it is not necessary to define variable types (although recommended in projects that require maximum performance and guarantee a more precise debug), which is very useful when you need to produce prototypes or programs that are easy to modify, as is the case in this case.
Julia has other very interesting features, it is an imperative language, but also functional (in this case similar to NASAL), it allows to interface directly to almost all programming languages(C, C ++, Java, Java Script, Python etc) and produces a compiled very compact and "hidden" from the eyes of those who use it.
Finally ... Julia can directly use the GPU for some applications and OpenGL etc ...
In short, it can be a good investment for those who want to program and, observing its constant increase in popularity, they are very likely to think so.

Prerequisites

For all three environments (Linux, Windows and MAC) it is necessary to satisfy two prerequisites that can be easily downloaded at these two links:

ImageMagick https://imagemagick.org/index.php
It is absolutely necessary to convert the downloaded png file to dds type files which are used by FGFS for scenarios. It is also possible to use just png files, but if these are large, FGFS performance drops miserably and often overflows memory. For this reason I did not want to insert the option to use only png files, but I oblige the user to accept the automatic conversion of files from png (which are then deleted) to dds.
Imagemagick allows you to convert files transparently to the user very quickly.
In Linux distributions, Imagemagick is often already present, to find out just type the command "convert --version" and see if something responds. In windows it must be downloaded with an installer that can be downloaded here: https://imagemagick.org/script/download.php#windows
Imagemagick allows you to convert files transparently to the user very quickly.

Julia https://julialang.org/downloads/
The download link is this: https://julialang.org/downloads/
I strongly advise you to use the Julia 1.6 version as the compiler is much faster to compile than the 1.5 which is often present in Linux distributions. However, 1.5 works well with this program as there are no particularly advanced instructions.
Also for Julia download the program file and install, it is very simple and takes a few minutes of time.

The executable program
The executable program is called photoscenary.jl (jl is the typical extension of programs in Julia) and can be obtained at this link:
https://github.com/abassign/Photoscenary
Download the zip file with [code] (green button you see at the top right) and open it in any folder.

A complete program manual has been created on the FGFS wiki
https://wiki.flightgear.org/Julia_photo ... _generator
Last edited by abassign on Thu May 20, 2021 10:38 pm, edited 5 times in total.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenary generator for Julia compiler

Postby montagdude » Mon Apr 19, 2021 4:19 pm

Thanks for sharing. So what is the benefit of this tool compared to creator.py by Nathaniel Warner? And is "photoscenery" purposely misspelled?
montagdude
 
Posts: 266
Joined: Tue Dec 31, 2019 7:04 am

Re: Photoscenary generator for Julia compiler

Postby abassign » Mon Apr 19, 2021 11:47 pm

montagdude wrote in Mon Apr 19, 2021 4:19 pm:Thanks for sharing. So what is the benefit of this tool compared to creator.py by Nathaniel Warner? And is "photoscenery" purposely misspelled?


Thank you for your compliments, :D
However, the name of the program came about thanks to a typographical error that for me, who are not English speakers, was not intercepted and corrected. But honestly I like it that way, I think that this imperfection can bring luck to the program and therefore I gladly leave it as a perennial reminder that it is not necessary to write perfectly in English to write a good program. A lot of commercial product or program names are born this way, perhaps because in a certain language, certain names, they sound better.

For the benefits I listed them in the previous post, but if you missed them, I'll rewrite them:

1. Use a single programming environment that is compatible for Linux, Windows and MAC.
2. Do not use script language that only work on certain types of shells, which is hardly acceptable to a user without high programming skills.
3. Be very light on the CPU level to be able to run on PCs that are doing other things, such as working or flying with FGFS.
4. Being able to resume an image download session without having to start over.
5. Handle the images that have not been properly uploaded.
6. Being able to change the resolution of images in certain areas (for example around an airport) to reduce the RAM required to take advantage of this feature.
7. Being able to define a certain area only with a pair of coordinates (for example an airport or a city) and a radius within which to cover with the photoscenary.
8. Ability to perform multiple downloads that increase the download speed of the scenarios from 3 to 10 times (depending on the actual speed of the network connection).

From the requests, comments and thanks I received on Discord ... I would say that it has already been appreciated by many and I hope from you in the future, indeed when you use it let me know how it looks to you and if there are any additions or changes to be made.

Greetings
Adriano
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby montagdude » Tue Apr 20, 2021 3:51 am

Okay, sounds good. I'm sure it is a good tool. Personally, I don't understand your points #1 and #2 though, as Python meets those as well. But I don't mean to criticize; as with any open source tool, if it's useful and works well, it is worth using.
montagdude
 
Posts: 266
Joined: Tue Dec 31, 2019 7:04 am

Re: Photoscenery generator for Julia compiler

Postby portreekid » Tue Apr 20, 2021 6:28 am

I’m an architect and I have an addition to the list. Maintainability. As a developer you want to leave a legacy. Therefore bringing in a language not used in Flightgear is a hindrance. Take TerraMaster or OSM2City they have been around for ages and are not maintained by the original developer. Just a thought.
portreekid
 
Posts: 651
Joined: Tue Jan 14, 2014 4:36 pm
Location: Leipzig
Callsign: PORTREE
Version: 2020.2.1
OS: Windows 10

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 7:19 am

montagdude wrote in Tue Apr 20, 2021 3:51 am:Okay, sounds good. I'm sure it is a good tool. Personally, I don't understand your points #1 and #2 though, as Python meets those as well. But I don't mean to criticize; as with any open source tool, if it's useful and works well, it is worth using.


Ciao, I think I understand your perplexity, just as it is right that it is in the face of a novelty that modifies something that we previously thought immutable. The current script, to download a group of tiles, requires three different programming languages, Python 3,Perl, Bash shell, which makes it difficult to approach those who have Windows.
I noticed this by following for weeks various users of the Discord forum who wanted to get the image files.
For those who have windows the only solution is to install the Bash shell through the installation of a Linux distro that Microsoft allows through its Windows ecosystem.
Unfortunately, those who did the script did so in their own image and likeness, without thinking about usability by less experienced users.
Nothing bad, they are personal choices that I accept and consider stimulating within a complex ecosystem such as flightgear.
I think there's no problem proposing an alternative that simplifies some aspects, and for me, in my programming experience (I've been programming for 40 years) Julia was the most convenient way to get what I was interested in. So much for a user to install Python 3 or julia is exactly the same thing.
The difference of a name in return for getting a certain goal.
I could write the program in C++ but it had to be compiled for three different environments and those who use it cannot be sure that it is safe, an executable can contain everything, especially if written by a mysterious programmer..
So for this type of program the best solution is to use languages to compile on the fly, there is Go for example, but it is strongly web oriented, Julia Instead it is highly computational oriented and is making great progress with an absolutely excellent library of applications.
I could rewrite in Python 3 sure, I programmed for years in Python, but I find this script language absolutely confusing and difficult to maintain once you write code.
I could use Java, Java Script etc...
But in the end I used Julia as it turned out to be absolutely suitable for this task, produced a clear and compact code, and it was very easy to find libraries that solved the problems opened by my little adventure.
Finally, I also programmed a lot in NASAL and in the end I noticed that Julia is NASAL have many points in common, they are both imperative languages, but also functional, only that Julia is managed by a very large community of developers and allows you to operate easily in realtime through excellent solutions. In short, it could have good potential in Flightgear...
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 7:53 am

portreekid wrote in Tue Apr 20, 2021 6:28 am:I’m an architect and I have an addition to the list. Maintainability. As a developer you want to leave a legacy. Therefore bringing in a language not used in Flightgear is a hindrance. Take TerraMaster or OSM2City they have been around for ages and are not maintained by the original developer. Just a thought.


I understand your consideration, but here we are talking about a program that replaces three different programs made with three different programming languages to solve a problem that we do not know for how long it will be a problem.
In this case it is normal to use the fastest solution, so much in a year someone finds another better solution. This is how complex systems evolve and Flightgear is a system complex enough to evolve.
Anyway, I just wanted to draw an alternative in the programming model. The two programs you mentioned, one is written in Python 3 and the other in Java, neither language is officially present in Flightgear, so using a third language doesn't seem strange to me.
Julia, however, has a substantial difference, it is an extremely modern language that has a boundless amount of libraries and is followed by a large community of developers in university, science and engineering.
The performances are then incredible, the speed is on average similar to that of C++, but with the advantages of scripting languages, my program is multithreaded, that is, it allows you to run multiple download sessions simultaneously. The workload on my PC, with 15 sessions simultaneously downloading and converting to DSS, is on average 3% on a single CPU...
So let's see how it goes, in the meantime I invite you to try photoscenary is really great in FGFS.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby portreekid » Tue Apr 20, 2021 8:12 am

I already use Photoscenery. Since I fly in Scotland 100% of the time I used the fguk lazy solution. ;-) Don't get me wrong regarding Julia. It's of course good to reduce the number of different languages used for one module. It would be sad if work like yours would get lost due to a language barrier. Keep up your work and share information far and wide.
portreekid
 
Posts: 651
Joined: Tue Jan 14, 2014 4:36 pm
Location: Leipzig
Callsign: PORTREE
Version: 2020.2.1
OS: Windows 10

Re: Photoscenery generator for Julia compiler

Postby StuartC » Tue Apr 20, 2021 8:35 am

good work, but working towards a GUI based scenery generator would be best, and probably easiest solution for many. An Interface like Terramaster, where you can simply click on the tiles you want etc.
StuartC
 
Posts: 3175
Joined: Fri Jun 18, 2010 9:18 pm
Location: Arse end of the Universe
Callsign: WF01
Version: 2019.1
OS: W10 64 bit

Re: Photoscenery generator for Julia compiler

Postby benih » Tue Apr 20, 2021 9:04 am

Or a simple map where you draw rectangles for the area of interest...
User avatar
benih
 
Posts: 1689
Joined: Tue Aug 15, 2017 10:34 am
Callsign: D-EBHX
Version: next
OS: Debian Linux 64bit

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 9:14 am

StuartC wrote in Tue Apr 20, 2021 8:35 am:good work, but working towards a GUI based scenery generator would be best, and probably easiest solution for many. An Interface like Terramaster, where you can simply click on the tiles you want etc.



In fact, the program already allows you to insert only the images that have not been inserted before, this allows you to easily pass to phase two of the project, that of integration in Fligtgear. My idea is to add the port and possibly the IP of the FGFS program and analyze some information of the property tree, such as the direction of flight, speed and altitude, to automatically define which tiles to download and transform into DDS.
My wish is that the program is transparent to the user, for example the parameters can be inserted in a persistent file (with Julia it is very easy to manage persistence) so it would avoid that every time the user has to tell the program what to do.
But there is an underlying problem that I ask you (and I also ask others who know the Orthophotos option):
In your opinion, when does FGFS see if there is an image tile to upload in a certain area? Does it do this on the fly or does it only do it at the start of loading or when it enters a certain area?
It would be interesting to understand this, otherwise the user loses the perception of what is happening.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby wkitty42 » Tue Apr 20, 2021 6:57 pm

abassign wrote in Mon Apr 19, 2021 3:46 pm:Many users are asking me for clarification on how to create a photoscenery with simplicity.

you really should put this on the wiki if it isn't there already ;)
"You get more air close to the ground," said Angalo. "I read that in a book. You get lots of air low down, and not much when you go up."
"Why not?" said Gurder.
"Dunno. It's frightened of heights, I guess."
User avatar
wkitty42
 
Posts: 9148
Joined: Fri Feb 20, 2015 4:46 pm
Location: central NC, USA
Callsign: wk42
Version: git next
OS: Kubuntu 20.04

Re: Photoscenery generator for Julia compiler

Postby abassign » Tue Apr 20, 2021 11:04 pm

wkitty42 wrote in Tue Apr 20, 2021 6:57 pm:
abassign wrote in Mon Apr 19, 2021 3:46 pm:Many users are asking me for clarification on how to create a photoscenery with simplicity.

you really should put this on the wiki if it isn't there already ;)


As soon as the main debug is finished I put it on the wiki, today for example I have inserted some changes that some users have requested, such as the absolute path and not linked to the user's home, this is particularly useful for those who use external storage units. There is always something to learn when putting a program on the net!
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Wed Apr 21, 2021 1:32 pm

A new version 0.1.1 of the program has been included in the GIT

Code: Select all
Photoscenery generator by Julia compilator,
Program for uploading Orthophotos files

<versioning>
    <version>0.1.1</version>
    <autor>Adriano Bassignana</autor>
    <year>2021</year>
    <licence>GPL 2</licence>
  </versioning>

Start the elaboration for 35 tiles the Area deg is latLL: 45.800 lonLL: 12.100 latUR: 46.400 lonUR: 13.000
The images path is: /home/abassign/fgfs-scenary/photoscenery/Orthophotos

Time elab:   7.6 Residual time to finish:  237.6 Elab. tiles: 35 residual tiles: 34 Thread: 5
...


A new version of the program has been included in the GIT, it has been updated by removing various defects, such as a problem with the --lat and --lon coordinates (those used with the -r value that is greater than 0.0 nm) they have values not multiples of 0.125 and this causes the reference to move with respect to the real one.

A second change was to manage the loading of julia system modules directly from the program itself, it is no longer necessary, the first time, to run setup.jl, the program will automatically manage the updating of Julia's system modules if necessary .

A third change was to allow the use of absolute paths and not just relative ones with respect to the home of the user running the program. Therefore a path becomes absolute when the last type: -p / home / username / xyz ... or if in windows: x: \ hjhk \ hkjhkj or \ jkj \ jkjk \ jkjk This seems to me a very convenient way to decide where to put the downloaded files.

A recommendation!
Do not use Julia version lower than 1.5.4 ...
Often the --over 1 option may be useful
This option allows you to rewrite the previously downloaded tiles, perhaps because you want to increase the resolution of the tiles (or reduce it if it is too high for your system), perhaps only in certain areas, avoiding having to reload everything.

One last recommendation, don't increase the julia -t xy parameter too much ... when you run the program. Running too many sessions could be a problem for the server that releases the image files to you and lead it to drop connections with a consequent lengthening of download times. In my opinion 4-6 is a very good value. For now I don't insert the block of the maximum number of sessions because I have to figure out which is the best limit for everyone. However, with an internet bandwidth of 40-60 mbit / s 6 is a very good value.

photoscenary.jl GIT
https://github.com/abassign/Photoscenary
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Fri May 14, 2021 8:39 am

Finally released the 0.2 version of Photoscenary.jl, this version contains important news and corrections.

1. It is possible to download images (tiles) from 512 pixels to 32768 pixels (the maximum allowed by FGFS) in DDS format. The images are composed in the form of a mosaic using a matrix in memory.
2. The system downloads in parallel all the images necessary to compose a tile, the maximum number of simultaneous downloads is 64. This method allows you to compose an image of a 16K pixel tile (64MB in DDS format) in a time of 90 seconds.
3. The system automatically limits the maximum number of threads in order to limit the memory in use.
4. A windows 10 machine with 4GB of RAM can now download images up to 32K pixels (256MB in DDS format).
5. The low CPU load allows you to run the program at the same time as using other applications ... like FGFS.
6. The program no longer requires the use of an installation program, during the start of execution it checks that you have all the necessary modules, if these modules are not present, it downloads and compiles them, at the end of the compilation it closes. At the next execution it completes some functions and immediately begins to perform the assigned task.
7. The installation quality check of "ImageMagick" is always performed to avoid the execution problems complained by many users. If "ImageMagick" is not available, the program ends by releasing some tips to solve the problem.

Download speed has been accelerated by using a smaller image size (2048 pixels)

Essential fixes to be able to use the program across the globe:

1. The problem on the correctness of the tiles for negative longitudes (West) has been solved.
2. The problem on the correctness of the tiles for latitudes less than +/- 22 degrees and latitudes greater than 62 degrees has been solved.
3. Some rounding problems sometimes shifted the tiles with respect to the real position, now the problem is solved.

Some options have been added:

-i or --icao
The command is followed by a string of characters (if they have a space, the string must be placed between double quotes). This option allows you to select an airport as a center by ICAO code or by its name, as it has been entered in the FGFS database. the coordinates of the airport are used to obtain the values of --lat and --lon.

-t or --tile
You can enter a numeric value that corresponds to the index of a tile that you want to use as a reference for the center.

--attemps 0|1|2 ...
In some cases an image extraction server, for certain areas (for example the sea), could give an error in the download, in this case the system can try, for a certain number of times defined by an integer numeric value that can go from 0 to n. Normally this parameter is set to zero (no retries), a value greater than zero will lead to a certain number of retries.

--over 0 | 1 | 2
It is an option that defines the behavior of the program when there are already DDS images with the same tile ID.
0 - If the image is already present, the program does not download it and leaves what is there.
1 - The program evaluates the size of the image that has the same tile ID as the one that will be extracted from the image server, if the image already present is larger than the one to be downloaded, the program does not download the image, if instead it is smaller the program performs the download.
2 - The program replaces all the images already loaded.

--size 0 | 1 | 2 | 3 | 4 | 5 | 6
All these dimensions are active and correspond to:
0 -> 512 pix long side
1 -> 1024
2 -> 2048
3 -> 4096
4 -> 8192
5 -> 16384
6 -> 32768

The program can be downloaded in this GIT link:
https://github.com/abassign/Photoscenary.git

I hope this version will make it easier for everyone to use photoscenery and the visual quality that this method gives to FGFS flights.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Next

Return to Scenery

Who is online

Users browsing this forum: No registered users and 6 guests