Board index FlightGear Development Scenery

Photoscenery generator for Julia compiler

Questions and discussion about enhancing and populating the FlightGear world.

Re: Photoscenery generator for Julia compiler

Postby Hooray » Fri May 14, 2021 8:55 am

You might want to consider using a dedicated wiki page for your project, and maybe add related news/updates to the corresponding newsletter.
Apart from that, I would suggest to reach out to the original developer of the photo scenery patches, so that he's aware of your work.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Photoscenery generator for Julia compiler

Postby merspieler » Fri May 14, 2021 12:09 pm

Download speed has been accelerated by using a smaller image size (2048 pixels)

I've found that 512px per tile gives the best and most reliable results... tho my script ran only single threaded (didn't bother with multi threading... I've just ran my script multiple times in parallel to increase overall speed)

-i or --icao
The command is followed by a string of characters (if they have a space, the string must be placed between double quotes). This option allows you to select an airport as a center by ICAO code or by its name, as it has been entered in the FGFS database. the coordinates of the airport are used to obtain the values of --lat and --lon.

How do you obtain the coords? The nav db?

--attemps 0|1|2 ...
In some cases an image extraction server, for certain areas (for example the sea), could give an error in the download, in this case the system can try, for a certain number of times defined by an integer numeric value that can go from 0 to n. Normally this parameter is set to zero (no retries), a value greater than zero will lead to a certain number of retries.

Depending on which information you get on an error, you can decide to retry or not.
For arcgisonline you get an HTTP 500 error code if the tile is not available. So no need to retry. For other errors, you can retry as specified.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Photoscenery generator for Julia compiler

Postby abassign » Fri May 14, 2021 9:35 pm

Hooray wrote in Fri May 14, 2021 8:55 am:You might want to consider using a dedicated wiki page for your project, and maybe add related news/updates to the corresponding newsletter.
Apart from that, I would suggest to reach out to the original developer of the photo scenery patches, so that he's aware of your work.


Hello,
I just put the manual in Wiki which should be quite clear, it is still to be completed, but currently it allows you to do almost everything that is allowed by the program.

https://wiki.flightgear.org/Photoscenery_generator
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Fri May 14, 2021 9:49 pm

merspieler wrote in Fri May 14, 2021 12:09 pm:
Download speed has been accelerated by using a smaller image size (2048 pixels)

I've found that 512px per tile gives the best and most reliable results... tho my script ran only single threaded (didn't bother with multi threading... I've just ran my script multiple times in parallel to increase overall speed)


Sorry if I answer only now, but today I wrote the manual on the wiki.
The value you indicate to me is not very fast for my type of program. If you use the resolution -s 0 (512 pixels) you will notice that the download time, proportionally, is longer. Obviously they are tests that I have done on my network and with my system, in the tests I have done with all the possible download resolutions (512,1024,2048,4096) I noticed that the most performing is 2048. However it is possible to vary this factor by editing the code here:

Line 829 approximately of the photoscenary.jl program

Code: Select all
if size <= 0
        sizeWidth = 512
        cols = 1
    elseif size == 1
        sizeWidth = 1024
        cols = 1
    elseif size == 2
        sizeWidth = 2048
        cols = 1
    elseif size == 3
        sizeWidth = 4096
        cols = 2
    elseif size == 4
        sizeWidth = 8192
        cols = 4
    elseif size == 5
        sizeWidth = 16384
        cols = 8
    elseif size> = 6
        sizeWidth = 32768
        cols = 8
    end


The value depends on the cols, if you want to try with images from 512 just rewrite in this form:

Code: Select all
if size <= 0
        sizeWidth = 512
        cols = 1
    elseif size == 1
        sizeWidth = 1024
        cols = 2
    elseif size == 2
        sizeWidth = 2048
        cols = 4
    elseif size == 3
        sizeWidth = 4096
        cols = 8
    elseif size == 4
        sizeWidth = 8192
        cols = 8
    elseif size == 5
        sizeWidth = 16384
        cols = 8
    elseif size> = 6
        sizeWidth = 32768
        cols = 8
    end


If you want to try it would be interesting to see how it goes on your system.
Note that I never exceed 8 for cols in order to avoid having to open too many download processes, however you can try with a maximum of 16 (256 images per tile).

Thi is the manual wiki link: https://wiki.flightgear.org/Photoscenery_generator
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Thu May 20, 2021 10:35 pm

I updated the photoscenary.jl program with version 0.2.5 which contains an important new feature:
I have introduced a database of all ICAO airports (over 60,000 airports) which can be selected as a central point of reference for the creation of the photo-scenario.
This feature is documented here:
https://wiki.flightgear.org/Julia_photoscenery_generator#Let.27s_go_to_Fiumicino_.28Rome_in_Italy.29_by_-i_option

A complete program manual has been created on the FGFS wiki
https://wiki.flightgear.org/Julia_photoscenery_generator

The program can be downloaded from this link:
https://github.com/abassign/Photoscenary
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby Volador » Fri May 21, 2021 11:27 am

abassign, this is a wonderful and detailed piece of work. I am currently using the method provided by nathaniel515 described here
nathaniel515 wrote in Mon Sep 21, 2020 10:29 pm:I recently started working on patches that add photoscenery support to modern FlightGear. I have instructions here: https://github.com/nathanielwarner/flightgear-photoscenery

Let me know if it works, and if you have suggestions! Right now, you have to provide satellite orthophotos manually, but I'm working on automating this.


Is there a plan to integrate this function into FG in the future? A graphical interface (or look ahead) would be brilliant -drag selecting the coordinates to download, I realise that for licensing reasons individuals may have to download to their own systems, and further to that would put too heavy a demand on the satellite photo services. Are there further thoughts on a legal way to use a repository on other servers for the dds files? (if that's how it works, I'm not a programmer).
User avatar
Volador
 
Posts: 1140
Joined: Tue Sep 01, 2020 4:58 pm
Callsign: Volador, G-VLDR
Version: 2020.4
OS: Windows 10, 64 bit

Re: Photoscenery generator for Julia compiler

Postby merspieler » Fri May 21, 2021 4:33 pm

The issue is the license...
I could spin up a server with global low-res ortho scenery in a matter of seconds (making it terrasync compatible would only require generating the .dirindex files... that might take a bit longer).
But the license prohibits this.

Also there would be another issue with hosting... we had issues finding hosts for osm2city and get over and over in trouble maintaining mirrors for terrasync, each of which require less than 100GB of storage.
Global, lowres (roughly 4m/px) does require >5TB.
And then imagine the bandwidth, that would require.
For osm2city and terrasync I've got currently around 3.3TB data volume a month (tendency is going up)...
The bandwidth would increase about as much the disk space: 50x... so you can expect some 165TB a month.
Also the max bandwidth available to the server has to increase. Most of the time my 200MBit/s uplink is just fine but at peak times, users can't pull at their full speed... we'd need at least hosts with 1GBit/s internet connections.

So global scenery is absulutely not feasible right now.
If we could get an agreement with an image provider for a limited area, that might work. Still... we need to find one, and convince them
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Photoscenery generator for Julia compiler

Postby Hooray » Mon May 24, 2021 9:17 am

@abassign: using the route manager to pre-fetch tiles/imagery is a neat and clever idea - I think you could even do this without using a separate/exported file, by using a "live" connection into FlightGear (telnet/props or httpd) and query/monitor the route manager that way.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Photoscenery generator for Julia compiler

Postby merspieler » Mon May 24, 2021 1:29 pm

I'd advice against such massive automatic downloads, accessible for everyone except if we get permission from the provider to use their service for that.
Nia (you&, she/her)

Please use gender neutral terms when referring to a group of people!

Be the change you wish to see in the world, be an ally to all!

Join the official matrix space
merspieler
 
Posts: 2241
Joined: Thu Oct 26, 2017 11:43 am
Location: Wish to be in YBCS
Pronouns: you&, she/her
Callsign: you&, she/her
IRC name: merspieler
Version: next
OS: NixOS

Re: Photoscenery generator for Julia compiler

Postby Hooray » Mon May 24, 2021 7:25 pm

As has been discussed elsewhere already, that's a very valid concern/recommendation - however, setting up a mirroring proxy would not seem that far-fetched - and then we could use a single proxy to cache FG related queries, so that most queries would be handled by the proxy once they're in the cache
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Photoscenery generator for Julia compiler

Postby abassign » Mon May 24, 2021 8:50 pm

Hi everyone, I have added a 0.2.7 version that allows you to download the scenario from other servers, besides the ESRI server used up to now. Currently I have decided to insert the USGS (USA) and PNOA (Spain) servers which seem to me to be the most reliable ones.
The USGS server is very fast and allows me to download at the maximum speed I have on the network (download with Julia -t 10 at 4MB / s with a 5-6MB / s connection). Therefore, for those who want to cover the USA they can comfortably do it, plus the USGS server has a much more GNU compatible license than the others.

Here I have inserted the paragraph of the manual that introduces the use of the new option --map n

https://wiki.flightgear.org/Julia_photoscenery_generator#--map_n

What I think is very interesting is that the access rules to the various servers have been inserted in the XML file params.xml and therefore anyone can modify them or add new ones.
Before doing the operation, however, read the manual, it is important to understand how to do it and not to encounter error messages, as is convenient before, to make changes on the file, save an original copy.

I very much hope that others can find other orthographic image servers so that we can increase availability for everyone.
If you report them to me, I am immediately available to insert them as an update.

Server list that are inserted into the params.xml file:

Image

In this version it was also introduced the possibility to download the photo-scenery images only along a path defined by the FGFS route manager. This is the first time that the program has interacted with FGFS and it does so simply by requesting the name of the route file you have saved with the --route parameter. The program searches for it within the user area and then uses it to generate a series of sections that have a radius defined with the -r parameter.

The advantage of this option is that it is possible to make flights, even very long ones, with a relatively limited number of images. To test the option I left the LOWI-LIME.xml file which can be used for the route connecting LOWI (Insbruck) with LIME (Bergamo Orio) which can then be loaded with the option:

Code: Select all
julia -t 10 photoscenary.jl --route LOWI-LIME.xml -s 2 -r 10


https://wiki.flightgear.org/Julia_photoscenery_generator#We_follow_a_route_generated_with_route_manager_by_--route
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Mon May 24, 2021 9:14 pm

Hooray wrote in Mon May 24, 2021 7:25 pm:As has been discussed elsewhere already, that's a very valid concern/recommendation - however, setting up a mirroring proxy would not seem that far-fetched - and then we could use a single proxy to cache FG related queries, so that most queries would be handled by the proxy once they're in the cache


In fact, as an Information Systems manager ... I confirm the problem and therefore it is necessary, in this case, to use a value of Julia -t n with very low n (1-2).
However, in version 0.2.7 onwards there is a thread limiter depending on the number of CPUs used which limits too unscrupulous use of the -t parameter of the Julia compiler.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Mon May 24, 2021 9:20 pm

Volador wrote in Fri May 21, 2021 11:27 am:Is there a plan to integrate this function into FG in the future? A graphical interface (or look ahead) would be brilliant -drag selecting the coordinates to download, I realise that for licensing reasons individuals may have to download to their own systems, and further to that would put too heavy a demand on the satellite photo services. Are there further thoughts on a legal way to use a repository on other servers for the dds files? (if that's how it works, I'm not a programmer).


Actually I'm thinking about it, but honestly it takes a lot of work and I think the best way is to use FGFS directly. My idea is to use the route manager which compiles with the graphical aids of FGFS, as explained here:

https://wiki.flightgear.org/Julia_photoscenery_generator#We_follow_a_route_generated_with_route_manager_by_--route

This program is a very heavy program, there are users who activate it in the night to download tens of GB of images, so more than a graphical interface I think it is useful to have good start options, then maybe in the future ...
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Mon May 24, 2021 9:29 pm

merspieler wrote in Fri May 21, 2021 4:33 pm:So global scenery is absulutely not feasible right now.
If we could get an agreement with an image provider for a limited area, that might work. Still... we need to find one, and convince them


You are absolutely right, I think that this program, simplifying the downloading of images and at the same time making them fast at the limit of the available bandwidth, goes in the direction of using only the resources available on the internet, moving the problem of rights to the user with map service provider.
Instead we can do one thing ... insert a LOD on the maps, or be able to have Orthophotos with various sizes inserted on different folders, in this way we favor the overall performance.
For my program, taking the DDS at a certain resolution and reprocessing them at a lower resolution is not a problem, you just need to have an operating rule.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

Re: Photoscenery generator for Julia compiler

Postby abassign » Mon May 24, 2021 9:36 pm

Hooray wrote in Mon May 24, 2021 9:17 am:@abassign: using the route manager to pre-fetch tiles/imagery is a neat and clever idea - I think you could even do this without using a separate/exported file, by using a "live" connection into FlightGear (telnet/props or httpd) and query/monitor the route manager that way.


I'm working on this aspect, the idea is to use a Telnet to periodically read the aircraft position files. Unfortunately, even if technically feasible, FGFS does not dynamically update the image tiles when they become available, but the update can only take place through the reload of the scenario.
For this I have opted on the file produced by the route manager.
However soon I insert the option of "tracking" of the plane (or UFO) that produces the file of the path directly in the memory of the program, which, in turn, uses it for having and the list of tiles to download.
Developer of the program https://wiki.flightgear.org/Julia_photoscenery_generator
FDM developer of the G91R1B aircraft https://wiki.flightgear.org/FIAT_G91R1B
JSBSim collaborator
abassign
 
Posts: 947
Joined: Mon Feb 27, 2012 6:09 pm
Location: Italy (living 5 Km from airport LIME)
Callsign: I-BASSY
Version: 2020.4
OS: Ubuntu 20.10

PreviousNext

Return to Scenery

Who is online

Users browsing this forum: No registered users and 10 guests