Board index FlightGear Support 3rd Party Repositories

Cloning fgdata with GIT submodules

Re: Cloning fgdata with GIT submodules

Postby sanhozay » Sat Feb 14, 2015 9:22 am

COL,

Does this require a Github login and SSH keys set up and imported into Github?

I managed to do the initial checkout with https, which doesn't need a login, but I can't get the modules to update via https? The modules seem to be set up with git@github... urls.

I do have a Github account and I know how to set up the SSH keys but I'm trying to do it without, because we were able to clone from Gitorious without a login. Do you know if that's possible?
sanhozay
 
Posts: 1207
Joined: Thu Dec 26, 2013 12:57 pm
Location: EGNM
Callsign: G-SHOZ
Version: Git
OS: Ubuntu 16.04

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Sat Feb 14, 2015 5:44 pm

IMPORTAT INFORMATION

The FGDATA has succesfully been transferred to the NEXT GENERATION FGDATA
Now also available (NOT IN TEST MODE) with Submodules

The location of FGMEMBERS remains unchanged at: https://github.com/FGMEMBERS
The fork of the Official FGDATA can now be obtained at: https://sourceforge.net/p/fgdata/submod ... next/tree/

Read the Notification here:

viewtopic.php?f=28&t=25314&start=45#p234592

_______________________________


Thanks for reporting G-SHOZ
Indeed, I did not consider that particular but very important scenery!

Would you mind to run a few tests for me to see how to approach this situation?

1st, lets deinit your submodules,
Code: Select all
$git submodule deinit


2nd, git pull to update to a new configuration that should now fetch submodules with https:// instead
Code: Select all
$git pull


3rd, synchronize to the new urls
Code: Select all
$ git submodule sync


4rd, init an aircraft
Code: Select all
$ git submodule Aircraft/Douglas-Dc3


5th, update the module
Code: Select all
$ git submodule update


If this works for you, that means the problem of needing a github account to fetch the submodules has been eliminated now :oops: :)

I hope it works, but let me know please :P

IHCOL
Last edited by IAHM-COL on Mon Mar 09, 2015 9:13 pm, edited 3 times in total.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby sanhozay » Sat Feb 14, 2015 5:54 pm

Excellent :)
sanhozay
 
Posts: 1207
Joined: Thu Dec 26, 2013 12:57 pm
Location: EGNM
Callsign: G-SHOZ
Version: Git
OS: Ubuntu 16.04

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Sat Feb 14, 2015 6:50 pm

Did it work :? :D
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby sanhozay » Sat Feb 14, 2015 9:55 pm

Yes. Downloaded the base package and initialized a module without using a Github account.
sanhozay
 
Posts: 1207
Joined: Thu Dec 26, 2013 12:57 pm
Location: EGNM
Callsign: G-SHOZ
Version: Git
OS: Ubuntu 16.04

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Mon Feb 16, 2015 9:02 pm

FGDATA test submodules updated:: IMPORTANT NOTIFICATION

Hi All

As you are probably aware (or maybe not), a group of Aircraft developers are updating FG Aircraft in a repository outside FGDATA.
This repository was made as an attempt to remove non-base aircraft away of the FGDATA repository and alleviate FGDATA size.

Our FGDATA distribution is currently a test distribution to evaluate the feasibility of having a parallel FGDATA repository that still maintains the Aircraft but those are hosted as submodules. This gives any person the ability to initialize either all aircraft or only those that person is interested in fly and test. The tests that had been run in this FGDATA had been largely succesful. And the submodular approach had demostrated to be a feasible alternative for those that want to remain with a purely git-ted system, in addition to keep their FGDATA up to date, while still gain largely in the modularity of the aircrafts.

For the last four months or so, fetching fgdata had not successfully updated the aircraft whose improvements had been done in the SVN repositories.

Last week, IAHMCOL had rebased all SVN progress into the git aircraft repositories, until the SVN revision 295. Earlier today, the FGDATA submodules had been updated to the latest commit, therefore effectively updating every aircraft modified. This FGDATA test now therefore has all aircraft completely up-to-date with the latests progress on the submodules, some of which only exist in the FGMEMBERS repositories, and all those commits in the SVN.

Anyone testing the FGDATA submodular approach is now encouraged to fetch the updated aircraft:

Performing the submodules updates
Code: Select all
$ git pull    #Updates the FGDATA
$ git submodule update   #Updates the Initialized Aircraft submodules.


Note that git submodule update will only update those aircraft that you have initialized, and perform no alterations in any aircraft you had not initialized. For any submodule you had not initialized, but intialize from now on, the update will bring you the newest commit, that is the updated aircraft.

I will appreciate reports indicated me if this method updated your FGDATA aircraft succesfully as expected.

Sincerely,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Mon Feb 16, 2015 9:13 pm

Knowing which Aircraft had been recently updated?

The easiest way to know whether you have a submodule of an aircraft that has recently been updated is to look at those submodules that had been affected by the latests commit:

https://github.com/FGDATA/fgdata/tree/master/Aircraft

Here you can see which aircraft are affected/updated with changes until the SVN revision 295.

Best,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Mon Feb 16, 2015 9:50 pm

Gijs wrote in Thu Feb 12, 2015 10:48 pm:I'm not trying to downplay your effort (you can look it up, but I've been doing pretty much the same over three years ago, we got very close back then to what you're trying to achieve). But, this is such a fundamental part of the project that it needs to have a broad consensus to be accepted.

Your questions really have a far better chance of getting answers on the mailing list than on this forum. The mailing list is just that, a list of email address that receive a copy of your email if you send it to the list. See our wiki for details: http://wiki.flightgear.org/Mailing_list



Hi Gijs

Thanks for your suggestion.

As you requested me, I did send a devel-list email requesting everyone to consider the submodule as a feasible alternative to the aircraft being hosted in SVN. I am certain that you are subscribed and therefore you have access to first-hand information of what the comments and opinions that took place there. You were right that I was more likely to get some response from the dev-cores by talking on a more appropriate channel than the forum, so I definitely thank you for pointing me in the right direction.

To summarize, the dev-cores never reached a broad consensus that you pointed above, but a subset of them did definitely decide to take the lead move and displace the aircraft to a SVN fork. The discussions of how the aircraft migration were very long and took a large toll on many developers that felt rather tired to keep up the fight, so those that were more in favor of the git move as oppose to an SVN move just saw the thing happening and remained silent, which had been taken by the executers of the SVN move, much to their convenience, as a silent agreement.

Their defense on their system of SVN is basically that what I claimed before. It is not a question of whether it is the best move. It is a problem of what is done is done. No need to revert decisions. No need to evaluate potentially better ones. They claim it to be a sensible move, which off course for those running the trains, a centralized server makes a lot of sense, thus is a sensible solution. They do not care of possible consequences this can have on the fostering collaborations and maintain an opensource approach to the project. Some of them do consider that since they are not aircraft developers, a suboptimal decision is one not affecting them directly, and thus they are, conveniently apathetic.

In summary, there seems to be a very low interest to negotiate or discuss a possibility to improve and foster aircraft developement cooperations. That seems to be the consensus reached. Just leave the SVN without questioning if the submodular proposal is technically a sound and better alternative. The only consensus reached is apparently; "I don't care". "This is not my problem". and "It is too late for considering a reversal".

As a matter of fact, I think comradery, instead of professional judgement once again has taken a bigger lead into the decisions occuring in flightgear, some of which do occur actually in secret weekendly meetings. I hope one day I get to belong to that secret -almost massonic- group, but for now I am barely an end user of this fantastic software.

With all previous considerations in mind, I will be addressing the questions you proposed (and I quoted below):

Gijs wrote in Thu Feb 12, 2015 10:37 pm:I'd suggest you to bring this up on the devel mailing list. The move to SVN has been discussed in great lengths, switching back to Git is not something you can or should decide on your own. That's currently only adding more confusion to people wanting to submit their work to the project ;-)


Well. It is definetely something that a facet of developers are commited to avoid. At all cost. They have decided unilaterally an SVN move, and they will deffend it without even the need of focusing on the technical aspects of it.

As they have made their own decision, I thing it is morally appropriate for me to make my own decision, and offer the community a modular approach to fetch FGDATA with those aircrafts each want to initialize. Apparently no-one here is really interested in reaching a real broad consensus.

Anyone of the developers that want to have commit access and ownership of my git fork only need to contact me with express of their interest.

With that in mind, the FGDATA submodular approach has passed really nicely the technical test required to be considered feasible. There are not a current limit on submodules allowed by git, and they can be initialized and deinitialized comfortably. Tandem modification of aircrafts is also technically feasible and not more complex than doing a similar operation in other contexts. Finally, changes and improvements made to aircraft in SVN addon repository can be readily rebased and made available to the submodular FGDATA without risking its integrity. History needs not to be "lineal" in git repositories.

The FGDATA without aircraft will be offered soon by the core developers, and soon afterwards I will clone a fork, for the official FGDATA submodular repository. This important last step is still pending the developers to offer their official Aircraft less repository.

So in summary, I think having a fork of FGDATA that still contains aircraft as submodules is something that can be done, and in my opinion should be done.

And I found ethically and morally appropriate to invite any aircraft developer that finds the actions of the certain facet of core developers morally or technically unsound to consider submitting their improvement to the FGMEMBERS hosted aircraft. That off course is an invitation extended to everyone, including yourself, @GIJS.

I understand that aircraft development is an important part of Fligthgear development, but not one that cannot withstand forking, as it is appropriate to fork any opensource project.

Each user in the future will simply have the option to both fetch and develop their aircraft with the SVN, or using an FGDATA that includes these and other improvements from the submodular fork.

Please, do not doubt to address me with any concern or question you may have on this respect,
Duefully yours,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby wlbragg » Tue Feb 17, 2015 1:38 am

IAHM-CO,

I so much appreciate you efforts, I plan on using this effort as I was "un-educationally" opposed to a SVN and GIT solution VS one or the other.
My only concern at this point is if any aircraft developers follow the herd and use SVN exclusively, what will be the necessary steps, on a regular basis, to incorporate that into the GIT sub-module system and keep it up-to-date. Is that something the end-user, (me), has to do or is that something the GIT maintainer will do?

I applaud to tenacity.

p.s. I have "cloned" the SVN trunk as I prefer to have the entire (updated) package of aircraft at my beckoned call. It wasn't really much of a learning curve but there is just something about using two different methods for the same project that I find distasteful for whatever reason.
The one argument I keep hearing that I couldn't wrap my head around was the notion of SVN handling binary better. What does that even mean? They both just store the binary. It's not like you edit it line by line and make edits in that fashion in either system. Just my "uneducated" two cents worth.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7587
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Tue Feb 17, 2015 2:27 am

Hi Wlbragg,

Interesting question related to the current situation of having two forks for aircraft obtaining.

Your concern, as I follow is: If an aircraft developer follows SVN exclusively, and makes his changes there, what are the steps necessary to incorporate his improvements into git?

No step necessary at all. What happens in SVN is going to be rebased on top of the Github aircraft by a cronjob (an schedule in a linux server) weekly. So what's new in SVN will come up to github once a week. No need for a developer improving aircraft in SVN to do anything at all.
Rebasing in this context is very interesting. It allows the git repository to have a new commit that mirrors the changes that occur into SVN. The commit is a copy to that happened in SVN, but not the same commit, so the commit ids are actually different. A rebase like that took place today, bringing all aircraft in github to the same updated status as the SVN revision 295.
So in plain English, I am setting a program to keep the github aircraft up to date with respect to SVN.

But, then there is the other face of the coin.
Changes occurring in github are not being currently mirrored by nothing or anyone into the SVN. So expect accross the days to have changes (improvements/ additions/ deletions/etc) to be only existent in the github area.

>> Is there something the end user of the Forked FGDATA with submodules need to do to keep the planes updated?

Brief answer. Yes, there is.
An end user needs to update the submodules he/she initializes to fetch new changes
that is done as follow

Code: Select all
$ git pull #Updates FGDATA
$ git submodule update  #Updates the submodules initialized


This can be done as frequently as desired. But will have no effect if your FGDATA is up to date.

Please, keep in mind that the current FGDATA has been placed for testing purposes of the submodule methodology only. We are waiting for the core developers to provide an FGDATA repository cleaned up with aircraft removed, that we can effectively fork and top the submodules onto, and that will be the definite FGDATA repository to obtain the submodules from. It will be an exact copy of the mainstream FGDATA with the exception of additionally provide the aircrafts as submodules.
Last edited by IAHM-COL on Tue Feb 17, 2015 2:48 am, edited 1 time in total.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Tue Feb 17, 2015 2:41 am

wlbragg wrote in Tue Feb 17, 2015 1:38 am:
p.s. I have "cloned" the SVN trunk as I prefer to have the entire (updated) package of aircraft at my beckoned call. It wasn't really much of a learning curve but there is just something about using two different methods for the same project that I find distasteful for whatever reason.
The one argument I keep hearing that I couldn't wrap my head around was the notion of SVN handling binary better. What does that even mean? They both just store the binary. It's not like you edit it line by line and make edits in that fashion in either system. Just my "uneducated" two cents worth.


Well, although submodules will in fact provide a nice way to only fetch a subset of aircraft locally, without having all aircraft fetched, it also provides means to get the whole package with ease.

Code: Select all
$ git submodule init Aircraft/707
$ git submodule update


This will give you the 707 module. and update it. But only that one.
You can imagine that if you want all 470 submodules you will be typing for quite a while.
Unless you do

Code: Select all
$ git submodule init
$ git submodule update


The init command with no parameters means ALL!
So in two commands you get effectively the whole package!

further alternatives are
Code: Select all
$ git submodule update --init

That allows to update and init the whole package in one single line command!

And for the ones liking to save more strokes
Code: Select all
$ git clone --recursive <url>

Actually does all the thing. Clones the url. Initializes the whole submodules and updates them too. So that's the one liner that bring the world.


I am sorry about how distasteful it is to have 2 mechanisms to bring FG aircraft and FGDATA too. I had lobbied hard over the weekend to bring everyone into a decision to only have a submodules approach. But the SVN promoters are very sure that since they already created one big SVN repository, there is not coming back. Mis-paraphrasing some comment, it said something along the lines of <<There's one thing that I think is even worse than a suboptimal decision, and that's changing the plan all the time.>>

For reasons of allowing helping fostering a more cooperative environment on Aircraft development, I had nonetheless decided to keep the second fork purely git rooted available to the community. And from the point the final (not-test) FGDATA with submodules gets established, it will be on the hands of everyone else to do of the Flightgear Aircraft world, just a better one.

Why does git have somewhat issues handling binary? Partly because of how easy it is to compress text files. Partly because git is a "never forget" method. Imagine this:
you create an image file that weights 100MB, and that is 2048x2048 large.
Then you decided that is unnecesarily huge. so you reduce it to 256x256, and its size goes to less that 1MB. Great. You push the new image into git. But git will not just replace a file with other. Will store the large one, and make the small one available. Not great if you think that your repo size just grew. Never decreased.

How SVN is so much great with binary? I really am in your chair. I have no idea. In principle it should not be any better. And it someone can put the reasons of why SVN is a great with binaries, in plain english, I am very happy to learn this one.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Tue Feb 17, 2015 3:02 am

@Wllbragg

Another point
I Know you also argued about the unnecessarily decision of loose the commit log for the aircrafts on the subversion move.

Well, I just wanted to point you to the fact that every step necessary to recover those logs where taken, and each aircraft in github contains the full commit history of each aircraft from inception.

Recognizing the work to that of who deserves it is a polite thing to do in an open source community! (also knowing that those commits are wealth of information related about how an aircraft in flightgear is made, and how is it pollished!)
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby wlbragg » Tue Feb 17, 2015 4:36 am

Well explained, thank you.

I noticed I had a typo in a key phrase, that was suppose to be, "I applaud your tenacity".


every step necessary to recover those logs where taken, and each aircraft in github contains the full commit history of each aircraft from inception

While I really appreciate this, this is not the part of history I was referring to. The historical "gems" long forgotten, by some, not all, that I am concerned with are data that has previously been permanently removed from FGDATA for any number of reasons. Many, I'm sure, I don't even know about. Things you stumble on in the course of following a thread that maybe mentions an aircraft that was pulled do to a copyright issue, or something to that effect. That too is GIT history buried in the depths and truly irreplaceable.
I don't task you with the responsibility for this. But I do most certainly hold accountable core devs that make and implement these decisions. That they make sure they account for all the ramifications of changing midstream such an important system of the project.

Anyway, I've said my piece, now more than once, I hope someone in a position to act is listening and doesn't allow things to fall through the cracks.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7587
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Tue Feb 17, 2015 6:55 pm

10 new Aircraft submodules Added

https://github.com/FGMEMBERS/Kombi/blob/master/kombi-splash.png
@ALL

The prestes hangar created a set of Aircraft available for Flightgear: http://presteshangar.wikidot.com/start
Most of its work was released under creative commons.

Fortunately, by late 2010 some of the work of Prestes Hangar was released under GPL2 or GPL3 and presented as a series of git repositories in Gitorious: https://gitorious.org/prestes-hangar

The flightgear official distribution has completely ignored the amazing work by this Brazilian group, and therefore none of its work has ever been rebased on the FGDATA or the FGADDON, in spite of the compatible license, and hours of great job placed on these aircrafts, and regardless of the open possibility of improving any of these aircraft, many of which are inexistent otherwise!

Today, I had added a clone into the FGMEMBERS area and added a submodule for each of these aircraft in the FGDATA submodular test! :D [Only included those released on GPL and not those released under creative commons, list below)

Enjoy riding these FG secret jewels, and maybe consider editing and improving some of these

New Submodules Added
  • Aircraft/A319
  • Aircraft/A330-300
  • Aircraft/EMB-120
  • Aircraft/Embraer-195
  • Aircraft/Go9
  • Aircraft/KC-390
  • Aircraft/Kombi
  • Aircraft/MD-12
  • Aircraft/YF-17
  • Aircraft/fwta183
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4057
Joined: Wed Aug 08, 2012 6:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby Jabberwocky » Wed Feb 18, 2015 12:41 am

Well, IAHM-COL,

consider me repositroy challenged (as you well know), but as soon as we run into each other on Mumble again and you preach me through, I start to put my planes also on FGMEMBERS.

Lancaster-JSB
Condor-JSB
MD-83
Comet-J
A little bit handiwork needed, then the Victor-J with the newer autopilot
The VIPs (747-8VVIP, VIP and the Lineage 1000
well, and more upcoming.

J.
Jabberwocky
Retired
 
Posts: 1316
Joined: Sat Mar 22, 2014 8:36 pm
Callsign: JWOCKY
Version: 3.0.0
OS: Ubuntu 14.04

PreviousNext

Return to 3rd Party Repositories

Who is online

Users browsing this forum: No registered users and 2 guests