Board index FlightGear Support 3rd Party Repositories

Cloning fgdata with GIT submodules

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Tue Feb 10, 2015 1:57 am

IMPORTAT INFORMATION

The FGDATA has succesfully been transferred to the NEXT GENERATION FGDATA
Now also available (NOT IN TEST MODE) with Submodules

The location of FGMEMBERS remains unchanged at: https://github.com/FGMEMBERS
The fork of the Official FGDATA can now be obtained at: https://sourceforge.net/p/fgdata/submod ... next/tree/

Read the Notification here:

viewtopic.php?f=28&t=25314&start=45#p234592

__________________________________________

IAHM-COL wrote in Sun Feb 08, 2015 8:43 pm:Planes in the SVN and the FGMEMBERS FGDATA not present in the mainstream FGDATA as of FEB2015. Thus still their history is not recovered.
Code: Select all
‘Caproni-C22J/’
****

All these planes were incorporated to mainstream after the SVN forked. Thus they do not exist in the FGDATA.
Using "git svn" functions, it seems I was able to recover their history, and rewrite it to the FGMEMBERS git repositories.

Best,
IHCOL
Last edited by IAHM-COL on Mon Mar 09, 2015 8:12 pm, edited 2 times in total.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Wed Feb 11, 2015 1:05 am

A private communication with ONOX that I wanted to share openly:
IAHM-COL wrote:So these are in my perspective some advantages of the submodules method

1. The size of FGDATA repository remotely remains small. A submodule does not increase the size of the repo hosted in GITHUB, or gitorious etc. The submodule only alters 2 or three metafiles, a few lines per module. So, in the case of the FGDATA, we were having problems with gitorious because of the size of the FGDATA repo. But keeping it submodular aleviates the FGDATA size
2. This with a second advantage. The initial GIT CLONING is faster. Compare 1.4GB to 17GB (the current Monlithic FGDATA with all the planes). In general all traffic seems alleviated.
3. It provides a mechanism for users to git fetch and test aircraft in their git version INDIVIDUALLY.
Not needed to download all the planes to experience a few of them.
4. It provides a mechanism for users to deinit and remove aircrafts he/she doesn t intend to keep
5. Every aircraft becomes a git repository of its own. Imagine the endless posibilities of enablin every FG member to clone/fork any of the aircraft and propose modifications without needed to gain access to the FGDATA, nor all the aircraft in a monolithic SVN!
6. Being distributed, as you know, means a greater level of freedom on how to develop any of the aircrafts

7. A module always points to a commit in a development tree. So developers adding commits do not change the state of any aircraft as based in the submodule, until the FGDATA is changed to point to a new commit. Therefore it provides freedom to develop the aircraft without compromising the integrity of the data stored in FGDATA.

That is my summary of pros.
Cons:
1. Apparently it needs to be learned. Specially for the FGDATA commiters not to mess wth a submodule.
Learn here means: Get one single hard fact: The FGDATA is for development of the BASE FGDATA. Submodules MUST Be developed in an independent tree. AFAIK, that is the one single delicate issue hard to handle with submodules.

In my book that is 7 pros, 1 Con.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Wed Feb 11, 2015 1:22 am

IAHM-COL wrote in Sun Feb 08, 2015 8:49 pm:Notes about FGDATA test with submodules

The rebasing of all aircraft in FGDATA has not taken place. Thus the FGDATA still points to the history-less initialization commit.
I am unsure how that will work at the moment, and the rewritten history may have broken the FGDATA until all the aircraft submodules are rebased, and point to the latest commit.


Just as I was predicting.

I was rewritting the history of every aircraft, one aircraft at a time, and while doing so, I was bringing back the previous commit history of every of these aircraft!
In my opinion that was a really cool thing to do, as now every aircraft NOT IN BASE is a repo of its own, but also, it has inherited its commits log from the large fgdata hosted in gitorious.

While doing so, I was breaking the FGDATA test1 one aircraft at a time. Why? because this was pointed to a commit for every aircraft that ceased to exist every time I re-updated the aforementioned aircraft. To solve that, I had to generate FGDATA in this case, now with all submodules pointing to the latest dev stage of the aircrafts. And I DID IT NOW

SO

Submodular FGDATA is now ready for early adopters and testers

I welcome everyone to test ride this method, try to gather feelings, opinions, and suggestions. Compare this to developing ALL of our aircraft via a centralized SVN server that just a few can SAFELY access.
Remember, some general guidelines of how to test this submodular FGDATA are provided above (and linked below)

viewtopic.php?f=28&t=25314#p231506

Also remember, I will be very happy to get the FGDATA commiters to just provide me with a GITHUB account, and I will then grant access to the FGDATA and FGMEMBERS tree.!! :D

This FGDATA repo is STILL a test version

The ideal here will be that we have a FGDATA without the aircrafts that is smaller in size but has a preserved history, as well as maintain the branch structure of the current FGDATA. In addition one that the dev-cores in maintaining and developing. Then, from that, I could clone/fork and fetch all upstream changes they make.
Finally, I keep a little light of hope that SVN will be an abandoned idea, or one that is keep cool and calm instead. And maybe everyonce certain time it may be possible to update the modular repo with changes that stubbornly end up in the wrong place (a.k.a SVN fgaddon)

Best,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Thu Feb 12, 2015 8:27 pm

A little introduction on how anyone can propose Aircraft changes to the planes hosted in FGMEMBERs

For those unfamiliar to the GIT workflow, I prepared a very little instructive on how to get changes proposed in all the FG Aircraft hosted in FGMEMBERS.

I know it can be a little of an step-learning curve, but it gives the reward of being able to form a community of Aircraft developers that are able to propose modifications to any/all of the FG Aircrafts.

If you are comfortable-advanced with git, this doc may be a little too basic. Keep in mind I attempt to get anyone up2speed.

Best,
IHCOL

READ THE DOC HERE
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Thu Feb 12, 2015 9:35 pm

AndersG wrote in Thu Feb 05, 2015 7:49 pm:You could consider git-svn to pull upstream changes into your per-aircraft submodules (probably cherry-picking them across to a "pure" git branch rooted in the old history).



Thanks AndersG, once again, your suggestions are spot on!
WOW!

Problem
Since the aircrafts present in the SVN eliminated all history, the developers submitting modifications to SVN are actually creating a "parallel universe" where the same aircrafts did occur with a "different commit sha1 history list"

This situation makes the history-full planes in FGMEMBERS impossible to simply fetch-merge from the SVN, in the traditional way

The solution
The SVN can be cloned (per aircraft) using git-svn then fetched (per aircraft) into a temporary branch in the git-rooted aircrafts in FGMEMBERs
The histories of both branches are incomparable.

The solution is to add every new commit in SVN with "cherry-pick" on top of the master branch of the git planes, effectively updating the git planes we have located in FGMEMBERS to everything done/commited in the SVN side.

Therefore, allowing the planes in FGMEMBERS to be up2date with the SVN-side devopers.


The recommendation

Due to the bizarre nature of this patching, once again, I encourage everyone to submit changes to the git versioning not the SVN versioning of these aircraft development

The example

I had succesfully updated the changes executed in the p51d from the SVN-branching to today (r288 in the SVN), the updated aircraft can be seen at
https://github.com/FGMEMBERS/p51d
Last edited by IAHM-COL on Thu Feb 12, 2015 9:38 pm, edited 1 time in total.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby Gijs » Thu Feb 12, 2015 9:37 pm

I'd suggest you to bring this up on the devel mailing list. The move to SVN has been discussed in great lengths, switching back to Git is not something you can or should decide on your own. That's currently only adding more confusion to people wanting to submit their work to the project ;-)
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9447
Joined: Tue Jul 03, 2007 2:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Thu Feb 12, 2015 9:41 pm

Thanks Gijs

Ive heard this comment once before....
But who is the devel-mailing list?!
I am inviting switching back to git the FGDATA (with submodules) but definitely there are not ears listening in this forum :(

I agree I can't switch to git on my own. But about whether should or should I not have a clone of the aircrafts in GITHUB is really a major question on my mind right now.
I definitely don't want to add confusion. But in my opinion the current state of FGDATA (partly on the Git, expecting the planes to dissapear, planes moving to an add-on, rooted in SVN, etc) it is a very confusing state of mind by itself....

Now, is there anyone that has access/know of the mailing list, volunteer to raise a voice that says I am doing some effort to invite a git-return?!

PS: I understand and it is clear to me that the planes I host in FGMEMBERS in github are not FG mainstream contributions, and that at the moment, after extensive discussion of core-developers, mainstream contributions need to go throught the SVN key-gates :|
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby Gijs » Thu Feb 12, 2015 9:48 pm

I'm not trying to downplay your effort (you can look it up, but I've been doing pretty much the same over three years ago, we got very close back then to what you're trying to achieve). But, this is such a fundamental part of the project that it needs to have a broad consensus to be accepted.

Your questions really have a far better chance of getting answers on the mailing list than on this forum. The mailing list is just that, a list of email address that receive a copy of your email if you send it to the list. See our wiki for details: http://wiki.flightgear.org/Mailing_list
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9447
Joined: Tue Jul 03, 2007 2:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Fri Feb 13, 2015 2:27 am

Thanks Gijs
I sent my first Mailing list message
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby karam » Fri Feb 13, 2015 11:52 am

IAHM-COL wrote in Thu Feb 12, 2015 8:27 pm:A little introduction on how anyone can propose Aircraft changes to the planes hosted in FGMEMBERs
[snip]
READ THE DOC HERE


Nice presentation, I have a few comments

On page 1 (slide 2) the first paragraph says
"Initially it contains all aircraft present in the git repository of the FGDATA as of November 2015, and new additions in the SVN repository of FG “Addon” Aircrafts"

2015 should probably be 2014.


On page 2 (slide 3)

The information on git for macs could be extended with: "MacOS: Xcode terminal tools provides git v1.9. Macports, http://www.macports.org, provides git v2.2."

Actually, anyone planning to do anything serious on Macs should install Xcode and a software package manager like Macports.


And, I'd like to express support for your effort. I was not aware of it until you sent a message to the devel mailing list. I'm one of the non-developers using mailing lists rather than forums to track fg development and I actually registered to this forum now to send you this comment. The fg development environment is a fairly harsh at times and I think this is one such occurrence. Maybe because you entered the discussion to late, maybe not. You need to be persistent and there is nothing wrong in forking the source tree. I will adopt your way for my aircraft check outs and for tracking my local changes.
karam
 
Posts: 9
Joined: Fri Feb 13, 2015 11:04 am

Re: Cloning fgdata with GIT submodules

Postby Hooray » Fri Feb 13, 2015 4:57 pm

Many of those who have actually used "git submodules" don't like them, or at least prefer git subtree instead.

For some background, I'd suggest to consider these articles:

http://blogs.atlassian.com/2013/05/alte ... t-subtree/
http://www.mos6581.org/git_subtree_alternative

Like Gijs mentioned earlier, this is a pretty critical part of the project and "splitting" fgdata has been attempted several times already.
However, this is beyond just project dynamics and people, it's also a technical issue, especially considering that FlightGear tends to use "free" (or donated) infrastructure for most of its needs, and git isn't exactly well-suited for binary resources like those commonly part of an aircraft (think textures, sounds, 3D models etc) - so it's also a matter of finding freely available hosting/infrastructure capable of providing what's needed.

As you could recently see during the wiki downtime, hosting critical infrastructure yourself isn't necessarily the smartest idea if you aren't enormously experienced with redundant infrastructures (think load-balancing, mySQL replication, CDNs etc) - so should be better delegated to a 3rd party like sourceforge, github or gitorious.

However, the FlightGear project should have some leverage here, as long as the key people behind the project are willing to get in touch with the people behind said services, finding a compromise shouldn't be too difficult.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11924
Joined: Tue Mar 25, 2008 8:40 am

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Fri Feb 13, 2015 6:03 pm

@Karam
Thanks for your critical revision. I edited the document :)
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Fri Feb 13, 2015 6:12 pm

@Hooray

Yes. I did acknowledge before that the submodules structure is rather heavily criticized online.
I read both of these two documents before, when I was trying to understand the fuzz around subtree. But I couldn't. Maybe too technical for me?

In any case, what I got for my reading is that submodules open the path of danger. The solution is understanding two facts with submodules

1. That the submodule points to an specific commit in the tree of the subproject. If the project moves along, the superproject will not. That is a convenient behavior for FGDATA. Ie, you point an aircraft development to commit A. The project moves along to B->F, with new branches and all that type of stuff.
$git submodule update --init Aircraft/example
will still bring commit A to the table.
That's good in my opinion (for us). We only need to push the Aircraft/example to commit F, when seems that Aircraft development has reached maturity stage to be globally updated.

2. The most painful point is
The superproject git is to develop the superproject: NOT ANY SUBMODULE!
Apparently, the files of the submodules initiated are there locally, but not remotely. Meaning, making changes there, and hoping that a super-project push will update the Aircraft is a "sorry-but not". Besides, apparently, it can actually mess your clone!
The solution is easy.
Know fact 2, and develop your aircraft on an independent tree outside FGDATA. then fall back to model outlined above. Not that inconvenient for FG, in my opinion again.

Most aircraft developers are not even in git version of flightgear, and they flight official releases. They don't even need to git fgdata, for real. Just fetch and play with the aircrafts you are intending to develop

With the submodules model the advantage of fetching fgdata is the gained control of initializing, and deinitializing aircraft individually. It almost work like
% instal this aircraft as in FGDATA last version
% de-install this aircraft as in FGDATA last version

Kind of option

I did test it on my side, and it worked pretty good indeed. Planes came and go to my will.
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

Re: Cloning fgdata with GIT submodules

Postby Hooray » Fri Feb 13, 2015 7:24 pm

Personally, I don't care at all - I don't mind using git, and the fgmeta project is already using git submodules for the superbuild: http://wiki.flightgear.org/Superbuild

In general, requiring people to use git for doing aircraft development may still be a bit too much for the majority of folks interested in aircraft development.
However, splitting off $FG_AIRCRAFT from $FG_ROOT is generally a very worthwhile thing to aim for and it's been long overdue: http://wiki.flightgear.org/FlightGear_G ... ing_fgdata

Ulltimately, the people offering to handle all the ugly corner-cases are likely to be the ones to make the decision - we've seen other attempts at doing this, and while a few contributors willingly provided scripts to automate /some/ chores, there were quite a few corner-cases that were not addressed. It is my understanding that F-JJTH and others are dedicated to handle all that stuff.

Personally, I am unlikely to look at more than just a handful of aircraft, so I appreciate any efforts to reduce the size of the fgdata repository, as well as any other attempts at making FlightGear more modular, not just in terms of data, but also at run-time.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 11924
Joined: Tue Mar 25, 2008 8:40 am

Re: Cloning fgdata with GIT submodules

Postby IAHM-COL » Sat Feb 14, 2015 3:36 am

@ALL

The Aircraft I hosted in Github FGMEMBERS have been updated to the latest revision of the SVN series. [CURRENT R295]

Each of the 294 commits made in SVN after the [r1]Innitial import, had been rebased (actually using cherry-pick) on top of the respective aircraft.

This means a couple of things

1. Those taking the aircraft from the github FGMEMBERS are not currently outdated with the work people had commited to SVN
2. For those interested in switching to a "pure-git" system, nothing you have done and push into the SVN had been lost, and now it is perfectly possible to just pick up from here and use the git system. I know this will be easying a transition to a git system if that is decided for mainstream

Best,
IHCOL

https://github.com/FGMEMBERS

PS:
Once again, developers with commit access to the FGDATA aircraft in gitorious, and/or the SVN will get a "membership" of FGMEMBERS, allowing them to write to the planes in github directly (write access means forking is not needed, unless desired to do so).
Best,
IHCOL
If we gave everybody in the World free software today, but we failed to teach them about the four freedoms, five years from now, would they still have it? Probably not, because if they don’t recognise their freedoms, they’ll let their freedoms fall
User avatar
IAHM-COL
Retired
 
Posts: 4064
Joined: Wed Aug 08, 2012 5:40 pm
Location: Homey, NV (KXTA) - U.S.A
Callsign: HK-424D or ICAO4243
Version: 3.7-git
OS: Linux

PreviousNext

Return to 3rd Party Repositories

Who is online

Users browsing this forum: No registered users and 1 guest