Board index FlightGear Development AI Traffic

2.12 AI problems

Intelligent, computer controlled vehicles that drive/fly over the planet!

2.12 AI problems

Postby gooneybird » Fri Sep 27, 2013 9:18 am

The AI models and traffic files in 2.12 are a real mess, there are old files that should have been deleted mixed with some of the new/updated files, I know what files I have added/updated but have no idea what others have added so have given up trying to sort it out.

I've had a look at the AI files on GIT and they seem correct I want to know is it possible to copy the AI directory from GIT into 2.12 or would that cause to many problems?
Currently running 2018.1 but if it doesn't stop crashing I'm taking up tiddlywinks!
User avatar
gooneybird
 
Posts: 2912
Joined: Sat May 31, 2008 1:57 pm
Location: Essex, UK
Callsign: G-OONY
Version: 2018.1
OS: win 7 64

Re: 2.12 AI problems

Postby zakalawe » Fri Sep 27, 2013 2:13 pm

You can copy it over fine.

Note I am currently getting the wrinkles out of the 'terrasync for other stuff' code, and the traffic files are the first item on the list to attempt to move to a new system. What I will do is make an SVN repository on my server, and import the current traffic files from fg-data into. Anyone who wishes can than have access to improve the files. (Via git-svn if they love Git). When AI traffic is enabled FG will sync the traffic definitions before starting up the traffic manager.

Which should be a nice win for everyone :)
zakalawe
 
Posts: 1149
Joined: Sat Jul 19, 2008 4:48 pm
Location: Edinburgh, Scotland
Callsign: G-ZKLW
Version: next
OS: Mac

Re: 2.12 AI problems

Postby adrian » Fri Sep 27, 2013 3:33 pm

James, please note that most of the traffic files are automatically generated using my scripts from the AI traffic repository, and Durk's airline definitions. Editing those makes more sense, since any manual change to the traffic files coming from me is likely to get overwritten if the definitions or the schedules change. I can give anyone who's interested commit access to the AI repos, currently it's only Durk and me.
Also, I'd like to discourage editing of xmL files in favour of using the Scheduleviewer, which has the advantage of using a database backend and other useful features. I think currently most of the traffic comes from 3 people: Innis, Durk and me.
Cheers,
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: 2.12 AI problems

Postby zakalawe » Fri Sep 27, 2013 5:41 pm

Okay, that's fine, we just update your scripts to commit to the SVN repo as an optional final step :)

I don't care where the XML comes from, it just needs to end up in an SVN server so it can synced down to FG. (And I'm happy to host the SVN, though it will ultimately move to Google-Code alongside the main terrasync repo)
zakalawe
 
Posts: 1149
Joined: Sat Jul 19, 2008 4:48 pm
Location: Edinburgh, Scotland
Callsign: G-ZKLW
Version: next
OS: Mac

Re: 2.12 AI problems

Postby gooneybird » Fri Sep 27, 2013 8:01 pm

zakalawe wrote in Fri Sep 27, 2013 2:13 pm:You can copy it over fine.

Note I am currently getting the wrinkles out of the 'terrasync for other stuff' code, and the traffic files are the first item on the list to attempt to move to a new system. What I will do is make an SVN repository on my server, and import the current traffic files from fg-data into. Anyone who wishes can than have access to improve the files. (Via git-svn if they love Git). When AI traffic is enabled FG will sync the traffic definitions before starting up the traffic manager.

Which should be a nice win for everyone :)


Thanks that worked a treat but as I had downloaded GIT data I decided to go the whole hog and managed to install GIT (user) which at the moment is running nicely.
Looks like this old dog has a few more tricks to learn yet.
Currently running 2018.1 but if it doesn't stop crashing I'm taking up tiddlywinks!
User avatar
gooneybird
 
Posts: 2912
Joined: Sat May 31, 2008 1:57 pm
Location: Essex, UK
Callsign: G-OONY
Version: 2018.1
OS: win 7 64

Re: 2.12 AI problems

Postby qf707 » Sat Sep 28, 2013 12:37 am

adrian says
I'd like to discourage editing of xmL files in favour of using the Scheduleviewer, which has the advantage of using a database backend and other useful features. I think currently most of the traffic comes from 3 people: Innis, Durk and me.


I think it is a shame that people should be discouraged from contributing to FG but if thats the way you want to go then so be it.I happen to know as well as Scheduleviewer there is another data base program being used to create traffic files both of which I can't get to run on my Linux system.And the database programs make huge files that are at least 3 times as large as they need to be for not much time saving from the manual method.But if we want the AI system to be bogged down with huge files then fine.I will continue to use the manual method.Just I guess my work will not see the light of day.

Cheers
qf707
qf707
 
Posts: 134
Joined: Sun Feb 21, 2010 3:28 am

Re: 2.12 AI problems

Postby BecOzIcan » Mon Oct 21, 2013 1:11 pm

adrian wrote in Fri Sep 27, 2013 3:33 pm:I'd like to discourage editing of xmL files in favour of using the Scheduleviewer, which has the advantage of using a database backend and other useful features. I think currently most of the traffic comes from 3 people: Innis, Durk and me.
Cheers,


I am with Innis on this one as I already built my own XLS based traffic file generator and fail to see why I should start playing around with Python having no Dev background and a Win7 system :-)

Also, on the BackEnd DB topic and given my experience with traffic files (9 submitted via James since July) I'd rather restart a file from scratch than trying to identify the changes in routes/schedules/aircrafts made by an airline inside an existing ScheduleViewer file (My CPA file has 1466 services along 232 routes with 134 aircrafts) so I am not sure what benefits are there.

I have 5 more files to submit this month: Oman Air, Iceland Air, Brussels Airlines, LOT and Scandinavian (WIP); If manual files are no longer to be accepted just post here; I ll save the trouble of submitting and simply keep them available via MediaFire for those interested

Cheers
Ian
Current Projects: AI Traffic, Models & Liveries
User avatar
BecOzIcan
 
Posts: 961
Joined: Tue Oct 04, 2011 10:43 pm
Location: Sydney, NSW, Australia
Version: 2.12
OS: Win10

Re: 2.12 AI problems

Postby zakalawe » Mon Oct 21, 2013 3:57 pm

The first cut of the repository exists, imported from Git. SVN sure doesn't care where the files from and neither do I - so long as people agree which airlines they're doing, I can give anyone who asks commit access to the repo. If people want to interface their tooling directly to SVN that's fine with me (well, once they test their script!)

It also syncs the AI models dynamically, it's pretty nice. Should be appearing in Git soon, still some bugs to iron out.
zakalawe
 
Posts: 1149
Joined: Sat Jul 19, 2008 4:48 pm
Location: Edinburgh, Scotland
Callsign: G-ZKLW
Version: next
OS: Mac

Re: 2.12 AI problems

Postby adrian » Mon Oct 21, 2013 8:30 pm

BecOzIcan wrote in Mon Oct 21, 2013 1:11 pm:I am with Innis on this one as I already built my own XLS based traffic file generator and fail to see why I should start playing around with Python having no Dev background and a Win7 system :-)

Well, in order to exit the stone age? ;)

Also, on the BackEnd DB topic and given my experience with traffic files (9 submitted via James since July) I'd rather restart a file from scratch than trying to identify the changes in routes/schedules/aircrafts made by an airline inside an existing ScheduleViewer file (My CPA file has 1466 services along 232 routes with 134 aircrafts) so I am not sure what benefits are there.


I did 80 airlines in two months with Durk's help, and with the benefit that you can edit a couple of lines in a script and update the schedules anytime, in 15 minutes. Other than that, the GUI is useful in other ways too, like filtering duplicate flights and displaying aircraft fleets which lack a model/livery.
You can also use the powerful search facilities that a real database provides. XLS is stone age, please. When you're dealing with 10-20 airlines it's easy to say XLS works, try that with 80+ airlines.

I have 5 more files to submit this month: Oman Air, Iceland Air, Brussels Airlines, LOT and Scandinavian (WIP); If manual files are no longer to be accepted just post here; I ll save the trouble of submitting and simply keep them available via MediaFire for those interested

Cheers
Ian


What exactly are you doing to LOT and Scandinavian (I acknowledge that Brussels might not be complete)?
https://gitorious.org/fg-ai-flightplan/ ... s/LOT.conf
Are you aware these schedules exist?
Are you updating the schedules based on current year? Why not just edit a couple of lines in my Python tools and do it all in 15 minutes instead of manually trudging along? That's the real reason I wrote them, you know?
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: 2.12 AI problems

Postby BecOzIcan » Wed Oct 23, 2013 12:57 pm

Hey Adrian,

Well, in order to exit the stone age? ;)

Fortunately or not, Evolution is a byproduct of constraint, not one's conscious decision. Right now my tools are fit for purpose so I can comfortably stick to Jurassic age :D

You can also use the powerful search facilities that a real database provides. XLS is stone age, please. When you're dealing with 10-20 airlines it's easy to say XLS works, try that with 80+ airlines.

Believe me, working in IT I'd never recommend anyone to use XLS as replacement for a serious DB. It happens that I keep only one workbook by Airline so searching and sorting is not a problem.

What exactly are you doing to LOT and Scandinavian (I acknowledge that Brussels might not be complete)?
https://gitorious.org/fg-ai-flightplan/ ... s/LOT.conf
Are you aware these schedules exist?
Are you updating the schedules based on current year?

Nope, I didn't know ScheduleViewer existed and, not being on GIT, I can only rely on FGFS releases and forum discussions to know which files are worked on.

I just ingested the LOT file available in FGFS 2.12 (into my XLS converter, sorry) to have a quick look :

- You have 9 B737 registered, LOT has only 1 is left active, the rest has been replaced by Dreamliners which are not in 2.12s traffic file http://www.airfleets.net/flottecie/LOT% ... rlines.htm
- Simply looking up the first flight in the file (LOT270 EHAM EPWA), you have it running on Sundays ecept it doesn't http://info.flightmapper.net/flight/LOT ... nes_LO_270 (I couldn't double check on LOT site, timetabled are Err505 right now)
3. as indicated by Innis before, it looks like you are only handling WEEK schedules hence
- The LOT file packaged with FGFS 2.12 is 718 Kb in size where mine is a mere 203 Kb
- You have AI planes sitting around for a week -literally- and blocking ParkPos for nothing

At the end of the day I am not trying to dispute ScheduleViewer is a good tool and does the job. My only concern is that despite having produced traffic files for over 6 months and having discussed a lot on the forums about it, I discover/read about ScheduleViewer only now.

I sense from your post that it is de facto already the preferred tool for maintenance of traffic files; Shouldn't the AI traffic Wiki Page (Tools Section) be updated to reflect this? Right now it only mentions tools are 'in development' http://wiki.flightgear.org/Interactive_traffic

It would also be a good thing to provide instruction on installation/operation of your program if you are looking at attracting more participation.

Sorry for the long message, it shall be my last on the topic.

Take care
Ian
Current Projects: AI Traffic, Models & Liveries
User avatar
BecOzIcan
 
Posts: 961
Joined: Tue Oct 04, 2011 10:43 pm
Location: Sydney, NSW, Australia
Version: 2.12
OS: Win10

Re: 2.12 AI problems

Postby adrian » Wed Oct 23, 2013 3:48 pm

BecOzIcan wrote in Wed Oct 23, 2013 12:57 pm:Hey Adrian,

Nope, I didn't know ScheduleViewer existed and, not being on GIT, I can only rely on FGFS releases and forum discussions to know which files are worked on.


Please note this is not about the ScheduleViewer anymore, and it's about the 80 something airlines schedules extracted two years ago from public data. Duplicating that effort is wastefull, and has some other negative consequences (if you update only one company you are bound to have time slots conflicts, since all others are 2011/2012).

- You have 9 B737 registered, LOT has only 1 is left active, the rest has been replaced by Dreamliners which are not in 2.12s traffic file http://www.airfleets.net/flottecie/LOT% ... rlines.htm
- Simply looking up the first flight in the file (LOT270 EHAM EPWA), you have it running on Sundays ecept it doesn't http://info.flightmapper.net/flight/LOT ... nes_LO_270 (I couldn't double check on LOT site, timetabled are Err505 right now)
3. as indicated by Innis before, it looks like you are only handling WEEK schedules hence


You're not serious are you? The schedules and airline data are based on company published data for August 2011/April 2012. Are you seriously suggesting we update this every year? Take note that there are only a handful of volunteers running this project. If you want to start maintaining the schedules or airline data, you're welcome, but you have to use what already exists in order to have the best result. Does your XSL based tool check for duplicate flights with other airlines? (you know they appear on schedules, marked as such). I can give you access to the git repository if you want to start updating all airlines to todays schedules, but beware, you have to know how this works first. You also have to ask questions first, before deleting data. Malev has ceased operating since I did them, are you suggesting to remove them? Also you need to come with schedule data from the company, not from second hand sources like flightmapper, when we're talking an existing airline in our repo. If they are published, then it's simply a matter of minutes to edit the script (not ScheduleViewer, again) and re-generate the schedules.

- The LOT file packaged with FGFS 2.12 is 718 Kb in size where mine is a mere 203 Kb

I'll disregard this as irrelevant. My sources are company data, it may contain other flights flown under the POLLOT callsign.

- You have AI planes sitting around for a week -literally- and blocking ParkPos for nothing

Please report bugs the standard way: http://flightgear-bugs.googlecode.com/

At the end of the day I am not trying to dispute ScheduleViewer is a good tool and does the job. My only concern is that despite having produced traffic files for over 6 months and having discussed a lot on the forums about it, I discover/read about ScheduleViewer only now.

I sense from your post that it is de facto already the preferred tool for maintenance of traffic files; Shouldn't the AI traffic Wiki Page (Tools Section) be updated to reflect this? Right now it only mentions tools are 'in development' http://wiki.flightgear.org/Interactive_traffic

It would also be a good thing to provide instruction on installation/operation of your program if you are looking at attracting more participation.

Sorry for the long message, it shall be my last on the topic.

Take care
Ian


You seem to have misread my posts. I'm not trying to force you to use any specific tool. I'm just mentioning that FGScheduleViewer was built for a specific purpose, and you might profit from using it, as well as avoid a couple of pitfalls you don't know about. It's your choice whether you do or not, but the flight data submitted must be better than what already exists. Also, I'm not talking specifically about this application in my messages, but about the Python scripts which generate schedules based on public company provided data. This means very little effort on subsequent runs, although it took me a lot of time to develop them. It's unlikely that the schedule format provided by the airline changes a lot with time. If you are interested in this topic, read the forums threads dated 2011 here, and see that the development effort and the git repo was mentioned before, it's no secret. Again, if you wish to further develop our data, I can give you git commit access, and you can do this the proper way, rather than working alone disregarding previous work (which is recent, and has a significant number of airlines involved).
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: 2.12 AI problems

Postby zakalawe » Wed Oct 23, 2013 4:07 pm

Uh, just to reinforce what Ian wrote, I'd never heard of the scheduling scripts / tools / workflow either. People don't generally search the forum archives for such things - they look on the wiki, or ask on the mailing list. Obviously there is fairly complete solution developed, but equally you (Adrian) seem to be suggesting that updating it annually (or even more often) would be quite intensive. That doesn't make sense to me - if you've invested all this time and effort building a scripted solution, surely the whole point is so updating is easy / fast?

From my perspective I'm delighted to have Ian (and gooneybird)'s work because they are improving the models and adding traffic which is missing. If it needs to be integrated into some larger system going forwards, well, I guess we can deal with that - maybe there should be a README in AI/Traffic saying not to hand-edit the files? Or indeed an XML header comment generated by the tooling to the same effect.

Adrian, what do you propose going forwards for people who want to spend time now on updating their particular airlines to recent schedules? What format do you want the data in? And what is your time-frame for re-generating the traffic files from your tool with updated schedules?
zakalawe
 
Posts: 1149
Joined: Sat Jul 19, 2008 4:48 pm
Location: Edinburgh, Scotland
Callsign: G-ZKLW
Version: next
OS: Mac

Re: 2.12 AI problems

Postby adrian » Wed Oct 23, 2013 5:07 pm

Hi James,
Just to add to my point, I don't want to impose my workflow on anybody. But equally I don't want people duplicating what it already took me a lot of effort, and had good results in the past.
I'll post here again the git URL to the entire AI dedicated repo, for future reference: https://gitorious.org/fg-ai-flightplan
For anyone working on this, a pretty good start point would be cloning the repo, and start messing with the Python scripts and tools in order to use your data instead of our 2011 data. Also, keeping Durk in the loop is probably a good idea, since he has been doing this from the start, and all data formats, tools, workflows and other stuff are based on his original ideas.

Not updating the wiki is definetly my mistake, and I'll try to make up for that these days.
Please note that all work related to the AI schedules and airfleets is equally made by Durk too, and as far as I'm concerned Durk is head of the AI department, so he should make any calls regarding official policy, not me. I already made the offer that the git AI repository be transferred into Flightgear ownership, so that the project may have full control in the future, and all data and tools in it are officially supported and maintained by the project, instead of just Durk and me, who may at times be gone for a full year or more.

I equally welcome new contributions, but Gooneybird's work is not new, and of course without him our schedules would have gone unused for years, so thanks again for providing a face to the AI project, Gooneybird.
Otherwise, I encourage people to contribute, but please try to do that keeping in mind existing work, and try to keep up the same standards, and before commiting time on such work, try and ask what the common pitfalls are, because we have a lot of experience, having brought in over 80 airlines, with more than 70000 flights weekly.
Again, the workflow for updating existing schedules should be fairly simple, if the company data presentation has not changed in a radical way in the meantime. All my schedules came in an automated fashion from PDF schedules of the major alliances and individual companies where available. I have also extracted flight schedules for companies flying codeshares fleets under a different callsign, where that information was available in the PDF data. All the airline data is hand added by Durk or me, and we have even quoted our sources of information here, in the forums. With new schedules, replacing all this data should be a matter of 15 minutes per airline, and then of course sorting and filtering all the flights for duplicates, so we have as little as possible. We're perfectionists, but sometimes we just have to acknowledge that this is a lot of work for a small team, and just leave a couple of duplicates which may affect an airport or two, but might show up correctly in other parts of the world.
This about says it all.
Cheers,
Adrian
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: 2.12 AI problems

Postby adrian » Wed Oct 23, 2013 5:23 pm

zakalawe wrote in Wed Oct 23, 2013 4:07 pm:Obviously there is fairly complete solution developed, but equally you (Adrian) seem to be suggesting that updating it annually (or even more often) would be quite intensive. That doesn't make sense to me - if you've invested all this time and effort building a scripted solution, surely the whole point is so updating is easy / fast?


It's a misunderstanding, I probably worded it wrong. I just meant to say that updating these schedules should be made in a coordinated fashion using the same tools, and that all schedules should be updated at the same time, to avoid having traffic info from 2011 overstepping 2013 flights. The timeslots are presently those of 2011 for most of the airlines. Also, that we can't be expected to update it all every year, since we all have other projects, have to take some time away etc. We're just volunteers here, with limited time which can be spread over different things, see my radio code work, my texture work, my other related projects...
adrian
 
Posts: 362
Joined: Wed Sep 15, 2010 2:15 pm

Re: 2.12 AI problems

Postby gooneybird » Wed Oct 23, 2013 7:22 pm

I cannot see why all this was not emphasised in the many previous discussions on the forum about about AI traffic, many times I've asked for guidance on various aspects of the AI system with none forthcoming so naturally we have tried to find the best way forward ourselves.

It is no good saying this was all discussed on the mailing lists or IRC etc. it should all be filtered down to the forum where the majority of FlightGear users and contributors get their information.
Not many people wish to subscribe to a mailing list where 90% posts are way above their technical understanding but are quite happy to contribute in other areas but at the moment it feels that those that frequent only the forum are not getting the full picture regards the AI system.


adrian wrote in Wed Oct 23, 2013 5:07 pm:I equally welcome new contributions, but Gooneybird's work is not new,


Traffic files: (all done the hard way in Notepad++)

Blue Islands
Flybe (old file only had some EHAM flights)
EasyJet (old file only had some EHAM flights)
GermanWings
Monarch Airlines
Wizz Air

AI models:

Boeing 717
A319
A320
A321
Beech 200
ERJ 145

Plus appromimately 300 AI liveries

So I can't see how you can say nothing of mine is not new. :| :wink:
Currently running 2018.1 but if it doesn't stop crashing I'm taking up tiddlywinks!
User avatar
gooneybird
 
Posts: 2912
Joined: Sat May 31, 2008 1:57 pm
Location: Essex, UK
Callsign: G-OONY
Version: 2018.1
OS: win 7 64

Next

Return to AI Traffic

Who is online

Users browsing this forum: No registered users and 1 guest