Board index FlightGear The FlightGear project News

15 years of FlightGear, screenshot contest

News from the ever evolving world of FlightGear.

Re: 15 years of FlightGear, screenshot contest

Postby gooneybird » Wed Aug 01, 2012 11:09 am

How does this voting system work :?:
I gave up after a few minutes after getting the same pair of images three times :?
My Github repository (mostly AI stuff) https://github.com/gooneybird47
User avatar
gooneybird
 
Posts: 3046
Joined: Sat May 31, 2008 2:57 pm

Re: 15 years of FlightGear, screenshot contest

Postby Thorsten » Wed Aug 01, 2012 11:12 am

I guess it's to make sure you can't easily bias the voting by casting 1000 votes for your own work, since you have to cast votes for lots of other stuff alongside. So, during the course of hundred decisions your preferences drift slowly to the front, and the disfavoured drift to the back.

To be useful, this system requries us to cast a lot of votes though, in order to image your preferences you have to have seen every screenshot 2-3 times, which means something like 200 votes.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: 15 years of FlightGear, screenshot contest

Postby Gijs » Wed Aug 01, 2012 11:15 am

gooneybird wrote in Wed Aug 01, 2012 11:09 am:I gave up after a few minutes after getting the same pair of images three times :?
Hm, that seems to be a little bug in the system. Please close your browser and try again when that happens.

I'll look for a real solution ;-)
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9544
Joined: Tue Jul 03, 2007 3:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: 15 years of FlightGear, screenshot contest

Postby gooneybird » Wed Aug 01, 2012 11:41 am

Just tried it for a little longer this time with only one double duplicate.
I assume the images are displayed on a random basis so these duplicates would naturally occur anyway.
My Github repository (mostly AI stuff) https://github.com/gooneybird47
User avatar
gooneybird
 
Posts: 3046
Joined: Sat May 31, 2008 2:57 pm

Re: 15 years of FlightGear, screenshot contest

Postby islandmonkey » Wed Aug 01, 2012 8:18 pm

Hi Gjis, could you add a option to skip some image options because I found some images both not to my tastes?

EDIT: Oh wait, stupid me - I can just click refresh :roll: .
User avatar
islandmonkey
 
Posts: 786
Joined: Mon Jan 30, 2012 9:51 pm
Location: EGCN (uni), EGHI (home)
Callsign: G-MNKY
OS: Ubuntu 20.04

Re: 15 years of FlightGear, screenshot contest

Postby Gijs » Wed Aug 01, 2012 8:52 pm

The whole idea of this voting system is that you always pick one. Due to the large number of pairs, the most liked shots will end up on top.
Got another bug to fix ;-)
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9544
Joined: Tue Jul 03, 2007 3:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: 15 years of FlightGear, screenshot contest

Postby Madbyte » Wed Aug 01, 2012 9:00 pm

I agree. There should be a skip option. With the actual method I have to vote for pics that I don't like. And I found that some pics appear with more frequency than others, and that's not fair because someone may quit without see all the pics in the contest.

EDIT: Just an idea: the pics that you already picked must be excluded from the next pairs. And there should be a counter for the remainder pics.
Madbyte
 
Posts: 19
Joined: Mon Apr 16, 2012 6:20 am
OS: Ubuntu 15.10 64

Re: 15 years of FlightGear, screenshot contest

Postby Gijs » Wed Aug 01, 2012 9:28 pm

Just read my reply to Gooneybird a little higher on this page. Pictures re-appearing shortly after eachother is a bug, but it does not matter for the overall scores. You currently see each image twice, before you've had all images. The intermediate results show us that all pictures have been viewed almost an equal number of times. We're talking about some 5% differences at most here, while voting has only just begun.

The fact that you have to vote for pics that you don't like is part of the system. Due to the large number of voters, a couple of votes for an ugly picture does not make it a winner. Next time, that same ugly image is presented next to a nicer image and thus it will loose. Such things will cancel out due to the statistical properties. People do not need to see all pictures to make the system fair.

I don't have the time to explain the entire system right now, but we've been going through it with several people and we agreed that this is the fairest system for such a contest.
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9544
Joined: Tue Jul 03, 2007 3:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: 15 years of FlightGear, screenshot contest

Postby durk » Wed Aug 01, 2012 11:29 pm

Ocassionally being forced to choose between two "ugly" pictures is essential for the system. In essence, it's these votes that determine which pictures end up being last and which one ends up in the one but last position. Next time you'd see the ugly one next to a beautiful picture, you'd certainly not vote for it. That's the idea. It might not be as honourable as choosing between the best vs. second best picture, it is equally important to do so. That's why we also present these combinations of images you may not like so much, and don't allow you to skip them.

Likewise, we present random combinations, because we need to have as many different combinations as possible, in order to establish a clear rank-order of all the images. We do sample without replacement, but the occasional repetition of images may occur, because we balance them between the left and right side of the display. For different people this happens with different images, but that is part of the procedure. Across the boardwalk, each image is presented approx. equally often, resulting in a very well defined rank-order of images.

Cheers,
Durk
durk
 
Posts: 354
Joined: Mon Nov 17, 2008 2:01 pm
Location: Ghent, Belgium
Callsign: PH-DRK
Version: git
OS: linux

Re: 15 years of FlightGear, screenshot contest

Postby Madbyte » Thu Aug 02, 2012 1:05 am

Are you using the Condorcet Criterion for the vote method?

I think this... http://en.wikipedia.org/wiki/Pairwise_comparison
Madbyte
 
Posts: 19
Joined: Mon Apr 16, 2012 6:20 am
OS: Ubuntu 15.10 64

Re: 15 years of FlightGear, screenshot contest

Postby kyokoyama » Thu Aug 02, 2012 1:37 am

Sounds right.

Just one thing I noticed... I feel like some of the choices are really unfair because some of the two picture choices are from completely different categories (i.e. -the user is forced to choose between a scenery and a weather phenomenon or aviation art and cockpit picture instead of two pictures from the same category).
This sounds fine in theory, but doesn't that make the whole categorization process redundant?
Look for "B-BIRD" "N127KY" or "AVA0004" -that's me.

Despite having over 1700 posts here, I am not even close to being the most skilled guy here... I'm just words and bad drawing, not experience. :P
kyokoyama
 
Posts: 1981
Joined: Sun May 03, 2009 3:16 am
Location: Earth
Callsign: B-BIRD, N127KY
Version: 2.12.1
OS: Windows Vista

Re: 15 years of FlightGear, screenshot contest

Postby Thorsten » Thu Aug 02, 2012 7:12 am

This sounds fine in theory, but doesn't that make the whole categorization process redundant?


Yes and no.

Yes if typical users for example have a systematic preference for unmodified against modified screenshots - in this case, for a limited number of votes, 'aviation art' will typically be presented against any other category and get zero votes, so you never sort aviation art properly, that happens only when two aviation art images are compared against each other.

Of course, for a sufficiently large number of votes, you will eventually also see pairs of aviation art compared against each other, allowing you to bias-free sort inside the category properly. But I think you need to go well above 1000 votes per user to get rid of that bias.
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

Re: 15 years of FlightGear, screenshot contest

Postby curt » Thu Aug 02, 2012 8:17 pm

I agree that it can be frustrating to have to pick between two pretty weak images when you hate them both equally. And similarly frustrating to have to pick between two really strong images that each have great qualities, or two radically different images that are both strong in what they are trying to convey. But keep in mind that the results are averaged statistically over 1000's or 10's of thousands of individual contests. We have had over 12,000 mini contests (pairings) so far in just the first 24 hours. So it shouldn't hurt the process by very much if a voter makes a mistake, or can't decide and throws up their hands and makes an arbitrary choice.

For those that enjoy watching international soccer tournaments, they usually start out with round robin play. In our case a "match" costs very little, so conceptually imagine all the teams in a single giant group playing endless matches between each other where each match randomly selects 2 teams from the group to compete against each other. If a team loses a match (even due to a bad referee), it is not catastrophic because in the course of the tournament, the other strong teams are also likely to lose a few matches due to similar factors. Over some period of time, and thousands of matches the better teams will tend to accumulate a better win/loss record. I may not enjoy watching two lousy teams play for 90 minutes, and I may hate to see two great teams play and have one of them forced to lose. But every team must play every other team (statistically) for the ultimate results to be fair and make sense. And ultimately it is the strongest teams that rise to the top and generally the teams will sort out in the standings very quickly. And then at the end rather than playing a single elimination tournament between the teams that make it through group play, we just use the group standings to determine the winners -- and we can do that because all the teams are in one giant group. (And we can still pull out the best south american team, or the best european team, or the best team that wears long sleeve jerseys, or the best team that wears adidas boots if we want to.)

And at the end if there are 2 images vying for grand champion that are one vote apart and someone makes a voting mistake to swing it one way or another, then that's part of the fun and why we play the game in the first place. We could have easily (well it would have been time consuming) ranked the images ourselves, but we thought this statistical pairing method would be really interesting to try, and I think Gijs had a lot of fun coding it up.

We still keep track of categories so at the end of the voting period we will know the best images in each category, and we will have an overall ranking so we can have an overall winner.
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: 15 years of FlightGear, screenshot contest

Postby curt » Mon Aug 06, 2012 5:45 pm

Here's a quick status update on the screenshot contest. So far there have been 24077 individual votes cast. The current leader has won 3 more "pairing contests" than the 2nd place image. The 3rd place image has won exactly the same number of contests as the 2nd place image, but due to the randomization has played in a few more match ups so it's winning percentage is slightly lower.

I'd love to show the results so far, but the consensus among the folks setting up and advising on the structure of the contest decided they didn't want to bias the results. So instead I've been posting a few of the stronger images to facebook/google+ as teasers.

Generally, the strong images sort themselves right to the top of the standings and the weak images right to the bottom. But there have been some surprises (for me.) I have my own preferences and it's interesting to see how those are sometimes reflected in the voting and sometimes not. I've also been surprised to see a lot of movement in the overall standings. I had imagined the rank order would lock in pretty quick and then solidify as more and more votes are cast. But the reality is much more dynamic (and interesting) and we've seen some images move to the top after being pretty far back, we've seen some images switch orders even a couple times.

If you haven't had a chance to look through all the great contest submissions, I encourage you to run over to the voting page and cast votes. We have 12 days left, but the system works better the more you vote and the more days you vote. There are some really great shots that are a fun to look through:

http://liveries.flightgear.org/screensh ... t/vote.php
curt
Administrator
 
Posts: 1168
Joined: Thu Jan 01, 1970 1:00 am
Location: Minneapolis, MN

Re: 15 years of FlightGear, screenshot contest

Postby Thorsten » Tue Aug 07, 2012 7:45 am

I've also been surprised to see a lot of movement in the overall standings. I had imagined the rank order would lock in pretty quick and then solidify as more and more votes are cast.


That's to be expected. The voting system is based on bubble-sort, after all, which needs n^2 steps to sort through n elements. Which means that I alone would need to cast more than 10.000 votes to sort the list according to my preferences, after which the ordering locks.

24.000 votes from a number of users are not nearly enough to get rid of sorting artefacts, which is why things are expected to be on the move for a while...
Thorsten
 
Posts: 12490
Joined: Mon Nov 02, 2009 9:33 am

PreviousNext

Return to News

Who is online

Users browsing this forum: No registered users and 4 guests