Board index FlightGear Support Multiplayer

Jitter when having two flight gear instances on local machin

Trouble getting online, setting up a server?
Forum rules
In order to help you, we need to know a lot of information. Make sure to include answers to at least the following questions in your initial post.

- what OS (Windows Xp/Vista, Mac etc.) are you running?
- what FlightGear version do you use?
- copy&paste your commandline.

Please, also see Requesting Technical Help.

Note: If you did not get a reponse, even after 7 days, you may want to check out the FlightGear mailing lists to ask your question there.

Re: Jitter when having two flight gear instances on local ma

Postby ma15rysi » Mon Dec 21, 2020 10:52 am

jano wrote in Mon Dec 21, 2020 12:00 am:No it's not from the camera, it's only because lack of time synch is apparent when you got speed, if the 2 planes simulink driven are smooth when seen from an other fg session, you can use pilot list (menu multiplayer) and make the view attached to one of the mp plane, and see if they keep a good synch this way.

I also had that thought and tried it but unfortunately it didn't work either...

jano wrote in Mon Dec 21, 2020 12:00 am:if you try to create a mp protocol udp packet with an udp block in simulink, the only really needed props to transmit are the position, orientation, and a timestamp witch should be the same for the 2 planes each frame. the velocity is only needed if you need to predict the plane's position in the futur, but there's a way to avoid it.

Simulink uses 6DoF input data for sending to FG (i.e. 3 orientation angles fo the plane and 3 positions values) and a sampling rate. I use the same samping rate for both planes in Simulink and it should be the same frequency as for FG communication...I set everything to 30 Hz.
ma15rysi
 
Posts: 10
Joined: Fri Dec 11, 2020 6:22 pm

Re: Jitter when having two flight gear instances on local ma

Postby jano » Mon Dec 21, 2020 2:25 pm

here's a quick drawing to help explain what happens:

Image

you expect simulink to send the data, then fg to run once, next simulink to run then fg etc like in the timeline above.

But that can be only if all programs involved are exactly running at 30 Hz in a synchronized way, If it works, you don't need to use a precise timestamp for each fdm state, the way they are running together ensure the synchronisation.

but FG respect only the 30Hz "as much as it can" given the variable load from doing 3d stuff, so some frame are longer, wich lead to a shift in the timing, you then have lost the sync you were counting on, and got jitterly planes for multiple simulink driven planes.

the best way to see this is to capture the network traffic involved here with a high multiplay sending rate to ensure each fg run send a mp packet (wireshark is your friend)

We have two solutions, we can try to run FG synchronised to simulink, or we can use precise timestamp for the fdm states from simulink.

1) the "run in synch way"

- simulink part
I'm curious to see how precise simulink is in its run, (wireshark log welcome :) ) and if the 30Hz is very good, is the sending time adjusted to the wall clock (eg there are frame sent at exactly the full seconds, or the sending time depend the moment you start simulink. Can your check if different simulink run send the packet in nearly the same time windows inside a second ?

-fg part
For now, FG can't run at some precise decimal part of the wall clock second (eg 0.1s 0.2s 0.3s etc for 10Hz) for once because the starting time will add an offset, and mainly because huge frame will shift the reference (eg: 0.1s 0.2s then 0.44s 0.54s ) so we can't expect a repetably timeline inside each second.
with your current FG, you can first try to play with sim/model-hz, wich give fg the "step" it is allowed to have for frame time, by default it will be 120hz, either ask for 30 as for all the other props (and with the frame rate limitation you don't have in the lines you reported):
Code: Select all
 --prop:/sim/model-hz=30 --prop:/sim/frame-rate-throttle-hz=30


or call for lot more for model-hz, like 1000 :) and see if it's better.



if simulink is able to be consistent relative to the wall clock, i can send here a patch to have fg more fixed to a time grid (if you compile fg yourself), this is part of a solution to address a bug in mp protocol sending rate in the way.

ther could be a solution to trigger a fg run each time something new come from simulink, but the racing condition between 2 fg sessions would probably lead to jitter too, depending which FG run first each step so imho the best are time synch based on wall clock, or solution 2 :P

2) using a precise timestamp

with each fdm state having a precise timestamp, we don't need to have a strict timeline for simulink and fg, we only need to do a bit of prediction, to have the plane position adjusted to where it should be at FG time, mainly adding a (speed * time) difference vector. that's what the mp code do for multiplayer pilots, so we're back to either make a mp protocol from simulink (but for this i can't help, so obviously it's the solution i prefer as the job in fg is already done :) ) or change the time used in the netfdm protocol to have a precise timestamp, and add position prediction in fg netfdm (wich could improve master-slave fg sessions)

It's obvious for this kind of "real time mode", that the different pc you are using need to have clocks in sync, so ntp or ptp are your friend ...

hope this is a bit more clear for those reading this, this is always a bit difficult to explain what is evident to me :P not in my native language and without simulink .
Last edited by jano on Mon Dec 21, 2020 9:08 pm, edited 3 times in total.
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Jitter when having two flight gear instances on local ma

Postby Hooray » Mon Dec 21, 2020 2:30 pm

that's a very good write-up, and probably should be copied to the wiki so that this kind of information is preserved :-)
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Jitter when having two flight gear instances on local ma

Postby jano » Tue Dec 22, 2020 12:52 am

A first try is ready, if you're able to compile FG, it allow FG to start each frame very close to the time grid, doing something similar to the "what it should do" above.
you will be able to move the grid with an offset, if you want to ensure a bit of delay between simulink, fg1 and fg2.

simulink <1/3 frame> fg1 <1/frame> fg2

the planes should be in sync in the fg2 session, apart if some frame are too long, but this can be addressed later.

tell me if i need to provide a patch, a branch in my flightgear clone, or if you will wait till it come in the official fg, which can take time.

Image

difference between wanted timestamp, and the pc clock we had, 0 to 1 represent one grid step (here 1/120 hz) most of the points are very close to 0 whe fps limiting, and in clouds without...
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Jitter when having two flight gear instances on local ma

Postby ma15rysi » Wed Dec 23, 2020 1:42 pm

I played around with wireshark a bit and tried to do some analysis. I filtered the data for udp.dstport == 5001/5002 and for udp.dstport == 5502/5503 to see the data stream that is going from Simulink to FG and from FGMP server to my FG instances and found out the following:
Neither Simulink nor FG are stable in the sending frequency, i.e. that it is totally oscillyting around 30Hz. I also found out that the data that is send from Simulink is way more unstable in its timestamps (e.g. I set the sending frequency to 10Hz and then two packages were sent every 0.2 seconds instead of one package every 0.1 second). The data stream between FGMP server is way better under that aspect since it sends data almost every 0.1 seconds.

jano wrote in Mon Dec 21, 2020 2:25 pm:
Code: Select all
 --prop:/sim/model-hz=30 --prop:/sim/frame-rate-throttle-hz=30


or call for lot more for model-hz, like 1000 :) and see if it's better.

Unfortunately, this didn't help :(
ma15rysi
 
Posts: 10
Joined: Fri Dec 11, 2020 6:22 pm

Re: Jitter when having two flight gear instances on local ma

Postby Hooray » Wed Dec 23, 2020 1:46 pm

If needed, FlightGear can be modified to use a background thread and process a read-only mirror/copy of the property tree to deal with your I/O and timing requirements.
Keywords relevant in that context would be "SGSocketUDP", "SGThread" and "SGSubsystemMgr"

Speaking in general, things like Nasal will make the frame rate/spacing increasingly non-deterministic, so that subsystems get different time slices to their own processing work.
Please don't send support requests by PM, instead post your questions on the forum so that all users can contribute and benefit
Thanks & all the best,
Hooray
Help write next month's newsletter !
pui2canvas | MapStructure | Canvas Development | Programming resources
Hooray
 
Posts: 12707
Joined: Tue Mar 25, 2008 9:40 am
Pronouns: THOU

Re: Jitter when having two flight gear instances on local ma

Postby jano » Wed Dec 23, 2020 2:59 pm

ok, there's no good news here :(

I think simulink can do better than this, are you using something like a "real time" run, there are option to choose clock behaviour if i understood well the bit of doc i read. is it 2 packets together for only a plane simulated ?
it's supposed to be precise for the run start timestamp, sending udp when it have time, witch should be each iteration if simulink is not overloaded.
Maybe you can find some help for the simulink part on a dedicaced forum ?

if you are using simulink and fg on the same pc, (and wireshark) be sure each of them run on a different cpu core, or different pc, simulink need a core alone in real time mode imho, and fg is better alone too, and by default it's core affinity is to the first core so i need to change core affinity when having more than one running.


the solutions lefts are the same as before, and 2 of 3 need you to compile fg, quite easy if you are using linux

-make simulink be a good boy and respect the sending rate, here the patched FG is fired in a narrow time window most of the time, you'll have some jitter but only for some frames.
-change the time used in the simulink netfdm and replace the unix time with a precise timestamp, i don't know how easy this is, and i don't have the code ready in fg for it now and the following days won't be productive :) .
- write a mp protocol block in simulink, but that can take even more time :)

I'll ask on the devel list too, if by chance there are simulink expert.
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Jitter when having two flight gear instances on local ma

Postby Gijs » Thu Dec 24, 2020 4:49 pm

Nice work on investigating Jean! I can have another closer look at the Simulink-side of things, but only after the coming days (I try to stay away from my computer as much as possible for a bit).

Gijs
Airports: EHAM, EHLE, KSFO
Aircraft: 747-400
User avatar
Gijs
Moderator
 
Posts: 9544
Joined: Tue Jul 03, 2007 3:55 pm
Location: Delft, the Netherlands
Callsign: PH-GYS
Version: Git
OS: Windows 10

Re: Jitter when having two flight gear instances on local ma

Postby jano » Thu Dec 24, 2020 9:26 pm

the result for FG from wireshark, exporting the analyse in .csv + gnuplot:

Image

green is running throttle hz at 30 and modelHz at 120, we jump to the 120hz step time to time, other are both at 30, with different offset
vertical scale is a time of 1/30 sec from 0 to 1. 0 (and 1) is on the theorical 30Hz time.

the result is quite good without a windows restricting fg timestamp around the theorical value, but adding this that will wait some days, for sure :)
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Jitter when having two flight gear instances on local ma

Postby jano » Sun Dec 27, 2020 3:17 am

some news , after installing simulink for one month trial :)

I tried the hl20 model and:

it's quite easy to change the unix timestamp to something else, keeping it as unsigned integer is fine, we just need to have a timestamp in milliseconds, or microseconds

i suggest to change the input to "cur time" in the pack netfdm, to something like the simulation time * 1 000 000 and applying a mod 3 600 000 000, to keep it under 3600 secondes and not go out of the uint32 (if i did the calcul right :) ) that will provide us a good timestamp and it's only a basic exercise in simulink :)
a quick hack would be to use fg as intermediate step, to send mp protocol with reliable timestamp to a 3rd fg session where they should be in synch, making this used in netfdm inside fg will need more time...

a good exercise would be to create the mp protocol from matlab/simulink making it simpler, but that's probably something outside your available time

you're right simulink is maintaining 30hz on average, but not stable enough to synch FG only relying on this, the precision i got here is inside a 50ms windows, after a pic at startup (i don't start FG so the missing entry from FG slow down simulink until it decide it will not wait anymore :) )
Simulink is stressing a bit my old pc, and you can maybe have a better result. There's the possibility to use a "real time" mode too, but i' don't know if this would be way better. (.csv analyse log from wireshark welcome, only the packet number, and the unix time is needed in the log)

If the fg visualisation is not needed when you run the simulation, recording the simulink output in wireshark, and replaying them at 30pps with tcpreplay works very well the time windows is within 0.2ms, it could be a way to make nice video, with only the change in time management in fg.

anyway that's a good topic to push a bit FG , simulink and their network link to the limit :) and probably the base for a wiki page !
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Jitter when having two flight gear instances on local ma

Postby jano » Wed Jan 06, 2021 2:25 am

here's a video, two planes, (in fact it's the same plane duplicated) with 2 fg sessions using external fdm.



it's not perfect for all the run, but i don't have the jitter , except for times FG load stuff on an antic HDD.
It was a lot of work:
- simulink (here) can't give a steady 30fps for the net-fdm packets, so i recorded one session with wireshak, duplicated with a bit of time change and merged the 2 to simulate 2 planes from simulink.

on linux the following line:
Code: Select all
tcpreplay -K -p 30 -l 100 -i enp0s10 --pps-multi=2 deuxrun1.pcapng

give a steady 30 pps, 2 packet each time, this give the perfect sending source :) (nearly perfect, i can't have it starting at a precise time, so you need multiple runs to have some where the tcp=>fg1 => fg2 is respected.)

I used a patched FG (merge request here to run the 2 FG session synched, one being put 16ms late from the other.

on the left screen, we have both planes in the same position, the delay allow the mp plane position arrive before our update, and the right session got the mp plane position one frame later, hence the mp plane is behind for 33ms flight time.
I struggled with the mp protocol send rate not being respected, took me a lot of time to manage a send each frame playing with the properties

this was with the solution 1, synchronisations by running in synch, and the simulink plane don't send the velocity.
having something easier to set up, need the vélocity from simulink, and a precise timestamp (present here but not used) and will need more time.

I won't try to do the mp protocol packet from simulink, but it sure is a simple solution to the problem, for the simulink experts ...

jano
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Re: Jitter when having two flight gear instances on local ma

Postby jano » Fri Oct 22, 2021 7:44 am

Here's the wiki page with a summarize of the command line used:
the wiki page about a simulink experience
jano
 
Posts: 221
Joined: Fri Nov 30, 2007 12:32 am
Location: france
Callsign: jano
Version: git
OS: debian SID

Previous

Return to Multiplayer

Who is online

Users browsing this forum: No registered users and 2 guests