Board index FlightGear Development Spaceflight

Space Shuttle : Massive Failure/Emergency Deorbit report

Discussion about development and usage of spacecraft

Space Shuttle : Massive Failure/Emergency Deorbit report

Postby GinGin » Fri Jan 31, 2020 11:22 pm

I had so much fun lately playing with extreme scenario and failures that I wanted to do a proper sum up of it.
There are around 50 pictures,it might be a bit long to load the thread if you have a low bandwidth like me, but I think it is worth it


I condensed and highlighted some part of the ascent pocket checklist in order to have a good guide in order to follow the failure checklists



It is available here, and it is better to follow the thread


https://drive.google.com/open?id=1uZPOso6LROCYFCFIn79bMmvJxpdQ1-ab


A bit more infos about checks here:

http://wiki.flightgear.org/Flying_the_Shuttle_-_Space_Shuttle_Checklists

And normal Mission Phases there:

http://wiki.flightgear.org/Space_Shuttle#Mission_phases





Let's start our terrific journey.
Another forecasted mission, months if not years of training for the crew.
Veteran Commander, first flight for the Pilot and 2 Mission Specialists.

No payload for the ascent, aim was to take back a so called " observation satellite" for maintenance.
Launch at Sunrise, for a polar Orbit.

Cavok over the pad.

Image



Last 6 mn before Launch, APU are fired up, and last checks accomplished.

Image

Image



Go for Launch, and final coutdown before the 3 G's danger zone.
That was an almost perfect first stage, Atlantis already way above our heads.

Image





Entering the second stage on schedule, heads still down

Image



After 8mn and 30 seconds of roller coaster, MECO time and ET separation
No need for OMS 1, next meeting into the check for OMS 2

Image




End of propellant Dump, annonciated among other things by the body flap light extinction

Image



Approaching Higher latitudes, and what a show of lights

Image


Pre OMS 2 actions are performed ( APU and MPS shutdown, Umbilical doors closure, ..)
Burn is loaded into the computer, Ignition in 17 mn to circularize the orbit around 130 Nm

Image


Splendid Scenic while waiting...

Image



For sure, not waiting for that shrill Alarm, and all those lights shining on the Caution and Warning center panel.
Amongst them, MAIN BUS UNDERVOLT is quite bad

Image


Time to take the first checklist in the doc I linked, page 2, MN BUS UNDERVOLS/FC VOLTS and to have a look like suggested into the BFS , system management , sum up 1 ( same troubleshooting page for all failures whith only the BFS as system management GPC)

Image


We can see three things that are not normal.
It concerns fuel cell 2 and Main bus B.
Low voltage for FC 2 and MnB ( 6.1 V) and crazy amps ( superior at 1500 A)
Probably a short, but where ?

Let's enter into the check.
I highlighted in Green the first two conditionnal choices, either we are in one case or the other, and then we follow what is written below it.

For us, it is the first case ( Mn Bus and Fc volts below 26.6 V, and amps superior at 480 )
We don't have bus tied to each other, so no need for step 2 and 3
Step 4,5 and 6 would be to unload the Payload bus from fuel cell 2 if we had a payload ( Space Hab, Satelitte ..) , but it is not our case

Image


Step 7 is for us.
We will remove the FC 2 power from the essential bus 2 and Main bus B to see where is the short

Image



Result here :

Image


Fuel cell voltage recovered ( 32.5), no volts on Mn B and essential bus powered ( by main A and C buses, triple redundancy)
Also amps is now zero, as fuel cell 2 is not connected to any main buses.

Verdict: Main Bus B short

Impossible to tie it to another fuel cell, bus is lost and as directed by the step 10, we will have to perform Main B bus loss action

That means we will have to reorganize all the systems that were controlled by Main bus B and AC2 coming from that bus. ( Everything in the cockpit where it is written Mn B or AC 2 for power source)
In reality, it was possible to put cable into the cockpit to tie AC lost bus to a good AC bus, quite crazy maintenance action :)



Let's have a look on that nice sunset before diving into the next checklist

Image




Page 3 of our checklist now.
MNB DA2 (Entire BUs)(Includes AC2)

That means we lost all the Main Bus B ( we could have lost just a part of it, a lot of subdivisions behind the term Main Bus) and the AC bus associated ( AC2)
Let's go for some cockpit reconfiguration.

Step 1 for a back up valve that will allow Remaining Gaseous Hydrogen to be vented outside ( we need to be sure that this valve will be opened in case of another failure that would prevent it to be opened)
Step 2 to change the alimentation source of the boiler Heater of APU 1
Step 4 to turn off AC bus sensor. That sensor allows AC bus to be shed automatically in case of overload ( no need for it now that we don't have AC 2 anymore)

And Step 5 , telling us to shutdown FC2 as we don't have any use of it.
Cue Card means that we can directly look to some Papers in the cockpit near the electric panel to read the action ( no need to dive into the book, that was used for time efficiency in case of important failure before reaching orbit)

Cue card there:

Image


Let's go then.
No step 2 ( bus tie) as bus B is short, it would lead to bad things ( shortage of other main buses tied to the B )
Step 3 and 4 to stop the FC and close the reactant alimentation ( O2 and H2)

Image


Step 5 direct us to LOSS of 1 FC PWRDWN
We have 2 FC remaining, we will have to shutdown some systems and reorganized a bit to avoid unnecessary power consumption.


Let's meet on Page 4 of the checks.
It is for loss of either 1 FES ( Flash evaporator) or 1 fuel cell.
Step 2 and 3 tell us to dim the cockpit and use just one screen for the computer and 2 for instruments ( PFD and engine settings for example)
Step 5 and 6 are for navaids shutting down.

Step 7 through 12 are for ECLSS ( Environment, conditionning and life support system) reorganisation.
We already mainly did it before.
We make sure we have on IMU fan working and that all freon cooling sytems and evaporator heaters are not powered by AC 2


Image

End of that check, rest of it is in case of Abort once around, or if we were already in orbit, plus some changes for entry etc, not for us, at least not now ;)



View of our reorganized cockpit

Image






After some Nasa analysis and crew concertation, decision was made to continue the mission at least post OMS 2
Loss of 1 Main Bus is not critical, as Shuttle is supposed to work with only one FC with heavy load shedding, and as we have no payload to power, we can continue the mission futher
However, only the critical objectives will be accomplished, and mission time duration will be decrease at its minimum.

It happened a couple of times in real ( STS 2, 83) ( Fuel cell problem)
You can have a look there if you are interested

https://spaceflightblunders.wordpress.com/2017/05/24/in-space-no-one-should-see-you-belch/





Let's continue.
Next step, OMS 2 burn and nominal checklist for Post insertion.

Awkward monitoring

Image

Image




Safely on Orbit now, 40 mn after Launch and a Bus failure.
130 Nmish above the Earth

Image



Another unpleasant alarm is ringing.
What now....
New alarm for FUELL CELL PUMP
That is linked to a fuel cell pump that is used to cool it, with liquid circulating around the fuel cell, cooling it and exchanging its heat with freon loop.
Not good then.
We can see the little ( delta P) below FC 1, with Stack temperature increasing.

Image




Checklist page 5, FC COOL P
Pressure to the cooling sytem is lost, no more cooling liquid movement.
Step1: We can Tie the Main A Bus to the good one ( Main C ) to not lost systems powered by main A when we will shutdown the second fuel cell.


Image



Then another conditionnal in green , either cooling Pump Pressure high or low.
For us , it is low ( 0) as we lost the pump.
Step 3 then, another FC Shutdown.



Checklist page 6, 2nd FC SHUTDN
First a step that is not written there, I untied Mn A and Mn C
As we will shut down a cell, load will be heavy on just ONE cell to supply 2 main buses ( voltage below 24 V hard lower boundary)
It works in the sim with a lot of shedding, but it is very tight.

In reality, it was Ok ish, even with 3 mains buses on a single FC because they could open circuit breaker, and shed many more secondary systems not modelled here ( useless stuff for a sim, but that take power, like rcs breaker, water tank heaters etc)
To sum up, it is better to leave on the remaining FC just one main bus and then to try to tie them with other good buses afterwards ( if we don't lose all the screens, it is ok :) )

Step 5 and 6, like we did before, we disconnect the cell from its essential and main bus, then we shut if off and secure the reactant alimentation.


Image


Step 7, go to LOSS OF 2nd FC (PWRDN)
We will have to shed and check even more systems now that we just have one fuel cell remaining,one Main Bus ( max two with bus tie and chance) and one AC Bus .
That is becoming a quite critical situation.



A view on the very degraded cockpit
Image



Electrical sum up

FC 3, Main bus C and AC 3 are working ok
Essential buses one, two and three also, thanks to the triple redundancy

You can note the total amps ( 365) for 10 ish kw, max sustained of 12
We will try to decrease that with the next checklist

Image





Page 7 for the next checklist, LOSS OF 2nd FC
It will be very similar to what we see before.
To check that some systems are down, to check that other ones are well powered by Main C or AC3, and to see which systems will be inoperative ( very few thanks to redundancy)
And mainly to shed some energy ogre systems ( like fans, hydraulic pumps,...)

Step 2 for low use of screens ( we don't have the choice anyway)
Step 3 and 4 to check that navaids are off.

MS for mission specialist task (page 11 of the check , mainly circuit breakers to open, and APU/OMS/RCS/Hydraulics Heaters to deactivate)

Step 5 for dimming the light ( already done)
Step 7 to stop the cabin fan ( gonna get hot inside the Shuttle)

Step 9: Important one.
We will shut down some GPC computer.
We need to reorganize the computer buses on the only GPC that will remain, the one
More info and links on GPC handling stuffs here:

http://wiki.flightgear.org/Flying_the_Shuttle_-_Launch_And_Post_Insertion_Advanced#Post_Insertion:_Advanced_GPC_reconfiguration

Done, every buses on GPC 1

Image



Step 12: Shutting down GPC 2,3 and 4. Powering off 2 and 3 ( 4 will be use for a minimum of redundancy during entry, 5 for the backup computer)


Image



Step 21 ( steps before are for on Orbit reconfiguration, not for us)
MPS engine power already off, it was part of post meco actions ( and huge , huge power demand on fuel cells)

Step 23 through 32 are for ECLSS reconfiguration and check ( like we did during the loss of one FC)
Basically, one fan will still power the IMU, one water pump, one flash evaporator and one freon pump are working ( very important for cooling)
Every avionic fans work, except AV bay 2 ( either AC 1 or 2, not 3) We will monitor the temperature, near the limit during all the rest of the flight.


Image


Step 35 is for checking that cryo heaters are still working for H2 and O2 storage.
Step 39 for deactivation of the fes feedline heaters ( heaters that avoid the water line going from the cells to the fes to freeze) . Huge demand on power also, better to have it off.
Step 41 to 44: Other checks for APU Boiler heaters and recirculation pumps, already moved to off after APU shutdown earlier on.

And no bus tie for us as one of them had a short.


Page 10. Almost there. Quite long and exhaustive procedure. But the guys were three to do it .
We will switch our computer to entry software ( OPS 3)

That is the purpose of step 59 to 63
Every bus on GPC 1 for entry, and we can then type OPS 301 PRO on keyboard to transition towards OPS 3

Image


Finally step 65

Go to LOSS OF 2 FC LAUNCH DAY DEORBIT PREP
(ORBIT 2 OR ORBIT 3)


Quite frightening
Thats means we will have to deorbit as soon as possible, either to an Emergency Landing Site or normal one during second or third orbit.

While we are crossing the South Pole, MCC is checking the available landing sites, weather, deorbit opportunities , etc

Image




Alright, decision is taken.
It will be a deorbit in 30 mn for Diego Garcia, in Indian Ocean.
Cross range will be perfect, deorbit burn forecasted South of Mexico.

Image

Image



Flight dynamics ofccier and his team came up with a deorbit burn solution.
Peg 4 guidance, Almost no radial velocity, 3200 Nm for rentry range, and Time of Ignition at 01h35
Not too bad

More infos on Peg 4 here:

http://www.science-and-fiction.org/science/leo_05.html



Image






Time to tackle our last checklist, the LOSS OF 2 FC LAUNCH DAY DEORBIT PREP
That is part of a document called Contigency Deorbit, where an off nominal deorbit was needed due to severe failures after last opportunity for an Abort Once Around ( after OMS 2 basically)
It covers fuel cells, freon /water loops, cabin pressu .. failures




The menu

Initial Conditions are met.
We performed the loss of 2 FC check.
And we didn't started the Post Insertion activities ( Payload bay door opening mainly)

Image





The checklist is long, shared between Commander, Pilot and one Mission Specialist. It is supposed to be entered 1h30 before TIG. We have 30 mn, challenge accepted.
It is very similar to what we have done until now.
Reassigned some systems for entry, shed other ones. 3 screens usage , minimum lighting, ...
To sum up, a thin layer between the vital equipment needed for entry and the max load one FC could handled ( a kind of Apollo 13 scenario electrical wise)

I am not going to share that check as it is not publicly available ( L2 membership on nasaspaceflight.com forum), but make a sum up of it.
It is following a nominal deorbit preparation check with some power differences.


First, we activate the boiler heaters and controller for APU start up later on

Image




Mission Specialist instals all the checklists and cue cards for deorbit burn and entry,something close to that:

Image

He will also check that all checklists related to electrical problems have been well done and will eventually connect some lost AC buses ( AC 1 or 2) to the remaining good one ( AC3) with some AC power transfer cable ( helped by the pilot)
Space Mac Gyver
A lot of CB reactivation ( Mainly for RCS valves and drivers, flight control systems during entry, accelerometers and rate gyros assembly that allow a far better refresh rate than IMU for dynamic phases like entry, etc)




Then action are very similar to a normal entry preparation.
Landing site preparation on Spec 50 horizontal display.

Burn Datas entered into our only GPC left

Image


Purge of the fuel cell to have optimal power with no contaminants in it for entry.


That take us normally 45 mn before the burn ( 15 mn for us), and there we go through the entry check
Similar to that tutorial:

http://wiki.flightgear.org/Flying_the_Shuttle_-_Deorbit_Burn_and_Final_Entry_Preparation_Advanced



Main difference is the reactivation of GPC 4, to have two redundant GPC for entry ( instead of 4 normally)
GPC 1 and 4 for Guidance, Control and Navigation
GPC 5 for Back up flight system

Image


After reconfig.
2 strings on GPC 1, 2 on GPC 4 and 3 running GPC

Image




Time to Deorbit.
01h35mn into the mission

Image


Monitoring is really tough on 3 screens

Image



For that task, Mission Specialist seat is really the best, like an instructor in a simulator

Image



Good burn, a steep entry angle ( low perigee and short entry interface)
Peacefully derivating towards its Entry Interface in 20 mn

Image



Last look from Commander to the Aurora Borealis


Image


Apu put back online and full hydraulic pressure powers.
Just two are available for Entry, as they have a dual redundancy Main bus wise.

Image



Entry Interface as forecasted by the LEO tool

Image




Burning Phoenix out there, overhead Saudi Arabia

Image




We bit low on path due to the steeper than usual Rentry angle, but we will catch up

Image




Out of the danger zone, no more plasma

Image




Diego Garcia insight, finally

Image




Long final, good state vector thanks to GPS update ( no MLS there)

Image





Final Flare and max concentration

Image



Made it, not too bad!

Image









What a journey, awesome adventure, really.
I spent a lot of time reading through real life procedures to see how such failures would have been handled , that is fascinating to see the number of things they thought about.
Dozen if not hundred of pages for just a specific scenario.
Crazy STS program :)

And that is truly amazing to be able to repoduce it with a nice degree of fidelity in the sim.
Of course , there are some discrepancies with real stuff, as it is so complex in real for system intrication and handling in such a severe case.

Next journey, one with absolutely no failure in it :)
Last edited by GinGin on Wed Feb 05, 2020 8:57 am, edited 1 time in total.
GinGin
 
Posts: 1580
Joined: Wed Jul 05, 2017 11:41 am
Location: Paris
Callsign: Gingin

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby eatdirt » Tue Feb 04, 2020 9:39 pm

Superb story Gingin!

How did you generate the failures, some specific mission file settings?
eatdirt
 
Posts: 1012
Joined: Wed Aug 15, 2018 3:06 pm

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby wlbragg » Tue Feb 04, 2020 9:56 pm

Your imagery is stunning and really adds to the story. Thank you for all the mission stories you do, they're great reference material as well as entertaining.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7586
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby GinGin » Wed Feb 05, 2020 8:51 am

Thanks guys, always a pleasure to share when enjoyment was there.


How did you generate the failures, some specific mission file settings?


Yes, here:

Code: Select all
<failures>
  <section-defined type="bool">true</section-defined>

  <mode n="0">
  <node  type="string">/fdm/jsbsim/systems/failures/fc1-coolant-pump-condition</node>
  <occurs-met-s  type="double">2700.0</occurs-met-s>
  <probability  type="double">1.0</probability>
  <value  type="double">0.0</value>
  </mode>

  <mode n="1">
  <node  type="string">/fdm/jsbsim/systems/failures/electrical/main-B-condition</node>
  <occurs-met-s  type="double">1200.0</occurs-met-s>
  <probability  type="double">1.0</probability>
  <value  type="double">0.0</value>
  </mode>
</failures>


Pretty powerfull to pinpoint a specific failure / scenario.
GinGin
 
Posts: 1580
Joined: Wed Jul 05, 2017 11:41 am
Location: Paris
Callsign: Gingin

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby Gomendio » Mon Feb 10, 2020 6:33 pm

As always GinGin a very nice effort and quite a gem in terms of training value.

To some extent, I can say that you are a quite tight bunch of people that come to terms with the sim in all its capability with such an ease and grace that,
specifically for me right now is unachievable. Your blank screens reminds me of the typical Ac/Dc TR IDG failure scenario found in my A320 type rating sim Check.

I'am way behind for certain. The complexity of failures and how you dwell with LEO targeting to provide a flexible de-orbit solutions after just a couple revolutions on orbit is putting me off from the loop.

A lot of stuff and real knowledge lies behind. Trying currently to perform a consistent QRH training and failing in gathering all the technical stuff accordingly. The matted vehicle is a very serious piece of machinery I can tell.

Struggling but persisting in my effort.

My hat off gingin
Gomendio
 
Posts: 29
Joined: Fri Oct 25, 2019 11:03 am

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby GinGin » Tue Feb 11, 2020 6:08 pm

@Gomendio: Thanks for the return, glad it helped.

Your blank screens reminds me of the typical Ac/Dc TR IDG failure scenario found in my A320 type rating sim Check


Indeed :)
I had always been curious on how those failures would have been handled up in the space


The complexity of failures and how you dwell with LEO targeting to provide a flexible de-orbit solutions after just a couple revolutions on orbit is putting me off from the loop.
A lot of stuff and real knowledge lies behind


That is "just" a lot of small bricks assembled together after a lot of readings.
LEO tool is really powerful and handy ( in every space sim with true physics behind)
Step by step, and it is nice when everything is mixed together in a complete mission :)


Trying currently to perform a consistent QRH training and failing in gathering all the technical stuff accordingly


Nice to read that, hope you have a lot of fun also.
Like in your job, Fcom 2 (SCOM there) knowledge always help a lot for QRH troubleshooting
Don't hesitate if you have questions
GinGin
 
Posts: 1580
Joined: Wed Jul 05, 2017 11:41 am
Location: Paris
Callsign: Gingin

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby wlbragg » Fri Feb 02, 2024 5:34 am

What a trip, GinGin. After skimming through this a second time just now, I noticed something I might consider attempting.

He will also check that all checklists related to electrical problems have been well done and will eventually connect some lost AC buses ( AC 1 or 2) to the remaining good one ( AC3) with some AC power transfer cable ( helped by the pilot)
Space Mac Gyver


it was possible to put cable into the cockpit to tie AC lost bus to a good AC bus, quite crazy maintenance action


If you think it useful and can draw up some schematics of what exactly this looks like, including the storage location of such, I would love the opportunity to see if I can implement the graphical portion of something like this.

I don't think I ever posted or said anything about this, but I actually started modeling the lower deck. Nothing to really show so far, but it's started.
Kansas and Ohio/Midwest scenery development.
KEQA, 3AU, KRCP Airport Layout
Intel i7/GeForce RTX 2070/Max-Q
User avatar
wlbragg
 
Posts: 7586
Joined: Sun Aug 26, 2012 12:31 am
Location: Kansas (Tornado Alley), USA
Callsign: WC2020
Version: next
OS: Win10/Linux/RTX 2070

Re: Space Shuttle : Massive Failure/Emergency Deorbit report

Postby GinGin » Fri Feb 02, 2024 5:15 pm

@wlbragg: I forgot about this flight.
Deep in the systems for those malfunctions. I remember it was the deepest failure I simulated.
Close to real.
Nice idea for the schematics. I can work on basic one for main systems like electrical one.
Very nice for the lower deck.
I am ironing some small things, and working on some deeper ECAL / Bermuda about logic.
GinGin
 
Posts: 1580
Joined: Wed Jul 05, 2017 11:41 am
Location: Paris
Callsign: Gingin


Return to Spaceflight

Who is online

Users browsing this forum: No registered users and 2 guests