Actually, if you understand rotation matrices, you'll understand quaternions. They are extremely closely related to the Axis-Angle representation (that is, where you specify a rotation by a unit vector axis and an angle to rotate about it). It's much easier to do a cockeyed rotation in that representation than in the more traditional spherical coordinates. Great circle paths are equivalent to rotations on the sphere.
Quaternions are in very common use for 3D gaming. I've borrowed a bunch of techniques from those folks.
If you want some light summer reading, here are a couple of not-very-expensive primers:
1. For map projections (great circles can be calculated easily in gnomonic projections; constant-heading paths work best in Mercator), Synder JP, Map Projections, A Working Manual, USGS Professional Paper 1395, 1987, http://pubs.er.usgs.gov/usgspubs/pp/pp1395 . This is somewhat dated, but is still very, very widely used as the standard reference for cartography, in several fields.
2. For quaternions, given a background in linear and vector algebra, Kuipers, JB, Quaternions and Rotation Sequences: A Primer with Application to Orbits, Aerospace, and Virtual Reality, Princeton Univ. Press, 1999. It's a bit wordy, but it's the clearest introduction to a somewhat murky topic I've found.
Kalman filters would seem to be a bit out of reach for even a very well educated high school student. They are based on integral transforms, several major steps past where you seem to be. An integral transform is kinda like a basis change like you saw in vector algebra; however, they live in infinite dimensions.