With your background there are many possibilities actually.
Personally, I would suggest to stay within the confines of the base package, i.e. refrain from touching the C++ core source code, and instead concentrate on scripting.
FlightGear has a fairly well documented scripting language that can be used for doing many interesting things.
You mentioned physics and designing a new instrument, some time ago we talked about implementing a so called "FLARM" (wikipedia/google) in FlightGear using just Nasal scripting and some textures:
viewtopic.php?f=10&t=9547&p=96003#p96003A "FLARM" is basically a very simple TCAS-like device for glider pilots. It permanently recomputes 3D trajectories to provide situational awareness for pilots. This could be easily done for both, AI traffic as well as multiplayer traffic.
So such a project would comprise a fair amount of physics and maths, and it would be about creating a new instrument.
Still, it should be possible to implement this entirely within the base package, without touching any core C++ code.
Regarding the FMC idea, FlightGear has currently relatively insufficient support for modeling such modern FMC/FMS systems, these things are currently under active development.
You may want to check out the wiki or do a forum search.
Also, a forum or wiki search would also be useful for coming up with a list of useful software, such as for example Blender 3D (google) or GIMP for creating 3D models and textures.