My design is pretty similar, but much simpler: not as many 7-segments, not as many buttons and less rotatory encoders. The switch panel will be a 5x5 matrix with its 25 diodes, and I'm not planning to use any 8-to-3 encoder for them.
I also believe serial USB communications to a "driver" in the PC will be simpler, at least during the first iteration. In fact, if we configure the panels as USB HID, I'm not sure if FlightGear could send data directly to the panels through the USB cable. I find easier coding a "proxy driver" in the PC that connects to FlightGear using the usual channels, telnet or IO protocols, and to our panels using the serial port.