Secretary of Defense Leon Panetta  (right), with Allen (left) and Petraeus (center) in 2011, has indicated he will not stay for Obama’s second term.