His family moved to Hampton Virginia and he attended Hampton High school, where he was the class president and member of the 1937 state champion varsity football team. General Olds graduated in 1939 and turned down scholarships to both the Virginia Military Institute and Dartmouth University to attend West Point's Millard Prep School.
He was accepted into the West Point military academy in July 1940 and played for the West Point football team.
After the attack on Pearl Harbor on December 7, 1941, his academy education was interrupted so he could attend the Spartan School of Aeronautics. His entire class graduated a year early, in June of 1943, because of WWII. General Olds would receive his pilot wings and head to Williamsfield Arizona to complete his pilot training.
General Olds was assigned to the 434th Squadron and deployed to Europe where he would dogfight and provide air support for the Invasion of Normandy, participate in the air battles over Germany and Luxembourg.
After the war, he briefly served as the assistant football coach at West Point, took 2nd place in the Thompson Trophy, and performed transcontinental "dawn-to-dusk" flight from March Field in California to Washington D.C. and back again, before he returned to an air unit and commanded a unit with the Royal Air Force under the Royal Air Force Exchange Program.