His father, Elbridge Colby, was a professor of English, author, and Army officer who served in Army and in university positions in Tientsin, China; Georgia; Vermont; and Washington, D.C. His grandfather, Charles Colby, had been a professor of chemistry at Columbia University but had died prematurely.