Duffner became Holy Cross' head football coach prior to the 1986 season, taking over a team in a state of emotional despair and quickly transforming it into the nation's most successful Division I-AA program. He guided the Crusaders to a 60-5-1 record in six seasons, including perfect 11-0 marks in both 1987 and 1991. Holy Cross won five Patriot League championships, four Lambert Cups and four ECAC Team of the Year awards during Duffner's tenure, and also earned the No. 1 ranking in the final 1987 NCAA Division I-AA poll. He was named the Patriot League Coach of the Year four times, and also took home National Coach of the Year honors in 1987 and 1991. Duffner ranked third all-time at Holy Cross with his 60 career victories, while his .917 winning percentage was the best in school and Patriot League history when he left the program.