Twenty years ago, actor Mark Harmon gave up his movie career for a more stable and local job with NCIS so he could be a more present husband and father, but the CBS drama didn’t exactly work out like a 9 to 5. 

The 72-year-old recently spoke to PEOPLE while promoting his upcoming book, Ghosts of Honolulu: A Japanese Spy, a Japanese American Spy Hunter, and the Untold Story of Pearl Harbor. During the discussion, he reflected on his longstanding and hugely popular role as Agent Gibbs. 

As Harmon shared, being in the entertainment industry is hard on families. Traveling for filming and events can keep people away for months at a time. In 2003, Harmon’s two sons, Sean (35) and Ty (31) were growing up at home with Harmon’s wife, Pam Dawber. Harmon was constantly on the road as a movie star, and he was missing the little things, like weekend breakfasts. So, he made a change. 

“It was important to be able to make pancakes on Saturday morning,” he told the publication. “And there were some times where that meant not sleeping. You just came home and took a shower and came back down. And yet I look back at it and I don’t miss the sleep.”

Mark Harmon Struggled Through Long Days and Sleepless Nights While Filming ‘NCIS’

Unfortunately, NCIS wasn’t a magic bullet, however. There were many days when the actor spent long hours on set. Mark Harmon shared that his first day of filming with NCIS lasted 22 straight hours. As he moved forward, those grueling shifts became common, and he “still missed things.”

The actor remained with the show for 19 years and earned two Primetime Emmy nominations and a legion of fans. He’s largely retired now, but he still serves as an NCIS executive producer. Meanwhile, his sons are enjoying film careers of their own. Sean is an actor, and Ty is a screenwriter. 

Looking back, Harmon knows he made the right choice moving from the big screen to the small screen. Because even though he still had to juggle the demands of a high-profile job with those of his family, he was a “part of their lives.”

“I was able to make breakfast on the weekend,” he remembered. “I’m not about to complain about any of it.”