Five chiefs of staff explain why they didn’t want the job

(CNN) – Though they worked for presidents who would disagree on many issues, the five men who shared a stage Wednesday nightly mostly agreed on at least one thing: they didn't want their job.

Each of them had once served as White House chief of staff, but in many cases, only after initially balking at the job offer or outright turning it down.

“People came up to me and said, ‘Gosh, you've been serving for five years,’” said Josh Bolten, who had held a variety of positions including budget director in the administration of George W. Bush before ascending to chief of staff.   “You must be completely worn out.  How can you take the chief of staff job?

“And I'm thinking,” he continued, “thank God I don't have to be the budget director (again).”

Bolton and several of his counterparts from other presidencies gathered Wednesday at an event organized by The Aspen Institute and The National Archives and moderated by David Gergen, CNN’s senior political analyst and himself a veteran of many of the administrations represented at the event.

Those who resisted the appointment ultimately caved in, they suggested, because of a sense of duty.

“It's virtually impossible to be sitting with, in my case, a president-elect of the United States who is earnestly and sincerely asking you to help him and to serve the country and not to answer that call of duty so to speak, not to be dramatic about it,” said Mack McLarty, who served under President Bill Clinton.

He arrived at the White House after three terms of Republican presidencies.

“After 12 years of Republicans being in the White House, the first challenge you have is simply to get a cabinet and government in place,” he explained.   “It's no small feat.”

Donald Rumsfeld also spoke of the challenges he had on day one in the West Wing.  Before going on to serve as defense secretary under both presidents Gerald Ford and George W. Bush, he was Ford’s chief of staff.

It was chance, Rumsfeld said, that Ford became president when President Richard Nixon stepped down rather than Nixon’s vice president, Spiro Agnew.

“Had Agnew not resigned, we would have had a felon as president,” Rumsfeld said.

But the White House was still crawling, he said, with investigators who were looking into the Watergate scandal which ended Nixon’s presidency.

“He was torn between enormous pressure to provide continuity, to reassure the American people that the ship of state was stable – to reassure the world,” Rumsfeld said of his boss.  “The other side of that coin was when you’re presiding over a tarnished administration and the public considers that administration to have been subject to resigning for wrongdoing and you have a special prosecutor in the White House, running around interviewing people to see who knew what when, it was just a terribly difficult time. So there was pressure on the other side for change.”

Ken Duberstein had his own set of challenges when he stepped into the Oval Office with President Ronald Reagan.  It was the middle of Reagan’s second term and his approval rating was in the 30s.

“People referred to him not as a lame duck but as a dead duck and he had very little to do in those last two years,” Duberstein recalled.

But he set to work, because “Ronald Reagan understood that fundamentally the job of president was to build consensus in America and in order to win in Washington, you have to build a consensus in the country.”

He provided an inside window on Wednesday into the almost legendary bargaining between Reagan, a Republican, and House Speaker Tip O’Neill, a Democrat.

“It was hard to put the two of them in a room unless everything was prebaked because they both had strong philosophical beliefs,” Duberstein said.  “While they could tell good Irish jokes and stories after 6 o'clock at night, if you left them to argue a budget point or a tax point, you might as well not have been there because they both got red in the face.”

Still, he said, he saw O’Neill shaking Reagan’s hand after the president made the decision to take military action in Grenada.

“I will always remember Tip O’Neill sitting next to Ronald Reagan shaking his hand, patting him on the elbow and said, ‘God bless you Mr. President.  We're together and we are one as a country,’” he said.    “Can you imagine that happening today?”

For the most part, the former chiefs of staff stayed away from today’s politics, though when discussing the role of first ladies, McLarty noted – to laughs – that it was “no real surprise … Hillary Rodam Clinton was a major force in our administration.”

John Podesta, who would serve as Clinton’s fourth chief of staff, said he had the benefit of learning from McLarty and others.  One of the lessons, he explained, was what the role of the chief of staff really is.

“The most important word in the chief of staff title is ‘staff,’” he said.   “It was the president, really, that sort of guided what we were trying to get done, and it was his, I think, enthusiasm for the job that was really a touchstone for all of us, I think, who worked for him.”