Supreme Court justices have a job for life. But some left the court to make their lasting mark

In the summer of 1941, James F. Byrnes became a Supreme Court justice. Little over a year later, he had had enough and left the court to take a key role in planning the nation's wartime economy.

Americans have become accustomed in recent decades to justices who retire only after decades on the bench, or like Justices Ruth Bader Ginsburg and Antonin Scalia, who died while on the bench. Ginsburg and Scalia are among nearly half the 116 men and women who have been justices who died while still serving or within a few months of retirement.

Legislation pending in Congress would end life tenure on the Supreme Court, though there is little chance term limits will become law anytime soon even with the support of Vice President Kamala Harris, along with many others in her party and a majority of Americans. But those like Byrnes who did not regard the court as the pinnacle of their career and who left for other pursuits that would define their legacy suggest what justices might do if they couldn't spend the rest of their lives on the nation's highest court.

Byrnes, appointed to the court by President Franklin Roosevelt, later sat on a small secret committee of government officials that recommended the use of the atomic bomb against Japan and served as Secretary of State before ending his long public career as governor of South Carolina, where he emerged as a segregationist critic of the court’s Brown v. Board of Education decision that ended official segregation in public schools.

Like many justices until fairly recently, Byrnes came to the court from the political world. He had been a senator and member of Congress.

“It used to be that a lot of justices were former elected officials who kind of always had, you know, a kind of hankering to get back to it. And of course, that doesn’t exist anymore,” said Stuart Banner, a UCLA legal historian and law professor. His new book, “The Most Powerful Court in the World: A History of the Supreme Court of the United States,″ will be published in November.

Sandra Day O'Connor was the last justice who held elective office, as a state senator in Arizona. In today's court, every justice — other than Elena Kagan — has been a federal appellate judge. Kagan was the Obama administration's top Supreme Court lawyer before she took the bench.

A major reason for the recent preference for appellate judges is the heightened stakes that have attached to lifetime Supreme Court appointments. The paper trail of lower-court opinions is seen as a reliable indicator of how someone would vote on the Supreme Court.

But the line between politics and the judiciary was not always so distinct. Chief Justice Earl Warren had been California's governor before coming to Washington. The first chief justice, John Jay, resigned to become governor of New York.

Whether term limits would augur a return of justices with broader experience in public life is debatable.

But justices appointed for life may lose an “understanding of the broad public currents of the country,” said Cliff Sloan, who worked on Supreme Court nominations in the Clinton administration and helped vet potential nominees for President Barack Obama.

“Term limits would ensure a degree of measured turnover on the court that would be helpful in bringing different perspectives to the court,” said Sloan, who wrote about the court in his 2023 book, “The Court at War: FDR, His Justices, and the World They Made.”

Among the nine men who were justices when the U.S. entered World War II were former members of Congress, the onetime chairman of the Securities and Exchange Commission and a former attorney general.

After the Japanese attacked Pearl Harbor, Byrnes and other justices felt like they were sitting on the sidelines when they would rather be in the game.

Two days after Pearl Harbor, the court heard arguments in a case involving ships that had been built 25 years earlier, during World War I, Banner writes in his book. The case “appeared of little consequence in light of the news bulletins still pouring in from Pearl Harbor,” Banner quotes Byrnes as later recalling.

And Byrnes was not alone, Sloan said. At least five justices “made clear to FDR that they’d be delighted to leave the court and take on war-related responsibilities,” he said.

One of those justices was William O. Douglas, the former SEC chairman. Douglas would go on to serve longer than any other justice, more than 36 years. But he might have left the court after just five years.

In 1944, Douglas nearly became Roosevelt’s vice presidential running mate. Instead, the nod went to Harry Truman, who ascended to the presidency the following year when Roosevelt died.

Douglas stayed on the court even as the prospect of his candidacy was openly discussed. He only would have had to look back a couple of decades for guidance.

Charles Evans Hughes, a former New York governor, left the court in 1916 only after he had won the Republican presidential nomination.

On election night in November, he reportedly went to bed thinking he won the presidency. But he came up fewer than 4,000 votes short in California, and President Woodrow Wilson was reelected.

Hughes was far from done with public service. He eventually became Secretary of State and in 1930, returned to the Supreme Court as chief justice. He was the first chief justice to preside over the court when its new building opened in 1935.