Edward Kennedy: Liberal American senator who never escaped his brothers' shadows or the whiff of scandal

Edward Moore Kennedy was the youngest of the four sons of the ferociously ambitious Joseph Patrick Kennedy, bootlegger, ambassador to the Court of St James and the founding father of a dynasty whose luck was not the equal of their fortune.

After the deaths of his three older brothers he tried to pick up the family standard, but in spite of genuine political gifts and a spark of idealism that was somehow never quite extinguished, either by his family's arrogance or by a streak of coarse self-indulgence in his character, he was, by his own and his family's elevated standards, a failure. He served with distinction in the United States senate for almost half a century, longer than all but two senators in history. But he never achieved the goal his father set for one son after another, to be elected president of the United States.

His oldest brother, Joe, was shot down on a dangerous mission from East Anglia over Germany in 1944. His second brother, Jack, did become President, but was assassinated less than three years after his inauguration; and his third brother, Robert, was murdered in a Los Angeles hotel as he was leaving a rally to celebrate his victory in the California primary election, a victory that might well have made him president, too.

Ted, as he was known (it was only his close family and his political enemies who called him by the childhood nickname "Teddy"), also came close to winning the presidency. He would probably have been the Democratic candidate for president in 1972 had it not been for the worst of the several scandals that punctuated his life and clouded his reputation, the mysterious accident at Chappaquiddick in 1970, which Kennedy survived, but in which a young woman was drowned. In 1980, he did run in the primaries – and strongly – against Jimmy Carter, the incumbent president, only to lose the Democratic nomination after a bruising campaign in which the Chappaquiddick episode surfaced repeatedly.

In 1962, when he was still strictly speaking under the minimum age of 30 laid down by the Constitution, Kennedy was elected to the Senate for the family's home state of Massachusetts in a bitter fight with Edward McCormack, an experienced scion of a rival Boston Irish political clan scion which gave the United States a Speaker of the House of Representatives in the 1970s. In spite of all the scandals and his inability to fulfil his brilliant promise in presidential politics, Senator Kennedy was effective and respected in the Senate. His positions were unabashedly liberal, especially on health care, more in keeping with his brother Bobby than the more cautious, centrist track pursued by Jack.

Though he never succeeded in rising to the top of the Senate hierarchy as majority leader (he was defeated by Robert Byrd of West Virginia in 1971), he was for many years chairman of the Senate's labour and public welfare committee, which had jurisdiction over health policy. He used that position to good effect, just as he fought in later years for education reform. Kennedy was also one of the earliest, most consistent, and most outspoken Democratic critics of President George W. Bush's Iraq war.

Throughout the ups and downs, a constant of Kennedy's political career was his resilience. He might have been overweight, with a red face that betrayed his drinking. But to his great credit he never changed his political philosophy, even when the traditional liberal welfare policies to which he was committed were going out of fashion in the 1980s and 1990s. And in his political fiefdom of Massachusetts he was immovable – never more so than when he beat off a dangerous challenge from Mitt Romney, son of the Republican presidential contender George Romney, for his Senate seat in 1994.

Kennedy was born in 1932 and christened Edward Moore Kennedy, after Eddie Moore, one of his father's "rough and tumble but incredibly efficient" Boston Irish hangers-on. Friends and biographers have speculated that it must have been hard to grow up as the baby of the nine Kennedy children. Certainly he was pampered as the heir to great wealth. His father's assets were often estimated at about $400m, and Kennedy himself was given a trust fund of $10m which grew to at least twice that sum as a result of the careful management of the Park Agency, run by his brother-in-law Stephen Smith.

As the youngest male child of an intensely religious mother who was frequently abandoned for long periods by her husband, himself a notorious womaniser, young Teddy was in effect brought up by four women, his mother and his three surviving older sisters. (His eldest sister Katherine, Marchioness of Hartington, was killed in an air crash and another sister, Rosemary, was mentally deficient and confined to private nursing homes after a lobotomy insisted on by Ted's father). Kennedy himself said that, "It was like having a whole army of mothers round me", and his mother confirmed in her memoirs that "he was my baby and I tried to keep him my baby".

Although father and mother were scarcely on speaking terms after Joe Kennedy's affair with Gloria Swanson in the 1930s, the father, too, undoubtedly encouraged his youngest son to feel that life would be made easier for him. In 1962, when Ted ran for the Senate, his elder brothers expressed some doubt about whether he was ready for such high office. "Now it's Ted's turn," said Joe, "and whatever he wants, I'm going to see he gets it."

Whatever the reason, a whiff of scandal hung around Ted from a fairly early age. There was always a streak of dishonesty, and a dangerously quick fuse that could trigger off sudden, ugly violence. There was lifelong recklessness in the pursuit of whim or pleasure. And there was a certain arrogance, born of the certainty that he was not as other men, and that what Ted wants, Ted gets. These deep flaws of character were not the whole story of a complex character. But they were there, and he paid for them.

Six foot tall and weighing 15 stone, he was a useful football player and he hoped to make the university team at Harvard. But Harvard, anxious to show that it was not as other colleges that recruited giant dunces on football scholarships, insisted its players pass various academic requirements, including a foreign language. Aware that he would fail his Spanish exam, Ted Kennedy paid a friend to take it for him. He was found out, and expelled. The first result was that he had to do military service, in which he did not shine: he failed to rise above private in two years. Harvard readmitted him, but again he did something stupid. Playing rugby against a New York team, Kennedy three times lost his temper and got into fist-fights and was sent off the field.

Next he tried to get into the Harvard law school and failed, so he went instead to the law school at the University of Virginia, then a relaxed place for equipping southern gentlemen to practice law in what were unreconstructed, not to say somnolent, agrarian societies. Kennedy duly met the law school's modest standards of the time, after a career of riotous living and four arrests for reckless driving.

I first met him in Palm Beach shortly after he had announced for his brother's old seat in the Senate representing Massachusetts, a seat which had been "kept warm" since Jack went to the White House by one of his old Harvard room-mates. He was newly married, and when the Kennedy yacht, the Honey Fitz, stopped at the dock to pick them up, he and his young wife Joan, he in blue seersucker, she in pink, were both shining with good looks, good health and good fortune.

Yet there was a darker side to this Massachusetts Irish Greek god. One summer when Ted was on a cruise from the family compound at Hyannis on Cape Cod, up to Maine, he was rowing ashore for supplies. A yachtsman shouted some taunt. Ted stormed on board the yacht, threw the yachtsman overboard, and when his friends came on deck, he threw them into the sea too, not knowing or caring whether they could swim. In 1964, campaigning for his second term in the Senate, he was badly injured in a light plane crash, in part because he was the only passenger on board who had not fastened his seat-belt. His spine was broken in six places, he broke two ribs and he spent six months in hospital.

Ted took the murder of his two brothers very personally. For one thing, he was very close to them both, though in slightly different ways. Jack was so much older that he was almost a father figure, and one who supplied some of the warmth that was not forthcoming from his real father. The relationship with Bob was more complex: close partnership inextricably mixed with intense competitiveness.

Later, the youngest Kennedy wondered whether he, too, whom the gods had blessed, might be destined to die young. Once, on a trip to Alaska, he had too much to drink and told his companions, "They're going to shoot my ass off the way they shot Bobby's." Grief and fear fuelled a growing drinking habit, and that in turn reinforced his established traits of recklessness and erratic behaviour. The syndrome came together in his destructive behaviour on the night of 18-19 July 1969, after he had competed in a sailing regatta on Martha's Vineyard, an event he had taken part in since he was a child. That evening he went to a party at an isolated holiday cottage on Chappaquiddick island, which is linked to the island of Martha's Vineyard only by a ferry.

Afterwards, attempts were made to portray his attendance as an act of noblesse oblige, as if the great senator condescended to reward junior helpers by visiting their party. Anyone with the slightest acquaintance with the mores of the time, the place and the Kennedy sub-culture, would dismiss that out of hand. Six attractive, young unmarried women had been assembled and the senator and three favoured male guests, including a cousin, were there with partying on their minds. There was a certain amount of drinking before Kennedy left the party with a young woman called Mary Jo Kopechne, who had worked on his brother's campaign the previous year.

All that is certain is that the next morning Miss Kopechne was found in the back of the senator's car, drowned and without her underwear. The car had failed to take a right-angle bend on the road from the cottage to the ferry to Edgartown. Senator Kennedy's story was that he had come off the road, failed to save Mary Jo, then swam the narrow channel and checked into a motel. Other explanations, more or less plausible, have been put forward. Whatever precisely happened that night, a number of facts, collectively deeply discreditable to Kennedy, are undisputed.

A United States senator, married, attended a party with a number of young women in an isolated cottage. He left the party, having drunk heavily, with one of the young women. She was drowned or perhaps asphyxiated in an air bubble in his car. He then failed to report what had happened for many hours. Only when the car and the body had been discovered did he report the accident. Thereupon the dean of the Yale Law School and other high-powered lawyers were summoned to help the senator extricate himself from his legal difficulties, and Kennedy aides spirited away the witnesses who had been at the party.

Kennedy did briefly appear in court to plead guilty to a misdemeanour charge of leaving the scene of an accident. For this offence he was duly sentenced, as his lawyers had arranged with the prosecution, to two months, suspended, in jail, and to the loss of his driving licence for one year. He then retreated to the family compound at Hyannisport, where he read out on television a 17-minute apologia. It was a rambling, disingenuous and tasteless performance, in which, for example, he referred to the "Kennedy curse", though it occurred to most of viewers that he, not some non-existent curse, was to blame for the difficulty he found himself in.

There was perhaps more than a touch of hypocrisy in the national outburst of moral indignation to which Kennedy was subjected: stones were certainly cast by many who were not without sin. Yet the Chappaquiddick episode was disgraceful enough. A detail, the senator's inability to remember Mary Jo Kopechne's name, lingered in many minds. His cool appearance in freshly pressed yachting gear on the Edgartown dock before he knew that the accident was known stuck in other craws. The efficient way in which the well-oiled Kennedy machine went to work, too, was indecorous.

Then there was the senator's self-pity. "He did the worst thing he could have done," commented the Kennedy retainer Richard Goodwin. "He Nixonised the situation."

The punishment, if not inflicted according to due process of law by the Commonwealth of Massachusetts, was nevertheless a life sentence. However hard he worked, however loyally he sought to defend his brother's liberal legacy in Congress, Chappaquiddick would never be entirely forgotten or forgiven. In a sense, it was the end of the Kennedy legend of invincibility and of style. Ted Kennedy had behaved in about as unstylish a way as could be imagined. And now he was to become the thing his family disdained above all things: a loser.

In 1972, there was a brief flicker of the presidential flame. He had ruled himself out for the nomination, and the campaign in his absence turned into a deeply divisive contest between Senator George McGovern of South Dakota, representing the anti-Vietnam war insurgency, and senator Hubert Humphrey of Minnesota, champion of traditional Cold War liberalism. McGovern won. But at the convention itself Kennedy had his moment of glory – a speech that, whether or not he had much to do with the writing of it, was a magnificent call to arms for the Democratic party.

As the 1980 election approached, with Jimmy Carter mired in failure and unpopularity, Kennedy once again pulled on his campaign armour. But at the very first hurdle he stumbled irretrievably. In 1979 he agreed to be interviewed by the CBS correspondent, Roger Mudd. But when Mudd, as he could hardly have avoided doing, asked him about Chappaquiddick, Kennedy stuttered incoherently.

"I find," he mumbled, "as I have stated, that... I've found that the conduct that... evening, in in in in the... as a result of the impact, of the accident, and the... and the the sense of loss, the sense of hope, and the, and the sense of tragedy and the whole set of circumstances, that... the the behaviour was inexplicable." The dots represent, not excisions from a coherent text, but pauses in a tortured utterance that was painfully close to breakdown.

Essentially his bid for the nomination ended at that moment. Kennedy was well beaten by Carter in New Hampshire, which ought to have been his own New England territory, partly because two national magazines ran investigative features about Chappaquiddick during the campaign. He recovered doggedly, pressing his glamorous family into service and winning the New York primary. But at the convention, the last nail was driven into his defeat when he failed to change the rule that forbade delegates elected with a commitment to Carter to change sides.

In 1982 his wife, Joan, who had herself become an alcoholic, divorced him. Kennedy drank heavily and pursued young women with an apparent disregard for either his reputation or theirs: it was not unusual for a big black limousine whose licence plate proclaimed that it belonged to the senior senator from Massachusetts to be parked outside a young woman's house all night. In 1991 his reputation took a further knock when his nephew, William Kennedy Smith, was prosecuted for a rape in Palm Beach, Florida. Although Smith was acquitted, it suggested that Kennedy, at 59, was accompanying relatives half his age on expeditions to pick up young women in nightclubs.

Yet if these accounts of his private life suggest he was a lost soul, that would give a very misleading impression. He remained an assiduous and a surprisingly popular legislator. Some called him "the best politician in the family" – meaning that he was far better attuned than either his coolly ambitious brother Jack or his idealistic, aggressive brother Bob to the hail-fellow-well-met style of Capitol Hill, and to the hard-ball game of legislative negotiation and horse-trading.

He maintained an exceptionally able staff, and staunchly defended liberal causes, from health care and education to immigration. In 1987 he led the successful liberal attack on the Supreme Court nomination of the conservative Robert Bork. Under Bill Clinton he was a tireless facilitator in the search for peace in Northern Ireland, and early in George W. Bush's administration worked closely with a Republican President on education reform. But bipartisanship ended with the Iraq invasion, which Kennedy termed, "a fraud cooked up in Texas. He called the war "George Bush's Vietnam."

In 2008 he endorsed Senator Barack Obama at a critical point in the campaign for the Democratic presidential nomination, explicitly likening Obama to President Kennedy. Despite having been diagnosed with a brain tumour in May, he returned to the Capitol last summer to cast the decisive vote for the Democrats on Medicare, and was there again to see Obama sworn in as president, but suffered a seizure at a luncheon afterwards.

His private life had setted down. In July 1992 Kennedy married a Washington lawyer, Victoria Reggie, a divorced mother of two. It was by all accounts a very happy union. Much had been taken, but there was much to abide. Kennedy left behind three children by his first marriage with Joan, including Patrick, a congressman for Rhode Island, and Edward Jr, who has been rumoured to eye a Congressional seat from Connecticut. But in politics as in the abiding public legend of the family, it was their father – tarnished, flawed yet always somehow larger than life – who was the last of the Kennedy titans.

Godfrey Hodgson and Rupert Cornwell

Edward Moore Kennedy, politician: born Boston, Massachusetts 22 February 1932; US Senator (Democrat) for Massachusetts, from 1962; married 1958 Joan Bennett (divorced 1982, two sons, one daughter), 1992 Victoria Reggie; died Cape Cod, Massachusetts 25 August 2009.