Should doctors and patients learn to plan humane, happier endings rather than trying to prolong life?

Catherine is 72 and she has lung cancer. The cancer is advanced, despite recent chemotherapy. She has fallen overnight, and her husband tells me that she seems more unwell and asks me to visit. She has some pains in her bones, and she is breathless just walking a few paces. She has lost weight, looks frailer, and is clearly frightened. Getting out of the house is becoming more difficult, and more exhausting.

Catherine has been offered further palliative chemotherapy, which her family – especially her adored son – are desperate for her to keep going with. Her son Gary tells me, in a low voice, privately, at the front door: "we're not ready to let go of her yet." Her husband Tom asks me to try to persuade her into getting the treatment. He has told me, "I'm worried that she's losing the fight." Catherine has grazed her knee and her elbow, and she also has a chest infection. I think she is going to struggle being at home, and so we admit her to hospital. I find out a couple of weeks later that she had a further fall, breaking her hip, and died in hospital. When I speak to Tom, he tells me how upset he is: she had wanted to die at home, and he let her down. In truth: as her GP, I did.

Catherine and Tom are not real patients. They are fictional, but reflect the types of dilemmas and problems I come across frequently as a GP, and which patients, families and healthcare professionals face at the end of life. There is no doubt that standard treatment for cancers, such as chemo and radiotherapy, can effectively cure many cancers – for example, early-stage bowel cancer, or testicular cancer. But in other situations where cancer has spread, the benefits of medicine are less certain. A study published in The New England Journal of Medicine in 2012 found that 69 per cent of patients with lung cancer and 81 per cent of people with bowel cancer did not understand that the palliative chemotherapy they were having was not going to be able to cure their cancer.

Palliative chemotherapy can indeed extend lifespans. However, a systematic review of the evidence for chemotherapy in one type of lung cancer – non-small cell lung cancer – published in 2008 found that it increased the survival rate from 20 per cent at one year to 29 per cent. Clearly, that's an improvement – but an improvement for only 9 per cent of the patients taking the chemotherapy. When we consider what's called the "treatment burden" – not just the visits to the hospital, but the side effects of fatigue, nausea, hair loss or nerve pain – many people will weigh up the chance of success, and, entirely reasonably, opt not to have it. It can be entirely rational to prioritise quality of life over quantity.

Yet the way we talk about cancer and illness can make people think that they have to have treatment – no matter the cost. Take a recent campaign in Glasgow run by the local Beatson cancer clinic. The massive billboard posters proclaimed: "Be full of hope. Be courageous. Be ready to fight." Similarly, Cancer Research UK want to raise money to "beat cancer sooner" and wants to "inspire others to join the fight". In 1977, Susan Sontag wrote, in Illness as Metaphor: "Illness is interpreted as, basically, a psychological event, and people are encouraged to believe that they get sick because they (unconsciously) want to, and that they can cure themselves by the mobilisation of will; that they can choose not to die of the disease."

We see this all the time: obituaries that say that someone "lost his fight" against cancer. But there is a backlash. Recently, in the London Review of Books, Jenni Diski, who is being treated for cancer, wrote: "Under no circumstances is anyone to say that I lost a battle with cancer. Or that I bore it bravely. I am not fighting, losing, winning or bearing." The language used around cancer can make it seem as though only "fighters" deserve to "win", placing a moral judgement on people's choices about what treatments to have. This only makes it harder for people to stop or decline therapies – even when they are unlikely to help, generating only side effects that make quality of life poorer.

In his recent book, Being Mortal, the US surgeon Atul Gawande writes movingly about what he identifies as the conflict between medical priorities of safety and survival and individuals' priorities for simple well-being. In truth, there is often an awkward collusion between patients, families and doctors. Medics can feel as though they have to keep offering treatments – even though they are unlikely to help.

A study published in 2002 found that patients felt doctors were pushing a "positive" or "fighting" attitude, with – incredibly – patients sometimes feeling that they had to protect their families and doctors by hiding their distress. Most tellingly, a systematic review, published in the BMJ in 2002, examined the impact of psychological coping styles on survival and cancer recurrence. It found "little scientific basis for the popular lay and clinical belief that psychological coping styles have an important influence on overall or event-free survival in patients with cancer". In other words, there is no need for patients to feel guilty about not "fighting" their cancer. It won't result in longer life – and it may stop people from having the kinds of conversations that they want to. US and European studies suggest that the vast majority of people want accurate information about what is likely to happen to them – even if their lifespan is limited – and not to have the wool pulled over their eyes.

So why do we find it so hard to talk about what works at the end of life? Talking about death and the ineffectiveness of treatments can expose medicine and make me, as a doctor, feel impotent. And time can be in short supply. Raising an emotional conversation can feel difficult to contain – as one consultant said to me: "The patient wants to talk about his tablets, and has an insurance form needing filled out, and I still have six patients to see in clinic, and a theatre list starting in an hour. I know I should. But I don't have time to start that conversation."

This kind of "prognostic paralysis" can mean that patients don't know what is happening to them. In turn, it means that they can't make good choices about what they want to happen – or where they want to be. We know that most people would prefer to die at home. An audit published by the Royal College of Physicians this year found that, 87 per cent of the time, doctors realised that patients in hospital were dying, but in only 46 per cent of cases, it was recorded as having been discussed with the patient. So how can we get to go home to die if we don't know that we are in our last days? We need to give doctors and nurses more space to ask patients whether they want to talk about the future – and we need more patients enabled to ask.

A systematic review published in the BMJ in 2003 found that doctors tend to consistently overestimate how long they expect patients who are terminally ill to live. To identify people who are at higher risk of dying, GPs are encouraged to ask themselves, "how surprised would you be if this patient died in the next year?" The bottom line is that we are quite accurate in identifying who will die – but we also think that many people will die who do not. Our population is living longer, with more diseases that act as risk factors – such as diabetes or dementia – that aren't as easy to give a prognosis to as cancer is. Ideally, given the difficulty of making accurate predictions, every older adult should consider what kinds of choices they'd prefer to make at the end of life, whether or not their doctor asks them.

To do that, we need accurate information about what works and what doesn't. The John Hopkins Precursor Study examined what choices American doctors made about death. While only about 20 per cent of the US population had made an advance directive, 64 per of doctors had. They were asked what kind of treatment they wanted if they had a serious brain injury. The vast majority wanted pain relief but not cardiopulmonary resuscitation, (CPR) dialysis, or intravenous fluids.

Far from wanting more aggressive medicine – they wanted less. It's tempting to conclude that the more we know about how ineffective many treatments are, the less we want them. Meanwhile, there have been concerns that the reason palliative care seems to prolong life in people with metastatic cancer is because patients get less aggressive medical care – which can lead to complications such as immune suppression and end up doing more harm than good. As for CPR, while medical television dramas frequently overestimate the likelihood of it working, the truth is that when CPR is used in frail people with metastatic cancer, it almost never works. Instead, it changes the nature of death. The peace of the deathbed is exchanged for the chest compressions, cracked ribs and electrical shocks of CPR.

Is this really the way we want to go? Instead of seeing death as a repeated medical failure, we have to use medicine to care, if not cure, and ensure that at least by starting difficult conversations, we have a better chance to die in the way we want.

Here is Catherine's story again, showing what a better death could be like:

Catherine is 72 and she has lung cancer. The cancer has advanced despite chemotherapy. She is frailer, has lost weight, and looks scared. I ask her why. "Am I dying? How long have I got? Do I have to go back to the hospital?" she says. I tell her that I am not sure, but I agree, I don't imagine that she has more than a month or two.

She asks if she thinks that in turning down more treatment, she is giving up. She, her son and husband Tom cry together, and tell her that they love her no matter what she chooses. She wants to stay at home, but is struggling to the bathroom and slipped this morning.

However, local social services are able to step in with a commode and hand rail, and a carer is able to spend enough time at home to help Catherine wash and dress. She is eating less, but friends and relatives bring soup and cake, and she takes a little. Over the days, it becomes clear that Catherine is in her last days.

She has visitors most of the time. When I make house calls, there is sadness, but the room is also full of stories, jokes, soap operas on the television, and sherry in the evenings. The nurses set up a syringe driver, so that she has constant levels of pain relief. The next day, she takes only sips of fluid, and the day after that, slips into unconsciousness. Overnight, she stops breathing, her husband sleeping by her side.

That's the kind of death I hope I have.

'Living with Dying: Finding Care and Compassion at the End of Life', by Margaret McCartney (Pinter & Martin, £11.99)