Britain's 24-hour culture: With the 'leisured society' a distant dream we're working longer and less regular hours than ever

Unless a lot changes in the next 13 years, a lecture John Maynard Keynes delivered in 1928 will sadly go down as one of the more learned pipe dreams in history. At a school in Hampshire, the big-thinking British economist told some kids that, 100 years later, we would all be enjoying lives of such abundance that we would work 15 hours a week at most.

"For the first time since his creation man will be faced with his real, his permanent problem – how to use his freedom from pressing economic cares, how to occupy the leisure, which science and compound interest will have won," he wrote in a 1931 essay based on the talk, called Economic Possibilities for Our Grandchildren. Find it online if you like feeling depressed.

Keynes did not say which three hours of each day we would be toiling, on average, but it seems likely that his glorious vision still involved working nine to five, or at least a bit of it. Yet today we are not only still working all the hours God sends, but more ungodly hours, too: Keynes' grandchildren are working nights.

A report published last month by the Trades Union Congress (TUC) revealed that the number of night workers had risen by 7 per cent, or 200,000 people, between 2007 and 2014. More than three million of us now regularly work nights. They are also growing as a proportion of the working population, from 11.7 per cent to 12.3 per cent. As debates continue to rage about an out-of-hours NHS and, in London, the prospect of the night Tube, what happened to Keynes' utopia? And what does it mean for us, individually and as an increasingly 24-hour society?

Night shifters used to be mainly men in factories, but now the ageing population, as well as the rise of the technologies and systems that Keynes hoped might liberate us, mean that the people who keep the wheels of modern life turning are an increasingly varied bunch. Transport networks and the communications industries, call centres, care homes and security services now account for the bulk of the night shift.

"You've probably noticed that one of the groups least likely to work at night is managers," says Paul Sellers, a policy adviser at the TUC who helped produce the report. Less than 10 per cent of managers, directors or other senior officials report working nights, while at the other end of the scale, more than one in five of those working in care, leisure and the service industries, and people still in manufacturing, regularly work into the small hours.

Sellers, who has dimming memories of packing biscuits by night as a younger man, says that, while radical shifts in consumption and lifestyles mean we now expect to be transported home, buy a pint of milk, get the news or see a doctor at any hour, we should be "careful how we progress, because if there are no rules, you can be damned sure that some employers will act badly, and that the wider side-effects for society won't necessarily be positive".

Dave Caesar is at the forefront of this growing tension. As a student and junior doctor in Edinburgh in the late 1990s, he had some tough nights. "We would regularly work 70- to 80-hour weeks and I think my record was 84 hours," he says.

Some weeks, when Caesar would stay on site while he was on call, he could spend 125 hours on hospital premises. "There were a few incidents in those days of people crashing cars and making bad clinical decisions," he recalls. "I remember at the end of a night shift my notes had these long vertical lines on them where I'd fallen asleep mid-sentence."

"The expectation that society has of their service industries – wanting to call our banks 24 hours a day – is the same in healthcare," Caesar explains. "We're also more anxious about our health, and still try to fix diseases at the end point rather than looking for solutions before they arise."

Add an ageing, growing population and the demands on hospitals have never been greater. Yet shorter shifts require more staff, which is expensive. Doctors and nurses also have less time on the job to get good at what they do.

Moreover, people working in hospitals, like everyone else, have changing expectations of modern life. "We have a workforce that expects a better work-life balance, like their peers," Caesar says. "We have a society that expects more and better healthcare, all the time, and in the middle we have the law on what we're allowed to work. Somehow we have to combine all that in a tightening fiscal environment to create rotas that are sustainable and attract people to work, while also getting the right expertise in the right place. It's a really difficult conundrum."

It is the conundrum at the heart of 24-hour Britain. Just as pressure is piling on the NHS to increase staffing overnight and at weekends (the Government wants to ditch the 48-hour maximum in the NHS, among other things), a similar dispute is raging under the streets of London. Last week the RMT union called off strikes planned for next week in a dispute over proposed weekend night services on London Underground.

Rachel Jones (not her real name) already knows what the night Tube will bring. Her partner already works nights, often for a week at a time, as a station supervisor, clocking on well before the last train and off after the first train the next morning. "I'm effectively nine to five, and quite often he will be getting in at the time I get up, and we'll cross for maybe half an hour over my breakfast – his dinner –and I won't see him again until the following morning," says Jones, who is expecting the couple's first baby. "He's normally sitting there, knackered, desperate to go to bed."

The couple have to plan their lives weeks ahead in the hope of being able to attend weddings or grab the odd dinner out. "But my main problem with the night shift isn't so much the impact on our relationship as worries about his health," she adds.

Employers are also sometimes reluctant to communicate what protection there is, Sellers says. In many industries where health and safety is less of an issue, employees can choose or agree to opt out of the 48-hour maximum – but not on night shifts. "But we know it happens," he adds. Night shifters are also entitled to regular health checks, but not all of them know it. As demand for nocturnal labour grows, abuse of the system can be extreme. Earlier this year, a disability charity in Scotland was criticised for paying some carers at its homes £2.50 an hour for "sleepover shifts". (They would only be paid above the minimum wage if they were called into action.)

Night shifts can also offer a bad deal to society. Productivity is lower than in the day, studies show. Workers sleep less overall, which has knock-on effects.

A study published last month by researchers at University California, San Diego, calculated that just one more hour of sleep a week increases wages long term, through increased productivity, by up to 5 per cent. "And this could easily aggregate up to the entire economy level," says Andre Spicer, professor of organisational behaviour at Cass Business School in London.

Keynes's prediction seems fanciful now but economic theory backs it up. Data published in 2013 by the OECD, a group of rich countries including Britain, revealed a clear inverse relationship between hours worked and productivity per hour.

Chances are, the time you put in after what should be the end of the day (or night) is wasted. But try telling that to your boss when you wave goodbye after a six-hour shift. Economic facts have no place in our job culture. In 2013, the American anthropologist, activist and author David Graeber explored the way that we had replaced armies of domestic servants and other old-school roles, not with liberating technologies but what he called "bullshit jobs" invented "just for the sake of keeping us all working... And here, precisely, lies the mystery," he added. "According to economic theory, at least, the last thing a profit-seeking firm is going to do is shell out money to workers they don't really need to employ. Still, somehow, it happens." If you're not, say, a nurse or a bin collector, ask yourself this: if your job vanished tomorrow, would Britain suffer? Would anyone notice?

Now Hotchkiss regularly works from 5pm into the early hours. "It can be really hard work but when I wake up in the morning I feel like my time is my own again," he says. "I'll go to the park and read, or meet people for lunch. The freedom it has given me was unexpected, and I'm happier."

Hotchkiss is single, but there can be benefits to shift work for families. Luke Howard has worked nights editing this newspaper's website for 12 years, starting when it barely existed. Now it's huge and Howard's 5.30pm to 1.30am (think more 5-2am) shifts are manic. "It can be hard to find time to eat," he says. Howard is 38 and shares a desk with his partner, who is a writer by day. "Back in my 20s it was wonderful to have lie-ins every morning but we've got a three-year-old now, which can make things hard, but it also means I can spend all day with my son." Even so, he often wonders if at some point, on balance, it might be time to seize the day again.

Up in Edinburgh, Caesar has three young children and still regularly works night shifts, as part of a more manageable rota. "I'll come home and hopefully get three or four hours of sleep before they're jumping on my head," he says.

When you feel like you might be expecting too much from 24-hour society, think of him, or Jones and her partner, passing like Tube trains in the night over their morning tea, or the guy putting these words on a screen. In the meantime, there's still time for Keynes' 100-year dream to come true. Bring on 2028.µ