David Lammy: Mothers need the support of a loving partner. Just ask mine

Yesterday, women all over the country received cards, gifts and thanks from grateful children. But what many mothers really want cannot be given by their sons and daughters: more engaged fathers. In Britain, too many women find themselves sacrificing their careers to do the "double shift" of working and parenting, while men's lifestyles carry on unchanged. And too many mothers are watching their children suffer from absent or uninvolved fathers.

The issue of fatherhood has been on my mind for a long time – since about 1984, in fact. Aged 11, I watched my father walk out on my mother and away from his responsibilities. He had always been someone I looked up to when he was around; he became unimaginably important the day he left.

My mother was left with the enormous, lonely, frightening responsibility of looking after four children on her own. She had to carry the financial burden for the whole family, doing two, sometimes three, jobs at a time. At the end of the working day, this tireless woman came home to her children to do the job of two parents – reading to us, listening to us, helping with homework and getting ready to do it all again the next day.

While she toiled, I struggled to cope with what felt like a personal betrayal. Like many others in my situation I had to find other sources of advice about how to negotiate life as a teenage boy, from learning how to shave to how best to deal with peer pressure at school.

Today I see children doing the same. For nine months I have been video documenting discussions with black fathers supported by the Runnymede Trust. Their view is clear :too many young men drift into gang culture, mimicking the behaviour of celebrities and footballers rather than learning from the men in their own lives. And too many girls grow up without ever receiving affirmation from a man without the implication of sex.

This conversation cannot drift into the familiar barracking of single mothers. Instead, the rise of "baby fathers" – parenthood through casual sex – must be constantly challenged among both young men and women. Children from fatherless homes are disproportionately likely to grow up in poverty. Single parenthood halves the earning power in the family and doubles the caring burden, with obvious effects. In particular, I worry at figures showing 59 per cent of Black Caribbean and 44 per cent of Black African children grow up in single parent households.

However, I learnt the hard way that this is not just a question of family structure. The issue is the quality of the relationships that fathers have with their children, not just whether parents are sleeping under the same roof. My father didn't just leave the family home, he left the country. Whether or not relationships between adults last the distance, relationships between fathers and their children need to be strong. The "broken Britain" debate engages with this only on a shallow level with its narrow focus on facile marriage incentives.

The evidence shows children without strong relationships with their fathers suffer from lower self-esteem and diminished life chances. Yet surveys show between a quarter and a third of children with separated parents have little or no contact with their fathers. Where parents are together, a culture of long working hours can do damage: more than four in 10 men say they don't spend enough time with their children. This applies as much to the stressed executive as the overworked plumber.

And where fathers are present they are not always engaged. Only one in 10 children would go to their father first if they had a problem; more than three quarters would go to their mother.

The charity Relate argues couples need more help when children are born and better support when relationships break down to sustain civility between parents. This, they believe, is vital to keeping children in touch with their fathers. The Equality Commission has called for paid paternity to match maternity leave, to help fathers build close relationships earlier on in their children's lives. Trade unions point out that Britain has the longest working hours in Europe – and those who work the longest are fathers aged 30 and upwards. Others think that services like Surestart need to be more father-friendly.

This cannot all be down to government. Fatherhood needs updating from the traditional model. The "provider-protector" model is largely in our comfort zone, not least in the black community where there is a strong tradition of the disciplinarian father. When images of men in popular culture so often involve hubris and violence, we must confound those ideas by building loving, lasting relationships with our children. Providing structure and discipline matters as much as ever, but what children also need is an emotional bond with their father. That, in the end, must be up to all of us.

David Lammy is Minister for Higher Education and Intellectual Property. This is an edited version of a speech to the Runnymede Trust today