Portrait: Pride and prejudice

It was my partner, Jamie, who taught me to be more confident about myself. Sadly, he died, but in the short time we were together, he was a massive influence on me. We'd go into Diesel and he'd say, "He wants something young and fashionable, but it has to be classic, and it mustn't stand out." It came naturally to him, and I was so impressed. I had no idea you could go into shops and explain your limitations and concerns. He also made me try on clothes, which was something I was always too embarrassed to do. I wouldn't lift a pullover for fear of it unfolding and making work for someone. He gave me a completely new take on communicating, and your rights, and your abilities. The very fact that I can sit here, now, having any kind of conversation about clothes, is a revelation.

Some children grow up knowing how their lives will pan out, having ideas, liking pop music ... But I had a somewhat mixed, difficult background, and never even knew I had the right to make a statement through my clothes. Fashion was for other people. As a confused 19-year-old looking for some form of identity, I went through a phase of aspiring to the Sloane Ranger look - the Barbour, the Renault 5, and living in the countryside - although I could never afford it. I started work as a filing clerk, and tried to look more like a businessman. I'd buy Church's shoes, and save up for suits.

At 20, I started my financial advice business, with a very serious message about prejudice, and, to try to get this across, I smartened up. But as Ivan Massow Associates has become more successful, I've become scruffier. Even so, there are some meetings where you know you have to look as they expect you to. If you're going to do a pitch to a board of directors, and you've got 15 people sitting around a long table on the 24th floor wanting you to look smart, I can pull that off.

There are a couple of tailors who are very kind to me, who have coaxed me out of my shell. I arrive very nervous, on my moped, and they look after me, and bring out samples, and say, "This will look good."

It takes self-confidence not to dress up, and every so often you realise you haven't achieved that level of self-confidence. When I first had to go to see the people at the ICA, I spent ages wondering what they wanted a chairman to look like. I decided, "I am a businessman," so I just wore a business suit without a tie. That was as close as I could get to arty.

In the countryside, I have two pairs of old army surplus trousers, which I wear to garden, or to ride, or to go to dinner, and most of the time no one looks. In London, friends will take me to dinner and try to get me to wear a tie, but I really resent dress codes in restaurants. If anyone dared to look me up and down, I simply wouldn't go in, it would so annoy me.

I am not attracted to people who spend a lot of time pursuing personal fitness or thinking about clothes. I'm not saying I don't like a good body, but I'd prefer at least to pretend that it came through some kind of energy for life. Every three or four days I think, "Will people be too offended if I don't shave again this morning?" And when I get my haircut, the only instruction is, "Make it look scruffy."

By not conforming, you cut out dead wood. At a party, there will be people who never speak to you because of the way you present yourself, and, actually, I'm delighted not to speak to them. I'd much rather be the scruffy one, shifting around on the margin.