Andy Goldsworthy, artist: 'I wanted to make art. I would've done that as a gardener or signing on'

Andy Goldsworthy lives in Penpont, a small village in the glorious rolling countryside 40 minutes from Dumfries in Scotland. He moved there in 1986 because it was cheap, living first in a small flat until he and his family – he has four children – moved into this house that was designed with much of his work literally embedded in the walls.

We meet in his office, adjacent to the house. "I do not have a studio; the outside is where I work." He does have a drawing studio nearby that has plans of potential projects, many of them abroad, pinned on the walls.

Sitting with Goldsworthy, I am aware of the surroundings, the light on a tree outside the window, the leaf works ranged around us. It is as if his observation has permeated my looking. After our conversation, I feel the need to look more carefully at my surroundings. "That is all there is," he says. "Seeing what is there."

Goldsworthy owns one field outside his home but makes his work in the adjoining fields. He started working as a farmer's apprentice in his teens. "Farming was as important, if not more important, than art school as a training ground," he says. "Learning how to work hard all day is a really important thing."

The co-opting of his neighbours' fields has proved challenging. He tells me a story about making a series of works with a tree on a steep, inaccessible bank over a period of eight years, only to arrive recently to find that half the tree had been removed for firewood. He still has the other half as it is on another farmer's land. "It hurts, but that is good for me."

Goldsworthy was born in Cheshire in 1956, obtaining his first degree in Preston. He received an OBE in 2000. His success surprises him: "I guess that I never expected it. I just wanted to make art. I would have done that as a gardener or signing on." He is best known for his environmental, site-specific sculptures, but he has always done performance works, often involving himself and his body.

His rain shadows – in which he lies on the ground during a storm, allowing a body-shaped silhouette to form – are an ongoing series. "I am as embarrassed at lying there as anyone else. Everything is saying to me, 'Do not lie down, Andy!' And the next thing I am on the ground. I will always do it, it has to be done." These episodes, like much performance art, are ephemeral, preserved only by photographs or videos taken by "whoever is around".

In the past, Goldsworthy has been attacked for being overly aesthetic. "But I have to work with flowers and leaves, because they are part of the land." When he pulls a beautiful leaf piece from a cupboard to show me, I am first assaulted by the smell – sweet chestnut – then captivated by the sheer virtuoso skill of making, his hand evident in the tight spiral, the entire object secured by thorns.

Beauty is an issue, he admits. "I can't just choose the rough bits, stone and wood, the manly bits. I would have a very reduced view of the land and I like the challenge of that."

Andy Goldsworthy is part of Lookout, at the Folkestone Triennial (folkestonetriennial.org.uk) to 2 November