TO ME THE GREAT thing about being 40 is that I don't have to be around young people any more." Nick Cave lights a cigarette and gives a rueful smile. "I don't mean that in a snobbish or superior way. But I've always felt kind of old; a little misplaced as a young person." Once regarded as a hell-raiser and enfant terrible, Cave is able to find some consolation in encroaching middle age. Freed of his predilection for addictive substances, he no longer labors under the thankless label of "the rock star most likely to die next". His records sell respectably to a loyal audience; his concert performances are nowadays at seated venues, relieving him of the obligation he felt in earlier days to wade into his audience, fists and boots flying.

Singer, songwriter, novelist, sometime playwright and actor, Cave enjoys a singular and peculiar status. As the singer with the Birthday Party, the group that took him from Australia to Europe in the early '80s, he was regarded as an unruly, insurrectionary force, a personification of the self-destructive impulse to which rock music so often gives licence.

As leader of the Bad Seeds, formed from the remains of the Birthday Party with a few additional members, Cave has established himself as one of the most literary songwriters in rock, producing two of the finest - and most radically contrasting - rock albums of recent times: Murder Ballads, a lubricious collection of tales of mayhem and darkness; and last year's brilliant collection of melancholic love songs, The Boatman's Call.

After his years as a cult figure, these records have elevated Cave precariously close to popular acclaim - not a position that seems to bring him much pleasure. When, in 1996, he was nominated for an MTV Award for Best Male Artist, the nomination was withdrawn at his own request. "My relationship with my muse is a delicate one at the best of times," he explained in a letter to the organisers, "and I feel it's my duty to protect her from influences that may offend her fragile nature. My muse is not a horse and I am in no horserace and, indeed, if she was, still I would not harness her to this tumbrel - this bloody cart of severed heads and glittering prizes. My muse may spook! May bolt! May abandon me completely!"

The preciousness this suggests is misleading. The expression on Cave's face as he runs his big-knuckled fingers through his brilliantined hair is not so much one of overweening self-regard as anguished embarrassment. "I never had any interest in music as a career, and I still stand back aghast at successful things that happen; I still wonder, how the f... did I manage to do that? Because I still feel very much an impostor in the music world."

The striking-looking Cave folds himself into a sofa like an anglepoise lamp, the gaunt features, waxen complexion and shock of black hair lending him the appearance of some bird of prey.

He was born and grew up in Warracknabeal, north-west of Melbourne. His mother was a librarian, his father an English and mathematics teacher who instilled in his son an early awareness of the transforming and talismanic power of literature. "From the age of 12 or 13, he would read passages to me from books that he thought were good - Dostoevsky, Nabokov," Cave says. "But it was less a matter of what I learned from him than seeing what literature did for him. When he talked about literature, I saw him become alive."

Cave's father was killed in a car accident in 1978, when Cave was 19. "After he died, I very much shut off and ceased to have any real feelings towards him until I had a son myself. Through having a son myself, it clicked that perhaps my father wasn't functioning well." By his own admission, Cave was a problematic, attention-seeking child.

"I always actively attempted to separate myself from what was generally the mainstream of thought." His school life followed the typical course of the wilful, artistically inclined outsider - absent from sports and consigned to special classes in the art block - and rock music offered the usual escape route, a place, as Cave puts it, where "I could make a lot of noise and had a licence to behave as abominably as I wished".

At art school - which he left after failing his first-year exams - he formed a group, the Boys Next Door, which eventually became the Birthday Party. The role of singer fell to Cave "because no one else wanted the job, and I couldn't play anything".

Taking their cue from the burgeoning punk rock scene in Britain and America, the Birthday Party appeared to find onstage fisticuffs, skirmishes with the law, and drug and alcohol abuse more or less mandatory. But the calculated outrage of the group's performance and Cave's sustained experiment in self-dereliction were always less interesting than his literacy and obsessions as a writer - a fact that came increasingly to the fore when, in 1984, he formed a new group, the Bad Seeds, as a vehicle for his songs and based himself in Europe.

Like Dylan's and Leonard Cohen's, Cave's work has pivoted on the fine line between sacred and profane. Like Cohen, he is one of the few songwriters whose work has merited publication in hard cover - two volumes of King Ink cover his lyrics, prose and plays from 1980 to today - or to have extended his range into fiction. His first novel, And The Ass Saw The Angel, was published in 1990.

"Nick's writing can be broken into two parts," says Mick Harvey, founding member of the Birthday Party and the Bad Seeds, and Cave's friend and musical collaborator for the past 26 years. "In the '80s you could say he was obsessed with the Old Testament. In the '90s, he's obsessed with the New Testament. There's the old fire and brimstone Nick, and the newly humanitarian Nick."

Cave would not demur. The shadow of a vengeful, powerful and capricious God, more inclined to punishment than mercy, hangs heavily over his earlier work. "In those days, it felt like the appropriate text for the way I approached life," he says, "which was fuelled by hatred and rage against things. But I don't feel that any more." The transition can be traced to 1988, when, having spent much of the '80s living in Berlin in increasingly chaotic circumstances, Cave first underwent treatment for heroin addiction. "It started to get in the way of my work," he says simply, "and that became intolerable."

In 1989, at a Bad Seeds concert in Brazil, he met and fell in love with Viviane Carneiro, a fashion stylist. Their son Luke was born in 1991. The couple has since split up, but the two remain good friends, living close to each other and sharing the responsibility of bringing up Luke.

Cave lives alone above a pizza restaurant in west London, in a flat furnished with a piano and bookshelves lined with volumes on his two abiding passions - crime and theology. A settled life. "I've felt happier in the past four years than I've ever felt before," he says. "But, at times, I've also felt unhappier than I've ever felt before. I seem to have a much greater capacity to feel things generally. The circumstances of my life 10 or 15 years ago were horrendous in a lot of ways, but I can't remember actually feeling that unhappy."

Is that because he can't remember feeling anything? Cave shrugs. "It has a little bit to do with that, yeah. But it wasn't just about drug-taking; there was just an accelerated pace to my life where I didn't allow things to affect me. I didn't stand still long enough." On reflection, he says, he was "too busy to die", although he came close, overdosing on several occasions. "But I didn't have any real notion or understanding - and I certainly didn't have any fear - about that. And now, at 40, I do have a fairly clear idea that I'm going to die at some point; I wake up with an understanding of that, and that's something that affects the way you live, your relationship with people and with life. But it's not something I think young people should need to understand." He smiles and reaches for a cigarette. "It would just ruin a good time."

"I don't know if Nick is mellower or happier," says Mick Harvey. "I still don't get the feeling that he's comfortable with himself, but he never has been. Though I do think he's had some kind of revelation about human nature." The Boatman's Call was the most eloquent evidence of this, an account of two years in his life, embracing the disintegration of his relationship with Carneiro, and a consoling love affair with the singer P.J. Harvey and its subsequent termination. Few records have spoken with such stark and moving candor about love as a holy sacrament, agency of redemption and instrument of torture.

"I think I've realised the need for humanity; that we need to learn how to forgive each other for things. To reach a sense of compassion towards humanity is not an easy thing; but I feel it's something I need to aspire to. I have faith, and I have doubt. A lot of the time I feel the presence of God quite strongly; and a lot of the time I feel a great absence. But absence or presence, I still believe in him. I can't believe you just journey through life and then you die." He shifts uncomfortably in his seat and stares hard at the table. It is a source of some discomfort, he says, that nowadays he is asked to talk about God and morality in much the same way as he was once asked to talk about murder and drugs.

Cave is putting the finishing touches to a second novel, working on a film soundtrack and writing songs for a forthcoming album.

"Music is more important to me than it ever was. It frees me in a certain way. There's a lot of anguish in writing fiction and prose. But the really beautiful thing about music is the ease with which it presents itself. Very often, with songs, it feels like they've just found their way into existence. There's something very mysterious about that."

He falls silent, thinking on this. "The other nice thing about music is that you have music that's made by young people for young people, and music that's made for middle-aged people by middle-aged people. I feel that I've very much ceased to make music for young people. It's music made by a 40-year-old man who's accumulated a certain amount of understanding in life."

He stares at the table and reaches for another cigarette. "I know that if I'd read this interview when I was 20 I'd have thought, what a complete asshole this guy is, but there you go ..."