On the way from the airport, the Fijian taxi driver took a long look at Jan Morris before demanding: “Are you a man or a woman?” With her usual eye for detail, this most well-travelled of writers vividly recalls what happened next. “I told him, ‘I am a respectable, rich, middle-aged English widow’.

“‘Good’, he said, ‘just what I want’, and put his hand upon my knee.”

The incident with the cab driver is one of those fleeting encounters that has stayed with Morris, during her wanderings around the world, as both a woman and a man. Little could the taxi driver have imagined that the woman he was lusting after had, until the early 1970s, been James Morris, who during the Second World War served with the 9th Queen’s Royal Lancers, and had fathered five children.

It is equally unlikely that the lascivious cab driver knew he was seeking to take liberties with someone named in The Times as the 15th greatest British writer since 1945.

We can be sure that Morris would not have told him. For the very mention of her high-profile sex change is pretty well guaranteed to end any conversation with this remarkable woman of words.

“I am sick to death of it. It was a bore right from the beginning,” she told me, towards the end of what had until that moment been a friendly interview, marking what was to be her first appearance, at the age of 84, at the Oxford Literary Festival.

My earlier reference to her autobiography Conundrum, which offered a grippingly honest account of her ten-year transition from man to woman, had almost led her to call off the interview before it had started.

Perhaps I should have recalled that many years ago television’s greatest inquisitor Sir Robin Day had been met with anger when he touched on this subject.

So, we settled on talking about other people — lots of them from all across the globe: some incredibly famous, others homeless; the primitive and exquisitely civilised; the grand and the humble.

For they all feature in her book, Contact! Brief Encounters in a Lifetime of Travel (Faber and Faber), which was to be the subject of her festival talk.

After more than half a century of criss-crossing the world, producing 41 books in the process, Morris came to the conclusion that over a lifetime of travel and literature, she had written relatively little about the people she has come across.

Conjuring up ambience, mood and a sense of place in purple prose is what has delighted generations of readers, leading Alistair Cooke to call her the “Flaubert of the jet age”. Portraits of cities such as Venice, New York, Sydney and, most famously of all, Oxford, are what come to mind when we think of Jan Morris.

But she has finally decided to give us a glimpse of some of the characters she encountered along the way, with a series of vignettes, recording fleeting encounters with people who proved to be “sparks of her work”.

She explains: “Often I have given them only a few lines, or a paragraph; occasionally the people have known me as James rather than Jan, because, until 1972, I wrote in the persona of James Morris; but my fleeting contacts with them have fuelled my travels down the years, generated my motors, excited my laughter and summoned my sympathies.”

The first encounter in the book offers something of the flavour, recording what happened in Central Park, when Morris chances upon a young black man asleep on a bench. As the sleeper opens his eyes, Morris decides to break into Shakespeare. “Be not afeared the isle is full of noises,” she tells him. “Yeah,” the man replies, scratching mightily, “bugs too.”

What the New Yorker made of this strange English woman, is left to our imaginations.

But Contact! also offers glimpses into Morris’s own colourful past. She recalls the Christmas parties arranged by the canons of Christ Church when she was a schoolboy at Oxford’s Christ Church Cathedral School.

Later we are reminded that, as a journalist, it was Morris who broke the story of the British expedition that conquered Everest in 1953.

Two days after Edmund Hillary and Tenzing Norkay became the first climbers to reach the summit of Everest, Morris recalls being at the foot of the mountain and seeing a solitary figure moving gracefully down the valley with a smile that illuminated the glacier. It was Tenzing and they breakfasted together. “The last I saw of him there, he had stripped his lean, lithe body to the waist and was soaping himself with water from a tin basin. It looked a chilly operation.

“By the very next day he would be one of the most famous men on earth.”

And Morris would be one of the planet’s most celebrated journalists.

“I had thought of having no named people in the book at all. Most are anonymous,” she told me.

But the half a dozen or so who do get a name check are certainly worth waiting for. They include the likes of Yves Saint Laurent, King Hussein, President Truman and Peter O’Toole. “I wanted variety. But what I was most concerned about was covering the whole world.”

Home to Morris is in “the top left hand corner” of Wales, where she lives with partner Elizabeth, the woman, who he married in 1949 and is the mother of his children.

“Addicted to travelling? Yes, I’m afraid I am,” she said.

Well into her eighties, it turns out she is still embarking on long journeys, consciously expecting every one to be enjoyable.

“The only part I do not enjoy is the beginning of a journey. I live in the bowels of Wales. So just to get anywhere — a train from Bangor, a flight from Manchester — involves a couple of hours’ driving.”

The pleasures of wandering is among the great joys of life, she still insists, and the best way to do it is alone.

“If you are working, you can’t be worrying about somebody else. You have to be entirely egocentric — and I am certainly an egocentric writer.”

One English city continues to be a regular destination for Morris, who is a familiar guest at the Old Bank Hotel in High Street.

“Oxford made me,” maintains Morris, who become a member of the university in 1936 at the age of nine through being a pupil at the choir school of Christ Church.

He (as she then was) returned to Oxford after serving as an intelligence officer in Palestine and Italy during the war, to read English and edit the student paper Cherwell.

Then, in 1965, Morris’s classic account of the city — said “to bring the very stones of Oxford to life” — was published.

It turns out that she still loves the place.

“I just love wandering around. It feels very much the same. Many of the buildings are much better after restoration. What I really like is the feeling of vigour.”

The writer is full of praise for the Ashmolean, following the old museum’s £61m redevelopment.

“It is just wonderful. The architect Rick Mather has given it a stunning makeover and all its familiar wonders are now displayed with virtuoso taste and showmanship in gloriously lucent settings.”

Her fans hopefully made the most of her Oxford visit, for there are unlikely to be many more books. She had anticipated that her book on Trieste, which she first saw as a member of the 9th Lancers, would be her last book, given that she considers it her best.

Morris still regards her Pax Britannica, the trilogy about the decline of the British Empire, as her most important work.

For years, she had thought about writing a book about the First Afghan War, which might have offered some valuable insights to the British Government.

One of the most memorable exchanges in Contact! sees Morris meeting an old man in Kabul in the 1960s. “What would happen if another enemy attacked this capital as the British had, catastrophically, in 1845? Would they be exterminated too?” Morris asked him.

And the response?

“He gave an angry tug of his beard and threw me a look of piercing and bloodshot intensity. ‘The same’ he hissed through the last of his teeth. ‘The same again’.”

The story of the remarkable journey that saw her cross the strangest river that perhaps any man can cross is told in a book that she now views as “a period piece”.

In it she reveals how at the age of three or four, she realised that she “had been born in the wrong body”. The sex-change process, she notes, has become almost commonplace.

But it certainly was not when he booked himself a return ticket to Morocco in July 1972.

“Everybody in my predicament knew of Dr B,” she wrote, a man who “had rescued hundreds, perhaps thousands of transsexuals from their wandering fate”.

“He did not bother himself much with diagnosis or pre-treatment and expected handsome payment in advance; but his surgery was excellent, he asked no questions, and he imposed no conditions, legal or moralistic,” wrote Morris.

In Conundrum she addresses the connection between her wanderings and sexuality. “I spent half my life travelling in foreign places. I did it because I liked it, and to earn a living, and I have only lately recognised that incessant wandering as an outer expression of my inner journey. I have never doubted though, that much of the emotional force that men spend in sex, I sublimated in travel.”

But her loves have stayed the same; books, family and Wales, where she has proudly followed the literary achievements of her children. Her son, Twm Morys, poet, singer and broadcaster, is the Welsh Children’s Poet Laureate.

There is also to be one last Morris book, which alas will not be supported by any festival appearances. “It will only be published after I’ve kicked the bucket,” she chuckled. “It is called Allergorising. The theme is nothing in this life has only one meaning.”