'I have never had any opinion about politics at all." This statement, delivered in the well-modulated tones of a true aristocrat, will probably have surprised everyone who heard it on Tuesday afternoon in the Newman Rooms, off St Aldate's. The utterly apolitical woman gracing the second day of the Oxford Literary Festival was none other, you see, than Clarissa Eden, the niece of one British Prime Minister - perhaps our most famous one, Sir Winston Churchill - and the wife of another, Sir Anthony Eden.

Her duties as chatelaine of No 10, she explained, had nothing to do with public affairs. "My role was to make everything as easy and comfortable as possible for Anthony." Her implication appeared to be that others would have done well to follow her example. Even as one thought it, she continued: "Wives didn't muscle in on all these things as they do now."

But if the minutiae of politics passed her by, she brought a sharp eye to bear on some of its personalities. She is well-known for her 1956 lament: "In the past few weeks I have really felt as if the Suez Canal was flowing through my drawing room". Of the author of that crisis, Colonel Nasser, she did not think kindly. At diplomatic dinner in Cairo the previous year, he had been offended by the smart clothes worn by the British contingent. "Anthony spoke to him in Arabic. He thought that insulting too. Altogether chippy . . ." Chou En-Lai, by contrast, she thought "very sympathetic, very charming". He had brought a lot of Ming china with him to the Geneva Conference . . . "we thought it was very civilised".

Lady Avon, to give her the title she gained following her husband's retirement in 1957, was at the festival to promote her gloriously readable memoirs, Clarissa Eden: From Churchill to Eden (Weidenfeld & Nicolson, £20). The book has been compiled largely from her diaries, with cementing passages supplied by co-author Cate Haste (who joined her in the Literary Festival's question and answer session). The two met when Cate was working with Cherie Booth (Blair) - not, one suspects one of Clarissa's favourites - on The Goldfish Bowl, a book about Prime Ministers' spouses.

"If it hadn't been for Cate, I would not have been able to do the book," she told me in the festival Green Room at Christ Church before we set off for the Newman Rooms (at a cracking pace for an 87-year-old). I told her how much I had enjoyed the Oxford chapter of the book (she studied philosophy here in the period before, and into, the war), not least for her skill in capturing this intellectual golden age. Men such as Lord David Cecil, Maurice Bowra and Isaiah Berlin were her close friends. So, too, was the remarkable Lord Berners, who was then living in Oxford, away from his splendid home, Faringdon House, seeking relief from a breakdown. Clarissa writes: "He took digs in St Giles' where, of course, he immediately made his landlady the focus of funny stories."

Had there been time, I would have told Lady Avon that I once met that landlady, Miss Gertrude Alder. I interviewed her at a Summertown nursing home in 1981 on the occasion of her 100th birthday. She spoke fondly of 22 St Giles' in wartime "when I had half the celebrities in England staying there". Among the names in her visitors book were those of Ivor Novello, Margot Fonteyn and Robert Helpman. She presented a compelling picture of these stars clustered around Gerald Berners at his piano.

Lady Avon writes - and indeed spoke - of no one more affectionately than Sir Isaiah Berlin. The great philosopher figures prominently, too - and in the same pre-war period - in Justin Cartwright's paeon of praise to Oxford, This Secret Garden (Bloomsbury, £9.99), which also featured in the festival programme on Tuesday. This is because the South African-born writer (pictured left) was in Oxford and working on the book - in close contact with his old college, Trinity - at precisely the time he was researching a novel about Adam von Trott, one of the German conspirators hanged for the failed bomb attack on Adolf Hitler. Von Trott was a close friend of Berlin and, indeed, of Bowra and the historian A.L.Rowse, whose admiration for the strapping German aristocrat appears to have been rather more physical.

The powerful impact that Oxford had on von Trott, Cartwright told his festival audience, was shown in the fact that it was mentioned in his last letter to his mother, sent from his prison cell. The reason for his attraction to the city, the 'special' flavour of the place, will be known to many of my readers. If it isn't, you will find its essence distilled in The Secret Garden. Mr Cartwright's new novel, incidentally, is called The Song Before It Is Sung (Bloomsbury, £7.99).