Why do architects love the buildings we hate? That’s the question dons at The Queen’s College, Oxford, posed to architect Alan Berman when they asked his company, BGS, to investigate the controversial Florey Building in St Clements.

At the official opening in 1971, the Queen Mother was rumoured to have said it was the ugliest building she had ever seen. College bursar AA Williams described it as “a structure revolting and inhuman in its hideousness and defective in practically every aspect of its functioning”.

Just a year later, students were complaining that it leaked, was noisy, too hot in summer, too cold in winter, they couldn’t stand up straight in the showers, and there were no baths.

Last year, almost 40 years after it was built, it was listed grade II as being of special architectural importance, but Mr Berman says: “There are still people who dream of the wrecking ball.”

According to Mr Berman, when we look at this ‘flying saucer’ from the St Clements car park, we get the back view. Architect Jim Stirling had planned the building to be approached from the river, along a path from Magdalen Bridge which was never completed because one section is owned by a private landowner.

It is only from Angel & Greyhound Meadow, or from the cul-de-sac footpath — which, according to Mr Berman, is now used by drug addicts — that we can see Stirling’s vision.

But Mr Berman fears this already inadequate setting may once again be compromised by plans for student housing blocks on part of the car park.

Lord Florey, the pioneer of penicillin after whom the building is named, died the week building began in 1968 and Mr Berman believes his death contributed to a breakdown of relationships with Stirling. Florey put up the money, and was almost the architect’s sole supporter in the college.

“This culminated in a legal battle, an intense dislike of the building throughout the college, a reluctance to spend anything but the minimum on maintenance, and decades later, to the possibility of demolition,” said Mr Berman.

He is delighted that the college now realises it has a gem of architectural history on its hands, and is keen to renovate the building, which had fallen into disrepair.

With the support of Queen’s, he has assembled a distinguished panel of architects to explain the work of its designer, James Stirling, which Mr Berman believes has been badly misunderstood by the general public.

“I’m totally in favour of experimental architecture,” he said. “It is not something we do at BGS, but someone has to do it. If no one experimented we would still be driving horses and carts.

“People shouldn’t knock it just because they don’t like it on aesthetic grounds, because there is much more to it in terms of the development of architecture and the development of the use of new materials.”

Mr Berman wrote to 32 well-known architects, asking them what Stirling meant to them, and 24 of the replies are reproduced in a book called Jim Stirling and the Red Trilogy: Three Radical Buildings.

The first building, Leicester University’s engineering department, has always been more accepted than Cambridge’s history faculty and the Florey Building, which was Stirling’s third major commission.

Lord Rogers, responsible for the Millennium Dome, says in his contribution to the book: “In Oxford and Cambridge, the majority of people would probably have wanted something like copies of Florentine classical architecture, or a sort of vernacular.”

Another famous architect, Norman Foster, Baron Foster of Thames Bank, who designed the Gherkin in the City of London, says: “At Oxford, Jim established a very secluded, tranquil riverside world for the privileged few lucky enough to be housed there. Others, of course, would not share this view.”

Baron Foster accuses the dons at Queen’s, “horrified by the nakedness of architectural abstraction in the raw”, of “planting and prettifying” the Florey Building.

Mr Berman says: “It was a crucial step in architectural history, and that’s why it’s so influential worldwide. Richard Rogers says he couldn’t have built the Pompidou Centre in Paris had Stirling not showed him the path.”

Mr Berman believes many of the building’s problems stem from the fact that it was a prototype, and he compares it to another 1960s design classic, the Mini.

“It was designed in the same year as the Mini, which was equally innovative. The first car they made leaked and was a complete disaster. The problem with buildings is you don’t prototype them. Someone has to live in the experimental buildings. But we can now repair them with technology that wasn’t available in Stirling’s time. The Florey Building was ahead of its time.”

Delving through The Queen’s College archives, which were made available to BGS for a technical survey, Mr Berman uncovered clashes among the governing body that caused contradictory instructions to be given to Stirling.

Lord Florey, a scientist, responded to Stirling’s plan for double glazing, for example, by saying that it would cost more than the cost of the electricity for decades to come.

“Old-style buildings are not insulated and now we are insulating them. We need to do the same with 20th-century buildings. My thesis is people find them so strange visually that the problems, which I don’t deny, were used as a stick to beat the architect.”

The backlash meant that Stirling never worked again in the UK, and he is still more revered abroad — particularly in Germany, where he built an art gallery in Stuttgart, and in the US, where he built at Harvard University.

Mr Berman has recently returned from an exhibition of Stirling’s work at Yale which will move to Canada next year, and then to Britain.

He said: “The technology is now available to bring the Florey Building’s environmental performance much closer to contemporary standards without damaging the building’s architectural quality. When this is done, Stirling’s design will be better able to exercise its magic and Lord Florey’s ambition will have finally been fully realised.”

The Stirling exhibition will be at Tate Britain next April.

Meanwhile, plans for the St Clements car park site are still causing controversy.