Few public servants can match Robin Butler’s calibre as an observer of post-war British politics. Now ennobled as Lord Butler of Brockwell, he served as private secretary to prime ministers Heath and Wilson, ran Margaret Thatcher’s private office for three years, and worked as cabinet secretary during the premierships of Thatcher, Major and Blair. Ever true to Whitehall discretion, he has consistently declined to pen a memoir, but when we meet at his Westminster flat on a bright morning in May, the robust 71-year-old is happy to draw on his near half-century of experiences – and to apply their lessons to present-day government.

It is no surprise to hear him say that the organisation he entered in 1961 was “radically different” from today’s civil service – most obviously in its composition, but also in the educational background of its staff. “They took 121 fast-streamers that year, of which only half a dozen were women,” he remembers. “The vast majority were from Oxbridge, the vast majority were arts graduates. By the time I left, all those three [proportions] had changed enormously.” At the time the so-called ‘cult of the amateur’ meant that the British administrative class was dominated by people without specific professional training who moved from job to job, building ‘generalist’ skills.

At the time, of course, Britain was changing fast – and in Whitehall, people were starting to ask whether Oxbridge humanities degrees were the best training for those running a modern government and economy. Butler points to a book, Whitehall in Crisis, published by economist Thomas Balogh, which he says crystallised a sense that civil service generalists were being “brought into disrepute”. This was followed by the Fulton Report, which led to the creation of a dedicated civil service department and the Civil Service College (now the National School of Government), while fostering a new emphasis on personnel management. From these beginnings the modern service was born, with its focus on professionalisation and specialisation.

All these changes reflected a wider drive to transform the country into a dynamic, scientifically-advanced economy, encapsulated in Harold Wilson’s famous evocation of “the white heat of the technological revolution”. All exciting stuff, Butler says, but it was undercut by a pervasive sense of national decline. “It was a time when self-confidence in Britain was ebbing away; we weren’t doing well economically.”

Butler went on to serve as private secretary to both of the prime ministers of the era: Harold Wilson and the Conservative, Edward Heath. He admired both men, he says, but their styles could hardly have been more different. Wilson, though a former Oxford don, had a notorious instinct for populism; in his office, staffed by a loyal group of special advisers, things moved quickly and repartee bounced back and forth. “He was a master politician; very fast footwork, very deft politically.”

The diffident, reserved Heath, Butler says, was “much more straight up and down” – both politically and personally. “If you went up to his flat to put issues to him, there’d very often be loud music playing, you’d say something and there’d be a long silence before he delivered himself of what he wanted to say… You had to get used to that.”

Personality differences notwithstanding, both prime ministers struggled to haul Britain out of the mire of industrial strife and sluggish economic performance. Their success was decidedly mixed, and by the time Butler was back at the Treasury in 1976, the James Callaghan government was asking the International Monetary Fund for a large loan to prop up the public finances and the value of sterling. It was a crisis, Butler says – but it enabled the Treasury to “get a grip” on public expenditure. And there are clear parallels with today’s financial crisis, which has also seen the Treasury assert itself at the centre of events. “Whenever there is an economic crisis, a lot more hinges around the Treasury,” Butler says. 

The Thatcher years

He was still installed at the Treasury when Mrs Thatcher arrived in Downing Street in 1979, and in 1982 he joined her there to work as her principal private secretary. Butler himself is an amiable man and, by his own account, not ideological by nature; he expresses admiration for Edward Heath’s efforts to manage the country – industrial relations in particular – on a more “consensual” basis. Working for Thatcher, whom no-one could accuse of being a consensual politician, did his opinion change? “I don’t think my opinion changed. I think I had always preferred, and would always prefer, that people work in a consensual fashion. But [consensus-building] just hadn’t worked with Ted Heath.”

Thatcher’s style may have been “dominant”, her former cabinet secretary says, but she was also capable of listening to dissenting opinions. “She was somebody who could listen while she talked. Very often I heard her talk somebody down and afterwards say: ‘That person had a point, didn’t he?’” Moreover, though she could be contemptuous of what she saw as unwieldy bureaucracy, Butler says that once a civil servant had proved their willingness to help her, they became “one of her people”.

The 1980s seemed like an era of almost perpetual efficiency drives, beginning with a 1981 study by former Marks & Spencer boss Derek Rayner. By the end of the decade, the service had shrunk dramatically. Although Butler says that some of Thatcher’s early efficiency targets seemed arbitrary, he adds that periodic cost-cutting exercises – such as the current operational efficiency programme – are necessary. “It’s right that the civil service should be challenged to be more efficient; we don’t have the disciplines that the private sector has,” he says.

Butler is particularly proud of one efficiency change that he oversaw: the implementation of banker Sir Robin Ibbs’ review, which moved much of the responsibility for policy delivery away from Whitehall and towards new, so-called ‘next step’, executive agencies. This was certainly a landmark change, but Butler is keen to deny that it marked either a move towards privatisation or the shift of power to the much-maligned quasi-autonomous non-governmental organisations (quangos). “Agencies definitely aren’t that: they are directly responsible to ministers and they’ve got a job to do,” he says.

In fact, the former cabinet secretary is none too impressed with the growth of quangos in recent decades; their status, he admits, causes “a good deal of confusion”. “I do think it’s extraordinary how over quite a long period – including my time, and since – politicians have devolved responsibility to people we don’t elect and who aren’t answerable,” he says. “They’ve given a lot of power and accountability away to bodies over which the citizens have no influence.”

As cabinet secretary in November 1990, Butler had a key role in managing the handover of power to John Major following the coup that removed Thatcher from the leadership of both the Tory party and the country. “It was a very difficult time,” he says. Her dominance was such that her “traumatic” exit left rancour that festered throughout her successor’s premiership. Connected arguments over Thatcher’s demise and Europe seemed to doom Major’s cabinet – and the government in general – to a period of internal party conflict and policy drift. “There was a great deal of personal bitterness, and of course the administration was getting tired. That’s always likely to cause divisions and make people less self-disciplined.”

Now, of course, the Brown government appears to be plagued by similar problems. Is it difficult for civil servants to support a government of flagging or conflicting wills? “It can be,” he replies. “There’s nothing worse than a vacuum, and if the government isn’t doing very much then it’s rather debilitating for the civil service.” Looking back on the 1990s, he says that permanent secretaries had to put aside personal loyalties to their feuding ministers in an effort to keep the machinery of government running smoothly.

In the spirit of learning lessons from the past, what did the recession of the 1990s – which, like today’s, was preceded by a property market bubble – teach Butler about managing economic downturns? “The lesson is not to get into the situation in the first place,” he says, bluntly. “One way of avoiding it is to try and make sure that there’s not so much money around that [house] prices get inflated, and the other is to make sure that people don’t over-borrow.”

A new boss

In 1997, the apparent inevitability of a New Labour landslide after 18 years of Conservative rule presented its own challenges. “I remember having a meeting with the Cabinet Office in Central Hall, Westminster, a week before the election and saying to them: ‘The real challenge is not so much a change of government as the Conservatives being re-elected, because you’ve built yourselves up [for a Labour victory]. If [the Tories] come back you’ve got to serve them just as well.”

It’s an issue that’s suddenly very relevant again, with the opposition expected to reclaim power at the next election and some Conservatives of the opinion that Whitehall has been politicised under Labour. Butler is quick to dismiss the notion that incoming parties have any reason to be suspicious of a civil service fresh from long service for one party, but admits that some wariness is probably inevitable. “Politicians are by definition very ideological, and don’t find it easy to believe that others aren’t,” he smiles. “They suspect, having seen the civil service working for the other party, that they’ll have been brainwashed… but they haven’t.”

In 1997, the machinery of Whitehall had to adjust itself to New Labour. For Butler, this meant adapting to the informal style of Tony Blair, and for government press offices it meant turning themselves into the 24-hour media operations demanded by Blair’s infamous director of communications, Alastair Campbell. Lord Butler says Campbell’s criticisms of government communications were largely well-founded, but many have said that allowing Campbell and other special advisers to give orders to civil servants was a mistake. Butler insists that this was his decision, not Blair’s or Campbell’s, and was an attempt to be “tidy-minded”, since political appointees in governments of both parties had already led teams of civil servants. “What we did was make an exception to the statutory instrument – but only to reflect what had happened under previous administrations.”

Nonetheless, Butler says the experience convinced him of the need for a civil service bill – the first since the Northcote-Trevelyan report of 1854 recommended the setting-up of a permanent, impartial bureaucracy. Over the years his view has changed, he says, and he now agrees with Commons public administration committee chair Tony Wright that dedicated legislation is the only way to preserve the sanctity of the impartial service in an age of special advisers. “One could see in 1997 how easily [impartiality could be lost]. For example, it was not only heads of press offices but some permanent secretaries that were being changed [by Labour]. That was a rubicon which, once crossed, could never be gone back over,” he says. “If we want a permanent civil service, we need to enshrine it in law.”

On a personal level, Butler says that Blair was pleasant and approachable, but he admits to believing that government works better when the formality of Cabinet decision-making is respected. The phrase ‘sofa government’ has been commonly attributed to Butler as a description of Blair’s style and, though he says the term actually came from a journalist, it does capture his feelings. “Very often there weren’t agendas, there weren’t proper records, it wasn’t clear what people had to do, and I felt that this was a disadvantage in relation to very important decisions.”

The most important decision of those years was, of course, the one that took Britain to war in Iraq. In 2004, Lord Butler led a review into pre-war intelligence that found serious flaws, but stopped short of calling for resignations from either Blair or intelligence chiefs. Butler has defended the review in the years since – but might more formal cabinet decision-making have averted the war? He chooses his words carefully.

“I wouldn’t say that necessarily the decisions would have been different, but a lot of very good papers were prepared which weren’t circulated to the cabinet. Presentations to the cabinet were mainly oral; ministers came along and there was no set agenda, so they didn’t know what decisions they were going to have to take until they got there,” he says. “When you have a situation as important as a decision about war and peace, I think everyone’s got to be rigorous to make sure decisions are based on knowledge.”

Butler is a walking resource on post-war British government, and he acknowledges that today’s civil service has to confront some issues of greater complexity than in the past – on immigration policy, for example. Still, for a man who admits to being drawn to working in government by the highly-competitive entrance exam, handling complexity is what the civil service is all about; with the right leadership, he says, it will rise to the challenge. “There are a lot of very, very tough problems, but the civil service constantly needs to be challenged to produce the goods.”

Butler in service: a CV

1961 Graduates from University College, Oxford with BA in Literae humaniores and joins Treasury
1972 Appointed private secretary to Edward Heath
1974 Becomes private secretary to Harold Wilson
1975 Rejoins Treasury to work in General Expenditure Unit
1982 Becomes principal private secretary to Margaret Thatcher
1985 Named second permanent secretary at the Treasury
1988 Succeeds Robert Armstrong as cabinet secretary
1998 Retires from civil service, becoming master of University College, Oxford and chairman of HSBC bank
2004 Leads review of pre-Iraq war intelligence