Transparency in government is all the rage right now – so much so that even MI5 has jumped on the bandwagon. To celebrate its 100th anniversary, the agency gave Christopher Andrew, a Cambridge Professor of Modern and Contemporary History, access to its 400,000 files. Six years later, The Defence of the Realm has been published, charting the organisation’s development from its beginnings as a two-man team set up to counter pre-Great War German espionage, to its current incarnation as a 3,000-strong force working largely on counter-terrorism.

The decision to write an official history is quite a leap into the light, particularly considering that it was only in 1989 that the Security Service Act placed MI5 on a legal footing; traditionally, the service’s existence was not publicised in any way. When Andrew was selected for the job, he was already known to MI5, having cooperated with it on two earlier books based on the files provided by KGB defectors.

There have been accusations that he is too close to MI5 – a ‘court’ historian rather than a critical one – although comments in the preface show that he has made the case for transparency; he rails against one example of censorship which he describes as “hard to justify”. Interestingly, in this case, the gag was imposed by another government department, and Andrew has suggested that the body that scrutinises the security agencies – the Intelligence and Security Committee – should look into this and other examples of reticence.

Andrew, who leads the weekly academic meetings of the Cambridge Intelligence Seminar, probably knows MI5 better than any other outsider, and he’s clearly close to the organisation: his sympathy for it runs through the book. Indeed, in order to become MI5’s official historian he had to become a member. This proximity will foster some caution on the part of other historians, but it certainly gave him an amazing insight into this most discreet of organisations. On July 7, 2005, for example, he saw the aftermath of the terrorist attack on London “from the headquarters of the agency which was chiefly responsible for dealing with it”, he says. “I have been living this while I have been writing it.”

Morale – one of the book’s key themes – was understandably low after 7/7, he says. “This really got to people,” he says. “You are simply bound to ask yourself the question, if you are part of the operational team: ‘What could I have done to prevent this?’” Nonetheless, he continues, generally “morale has been pretty consistently high”. The external body that carried out the employment survey for MI5 in 2000 noted that it had some of the highest scores it had seen anywhere, beaten only by publisher Mills and Boon.

The historian says he saw evidence of that high morale on his own visits to Thames House, MI5’s headquarters on the north bank. “You keep going up and down in the lift and you get some impression of what the mood is,” he argues. “It doesn’t even need to be verbalised: you can tell if someone is unhappy with their colleagues, with the boss; if they’re talking in a positive way about what they’re doing next.”

Andrew attributes this positivity to the motivation of staff, the belief that they are making a difference, and the fact that there are “very few bastards” working there – as one MI5 staff member told a new recruit in 1953. Andrew also says the department has had a deliberate policy of employing people with a sense of humour: a 2002 recruitment poster shows a spy posting his application through a hidden hole in a brick wall. Andrew says MI5’s annual review, which has more “successfully satirised performance indicators than any other bit of Whitehall”, is more evidence of the humour that runs through the organisation.

Another characteristic of MI5 that has continued since its inception is the major role played by women. The story starts in the First World War, when female university graduates were recruited; goes through the Cold War-era, when Jane Archer, who joined as a secretary at 16, became the main Soviet expert; and is capped by Dame Stella Rimington’s appointment as the first female director-general (DG) in 1992.

It has not all been plain sailing. The agency’s inability to keep up with women’s expectations led to a minor revolt in 1972, in the form of a petition called the Woman’s Charter which called for an end to the ban on women becoming agent-runners. The consequent U-turn allowed Rimington to secure a place as the only woman on the first agent-runner’s course the following year. “It is little realised that the three most senior former intelligence officers to sit in the House of Lords are all women,” points out Andrew, who is critical of historians’ ignorance of the secret world. “It has not even occurred to gender historians that the secret world might be the area where the cracks began to show,” he says, arguing that women have flourished in MI5 because a secret organisation is “less bound by conventional bureaucratic forms”.

MI5 has become – and continues to become – less secretive: public acknowledgement of its existence was followed by the identification of its DG for the first time in 1992. This emergence from the shadows has been matched by a growing preparedness to engage with Whitehall and Westminster. Traditionally, MI5 avoided the bureaucracy of Whitehall because it could: unlike external security agency MI6 and communications hub GCHQ, it is not a “tasked” organisation whose work is directed by another department, explains Andrew. “MI5 just wished to cut itself off from government”, he says, explaining that senior staff members were keen to avoid interference from politicians or departments that might not understand the context of the agency’s work.

It wasn’t until 1946 that the DG began to make regular reports to the prime minister, and relations with Downing Street were cool for much of the 20th century – thanks in part to the withdrawn personalities of a number of DGs. Later, MI5’s reputation at Number 10 was seriously damaged when the prime minister was told that the former deputy director-general, Graham Mitchell, was being investigated as a possible Soviet spy (he was later cleared). “That’s a level of embarrassment which is almost beyond their imagination,” says Andrew.

The situation is quite different now, says Andrew. The secrecy that meant employees were unable to tell their own spouses about their job has been dropped, and there are now ‘family days’. Similarly, says Andrew, MI5’s presence in Whitehall has been growing ever since the mid-1980s, when outsider Sir Tony Duff was made DG. “MI5 officers take part in all sorts of Whitehall committee meetings, they know their opposite numbers [in departments], their opposite numbers come to Thames House, and it’s all a comfortable relationship,” says Andrew. This is very important when MI5’s main area of work is counter-terrorism, which “necessarily involves a wider range of government and counter-espionage bodies”.

MI5’s traditional secrecy has probably helped to contribute to its low profile in UK history, says Andrew: “Even many of the historians that have argued themselves silly over the ‘dodgy dossier’ [used by Tony Blair to justify the Iraq war] never took into account intelligence.” He dismisses the idea that this is because the files are not available: in the case of gender historians, for example, Rimington’s appointment was common knowledge.

This seems tough: it was common knowledge, but the files were not available to reveal whether her elevation was a freak occurence or signified an increasingly progressive organisation. But perhaps MI5’s growing openness will mean it will be ignored less in future. Andrew certainly thinks so: “The lapses and omissions of one generation of academics are the major research opportunities of the next,” he says.

MI5: from start to date

1909  
MI5 and MI6 start life as one organisation, the Secret Service Bureau, employing two army officers.

1910 
Rivalries over work areas and resources lead the organisation to be divided in two.

1914 
22 German spies arrested at the outbreak of the First World War, decimating the enemy’s intelligence network.

1939-45 
MI5 operates the very successful ‘Double-Cross’ system of double agents, feeding misinformation to Germany and contributing to the success of the D-Day invasion.

1963  
Kim Philby, one of the ‘Cambridge Five’ who successfully infiltrated the British secret services and spied for the Russians, defects to the Soviet Union after making it to the senior ranks of MI6 and becoming liaison officer with the CIA. Although he was the last of the five to flee to Russia, incorrect interpretations of intercepts meant the internal hunt for the ‘fifth man’ continued. An investigation of former deputy director-general Graham Mitchell on suspicion of being a Soviet spy was begun after Philby disappeared.

1972  
Women employees revolt over the ban on women becoming agent-runners.

1989  
The Security Service Act puts MI5 on an official and legal footing.

1992  
Dame Stella Rimington is appointed as DG and is the first to be named.