Why do people keep daily diaries

The news that the diaries of the journalist Kenneth Rose are about to be published this autumn has led me to wonder why do people keep daily diaries, saying what they have done every day of their lives? As someone who has been a diarist for over three decades, I have been pondering about this for years, but as I consider Rose to have been a great diarist who knew everyone worth knowing in Britain for half a century, it has made me confront the issue.

There are lots of reasons that people give for keeping diaries, but how many of them true, and might it just be a case of pure narcissism? I have walk-on appearances in the diaries of some famous modern British diarists, including James Lees-Milne, Alan Clark, Kenneth Rose, Alan Bennet and Woodrow Wyatt, am I am starting to wonder if the act of diary-keeping is a form of malign psychological disorder.

‘Writing a book is an adventure,' Winston Churchill told a luncheon in the Grosvenor House Hotel in London in November 1949. ‘To begin with it is a toy, then an amusement. Then it becomes a mistress, and then it becomes a master, and then it becomes a tyrant and, in the last stage, just as you are about to be reconciled to your servitude, you kill the monster ad fling him to the public.' But that is not always the way with diaries. They often stay as your master, and you end up doing things and going to places simply in order to satisfy your diary, rather than yourself.

It's a mildly boring chore to devote ten minutes every morning to remembering what happened the day before. Diarists tell themselves, and each other, that they will have something to charm and interest themselves when they reach old age, or maybe their grandchildren. There's also always the chance of perhaps publishing them before everyone's forgotten who the people are who appear in them: a friend of mine wants to call her diaries: ‘Embarrassing Things That Famous People Have Told Me in Confidence.' But is that really the true reason we do it?

Perhaps for some – such as politicians – diary-keeping provides a rare opportunity to be completely truthful, and to show that they were decent people all along. People tend to buy political diaries who would never consider reading the same person's memoirs. Another motive has been to show off how many rich, successful and famous people whom the diarist knew. For others it allows them to confess to sins from beyond the grave. Others might hope for a tiny fraction of the literary immortality that has rightly been awarded to great diarists of the past, such as John Evelyn, Samuel Pepys, Henry 'Chips'Channon and Harold Nicolson.

Some diarists want to be able to ridicule their enemies and celebrate their friends. (James Boswell's Life of Johnson, which is effectively a well-disguised diary as much as the first great work of biography, does both superbly.) Tony Blair's highly effective press secretary, Alistair Campbell, obviously wrote his for the vast sums of money it garnered him. By contrast, Britain's most senior general of the Second World War, Field Marshal Lord Alanbrooke, used his diary as a safety valve to complain to, in order to stop him exploding with anger and irritation at Winston Churchill and the American generals. Some people keep diaries in order to help historians, to defend themselves from accusations of wrongdoing, or to inform their autobiographies. Whenever I meet someone in an important post in which history will be interested, I urge them to keep a diary. About half say they will, but only one in ten ever really does. (Some politicians keep diaries as a form of self-defence, though in America they can be subpoeana'd, so few politicians there take the risk.)

Yet I have long suspected that the true reason that we are driven to record our daily existence is out of a low-level, but slowly rising, terror that we don't matter and that nothing we ever do or say or write will ever matter very much for very long, and certainly not once we're dead. It is the fear that a life left unrecorded is an empty, worthless life. Spending tens of thousands of hours noting down what we did somehow means that in some way it either matters now or perhaps might one day be interesting to someone else, perhaps centuries from now. Diaries are thus a pathetic, illogical, doomed way of trying to matter to an uncaring Providence, and because we diarists are irrational enough secretly to believe that we can seduce and titillate Posterity, our diaries start to dominate us.

I find that even at the age of 55 – when I really ought to know better – I will do things and go to places that I don't particularly want to, solely in the hope that it might give me something to tell my diary. However ridiculous that is, I believe I'm not alone. It is a disorder that I believe is fairly common to diarists; certainly you can spot acute forms of this deranged phenomenon in the diaries of Alan Clark and Woodrow Wyatt. Talking to fellow-diarists - especially chronic ones suffering from the most acute form of diary-disorder - they'll admit to it.

There ought to be a medical term for the compulsion to do something in order not to seem to have led a boring life long after you're dead, because you genuinely believe that your life is more worthwhile because of what you jot down the next day in a little book. The vast majority of the population simply get on with their lives, not enslaved to this compulsive need to record. Why not diarists?

Of course we know how incredibly important the historical evidence provided by diaries can be. They record what happened on the day, not what the writer remembered happening years later, and are therefore much more accurate. I have recognised this particularly strongly over the past four years while researching for my new book, ‘Churchill: Walking with Destiny', which will be published in Chinese in October.

Winston Churchill

When Winston Churchill became prime minister in 1940, he ordered that no-one in the Army, Government or Civil Service could keep a diary. Had the Germans invaded Britain, captured diaries would have proved invaluable militarily, and the German Propaganda Minister Dr Joseph Goebbels would have been able to make excellent propaganda quoting selectively from them. Yet virtually nobody obeyed Churchill's instructions, and even his own private secretary, John ‘Jock' Colville, kept a diary that has proved an historical treasure trove. As well as Colville, Anthony Eden the Foreign Secretary, kept one, as did his private secretary Oliver Harvey and the Permanent Under-Secretary, Sir Alec Cadogan. Sir Alan Lascelles, King George VI's private secretary, kept one, and so did the King himself, which I have been able to research for my new book. Overall, some fifty individuals around Churchill kept them. They thought they were living through extraordinary, historic times and that future generations would be fascinated by their actions, and they were right.

Yet today we don't keep our diaries because we know that future generations will devour very sentence about what it was like to live in this post-heroic modern age. Instead, we keep diaries for reasons that we have internalised, and which are really quite unhinged. And yet we know we will keep on doing it, probably until our eyesight goes – which is what finally stopped Samuel Pepys – or until we have no more days left to record.

I have no idea whether my own diaries will ever be published – or are even publishable – but I hope they'll remind me of people and events that will charm and amuse me as I enter my dotage. I picture myself in a huge armchair chortling at their contents, and stroking the vast pile of books I've used to write them in as a miser does his stash.

A political diarist needs three things for his journals to succeed, be he Joseph Goebbels, or the British foreign secretary Douglas Hurd, or the Labour Party chancellor of the exchequer Hugh Dalton, or the journalist CP Scott, or the founder of the BBC John Reith, or the Conservative foreign secretary Harold Macmillan (one of the very few prime ministerial diarists). The first is access. He – political diarists are very rarely female – needs to have been at the key place at the right time to record what was being said and done. One of the reasons why the Conservative MP Alan Clark regretted resigning his seat in the House of Commons was that it removed him from the centre of events.

The next prerequisite is an observant eye for the telling detail that instantly captures an event or a personality. Great literary talent is not necessary, but it is vital to have a sense, however biased, of the characters and moments depicted. Nor is much political judgment required: Samuel Pepys supported the disastrous King James II of England, Alan Clark defended the profoundly unimpressive Sir John Major, and ‘Chips' Channon admired perhaps the worst prime minister of the century, Neville Chamberlain.

The last and most difficult prerequisite is that the diarist himself must be an interesting figure, whose weaknesses, enthusiasms and moral (or immoral) development are worth following in themselves. We'd love to join Samuel Pepys for a glass of claret, James Boswell for a cup of coffee, and Chips Channon for one of his Benzedrine-laced cocktails at his parties in Belgrave Square.

The diary of Sir Anthony Eden, Britain's foreign secretary and future prime minister, for 18 November 1944, after two months of completely empty pages, sums up the paradox that lies at the heart of diary-keeping: ‘I have been disgustingly idle about my diary of late and what made it inexcusable is that recent weeks have been of vivid interest,' he wrote. ‘But they have also been so absorbing that I haven't had a moment to write.' It is a strange fact that the greatest diarists are usually not those written by decision-makers – very few prime ministers have been daily diarists, for example – but rather by those on the fringe of events. The people who are actually making history are often too busy and exhausted to write about it.

Samuel Pepys was Secretary of the Navy, but not one of the most important men in England by any means. 'Chips' Channon was a mere parliamentary private secretary, the most junior Government position. Alan Clark never made it to the Cabinet, and Harold Nicolson only lasted in office for one year, and then only in a minor role. Yet their diaries will live long after those of their more powerful contemporaries. Jock Colville entitled his fascinating diaries of life with Churchill ‘Fringes of Power' and counter-intuitively it is the diaries that are written from that perspective that usually last longest.

Yet for all their being written at one remove from the actual exercise of power, diaries automatically convey – so long as they are not doctored after the event – a far more immediate sense of what day-to-day life was really like, compared to later narrative accounts. Tiny details that illuminate are often worth pages upon pages of description. In James Boswell's Life of Dr Johnson and in his superb journals the eye for the unusual can cast sudden light upon an event or occasion like a lightning flash across the night sky. A couple of sentences from a diarist who was present can sometimes be far more useful than pages of official documentation. Diaries are invaluable for the study of history, because the writer crucially does not know what is going to happen next, whereas the writers of memoirs always do.

The perfect diarist, it seems from my reading of hundreds of them for a living over thirty years, is a sex-addicted junior politician with a good eye for detail. Fortunately, the House of Commons is full of such people.