I was reminded earlier today of the famous Donald Rumsfeld press conference in which he said:

“Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.”

There was a considerable uproar in the press at the time, which mostly seemed to focus on these words as if they were a radical new idea… but I think I just considered his words to represent a broad concept that seemed very familiar… probably so familiar in fact that no-one ever says them!

I have a sense that the reason I was unsurprised may in fact stem from a working life that has included a fair amount of maintenance and support of fairly sizable systems.  How often have I made a change expecting one set of results, but got something else? How many times has an innocuous change in one part of the system had consequences days or even weeks later in another part of the system?

When considering maintenance / support mind-set, or really any kind of change, we might use Donald’s words to guide us a little:

There are Known Knowns

I see this as a broad equivalent of the stuff we expect to happen when we make certain changes in a system. Perhaps the change is well understood and / or the code being changed is well understood. And ideally we will have some unit tests we can run to confirm that we know what we know!

There are Known Unknowns

I think of this as being a little like that feeling you get when you change a method that is used in many places… you were only expecting to change one particular use-case, but you have this vague notion that something else uses this and you’re not really sure what the results will be in those other scenarios.

Hopefully, you will have some integration tests to run that will help you identify of unexpected consequences of wider use of this shared code.  With or without the automated tests, I recommend that you research the other callers of this shared code; the research will help you make the unknown known, and even if it does not help you test, it can help you instruct others on worthwhile areas to test if that’s the way your business works.

There are Unknown Unknowns

You know, systems are rarely trivial to understand, and my last two roles over some 10 years have certainly featured sub-systems that for some reason or another I never had contact with, and possibly never even knew existed! How can you possibly understand the system as-a-whole, especially as a new starter?  I feel that this is an area where the idea of large-scale integration tests or characterization tests, or similar (a topic I hope to write about shortly) can help us. It’s not a fashionable idea as far as I know, but one which has proven very useful to me on a number of occasions.  By setting up a test structure that monitors the behaviour of large parts of the system, we can confirm if changes have unintended consequences within that area. The unknown unknowns stay hidden in this scenario, until the little red flag on a failing test may lead you (with some research) to identifying some problem in some area of the system you had not even known about before.  In fact this larger-scale testing has been hugely informative for me; it has helped me to identify huge problems within a system that would never have been identified even with thorough coverage of unit tests.