Much is being made of "serendipity" these days. While the term has many meanings, at its core it seems to refer to the phenomenon of accidentally discovering something fortunate while looking for something entirely different. This appears to be the same phenomenon that causes some to describe their application or note-taking system as an intelligent partner. See the first entry in this blog, for instance:
Because I would like to understand this phenomenon better, I recently read the following book, which promises a discussion of this phenomenon from a sociological point of view:
Merton, Robert King and Barber, Elinor (2004) The Travels And Adventures Of Serendipity : A Study In Sociological Semantics And The Sociology Of Science Princeton, N.J. : Princeton University Press, 2004
However, the book offers less a discussion of the phenomenon and more a historical account of the vicissitudes of an invented word. After along discussion of the history of the conceptual history of "serendipity" from the time of its invention by Horace Walpole in the eighteenth century to the middle of the twentieth century - which I found rather unexciting - Merton introduces some interesting notes and discussions of the true role of serendipity in science. Thus he discusses the relation of planning and research in a most interesting way. 1
Relying on the discussion of some scientists employed by laboratories in industry, he comes to the conclusion that the "art of profiting from the unexpected, then, that art, to return to Poe's phrase, of 'calculating upon the unforeseen,' is an art of general planning. The scientist lays out a general area of problems in which he is interested, but he stays clear of formulating specific problems too far in advance. One thing leads to another, and who can tell what will turn up in advance. It is foolish as well as presumptuous to make precise plans for future plans. ... The crowning reward of general planning is that it will preserve a freedom of inquiry, a freedom of opportunity, that it is not only rational and efficient, but is also a part of a good way of life" (192).
He also quotes another scientist, who speaks of "controlled sloppiness" as a principle that "permits the occurrence of fruitful accidents", tracing this idea to the fact that scientific work is never without loose ends and that in the absence of a rigid plan it is possible to pay attention to the untidy ends, which ultimately "may turn out to be of considerable importance." Indeed, "compulsive tidiness in experimentation" may be even more crippling than in other areas of life (193).
I would agree that this is at the root of the phenomenon of fortuitous and unexpected discoveries, namely that in our research we must formulate a "general plan" but refrain from defining the specific problems in advance. It is, in fact, important that we do not decide ahead of time what the outcome or result of inquiries must be. We leave this open, and organize our research without any preconceived systematic idea or classify the data in a rigid way. This is precisely what Luhmann's Zettelkasten method does. It does not super-impose an a priori systematic order on the observations and excerpts made in the course of one's research, leaving as much as possible open the possibility of viewing the information in all kinds of different contexts.
Merton also adduces an observation by Herbert Butterfield to the effect that of "all forms of mental activity the most difficult to induce ... is the art of handling the same bundle of data as before, but placing them in a new system of relations with one another by giving them a different framework, all of which virtually means putting on a different thinking-cap" (265). If this is true - and I think it is true (at least to some extent) - then the way we organize or plan our research is indeed much more important than most scholars and researchers think. It is therefore also more important to reflect on such a mundane thing as taking notes.
 This all has to do with for Merton with what he identified in 1948 as the " serendipity pattern," which, he thinks, "refers to the fairly common experience of observing an unanticipated, anomalous and strategic datum which becomes the occasion for developing a new theory or for extending an existing theory" (196). By the way, I would be very surprised, if someone like Luhmann (or his partner in communication) had not taken notice of this presumed pattern and found that it was important for his own purposes. Asking himself whether such an approach makes discovery the product of mere chance, he argues that "the real problem ... [concerns] the generation of chances with a sufficiently increased [verdichteten] probability of their actual selection." That is, we have to plan for the possibility fortuitous discoveries.