Emergent Systems - An Overview
These notes are an outline of points delivered to Adelaide Knowledge Management Group on 16 May 2018. The notes are to stimulate discussion. They make no special claim to expertise.
o These notes outline a few ideas about emergent systems
o the ideas are not original, but it is useful to summarize them
o on this topic my mind is full of doubts and confusion, which is a good way to be.
Are we agile thinkers? The adjective 'agile' really means the ability to adapt quickly to changes in physical or mental circumstances. Now 'Agile' has been adopted as a noun to describe a certain kind of adaptive computer programming, especially in business systems. However, the underlying principles of Agile are closely aligned with what has become called 'emergence' in natural phenomena. Except for those with a focus on scientific research and mathematics, many who are interested in emergent processes now come from a business background (the main audience for this particular article), although emergence is found in all kinds of human and natural systems.
o The emergent systems idea in business is most recently found in Agile processes
o I will not focus on Agile programming. I can't. My background is not in business.
o However, at the end of this overview I will indicate how Agile fits our theme, and you can add your own insights to that.
Let's begin with a literary analogy in what should be a truly creative arena, the writing of novels
o Mills & Boon romance novels are famous for being formula productions
o M&B novels are highly predictable. There is very limited room for variation.
o The production of M&B novels is highly efficient. They can be reliably be turned out in great numbers - currently over 100 per month
o The market segment for M&B obviously values predictability, and has since 1907
o Now think about writing a truly creative novel ...
o Authors vary in their initial mental road map for writing a novel
o Typically however an author will start with a thematic idea + a vague bunch of ideas about that theme.
o She may have some ideas about the characters who will turn up in the novel, but not all of them.
o Often she will first write some more or less unconnected segments about events and characters related to the theme
o As written segments related to the novel's theme accumulate, the author will begin to put them into some kind of order. She will look for ways to relate the segments rationally.
o As a larger written structure emerges she will see that some parts are redundant, and some connecting ideas are missing
o She will notice that some of the characters are too sketchy, and begin to have them developing as events in the storyline impacts them.
o As characters and events bump up against each other in the developing storyline she may realize that the logic of events has rather jumped beyond her control. Things are not really turning out in the way she expected.
o However that which is unexpected in a novel - or in life - is not always a bad thing
o The novel has emerged with an outcome that could not have been predicted at the beginning. It has become literature, not a formula production
o Presently we will return to look at what might be called Mills & Boon management versus novel writing management.
Feeling the stones to cross the river
o At the end of these notes are references to a couple of introductory books on complex systems (there are many other books available on this topic): a) Melanie Mitchell, “Complexity: A Guided Tour”; b) Somewhat more dry to read (it contains a little math), but with some very interesting and challenging propositions is Leslie Valiant, “Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex World”.
o Valiant begins by recalling a speech given by 1947 John von Neumann, who was predicting that computers would get by in future with only a dozen instruction types. This wasn't surprising he said because 1000 words were enough to get by in life, and life was much more complicated than mathematics, which is true.
o Valiant takes up the theme by noting that valid theories of science and mathematics -theoryful behaviour - account for only a small part of what humans accomplish. However the theoryless behaviour which mostly occupies our time is often surprisingly successful.
o How can theoryless behaviour be so successful? Valiant proposes that it works on a principle, not of certainty, but of being probably approximately correct.
o Valiant's book is really about how to develop computer algorithms that yield outcomes which are NOT certainly correct. Rather these algorithms yield outcomes which will be only probably approximately correct. If you run the algorithm a second time it is likely to yield a slightly different outcome. (Sound like human behaviour?).
o When approximating algorithms are given a feedback loop from external conditions (e.g. the real world) they will 'learn' and over many iterations converge on a reliable solution, just as humans can learn from experience.
o When self-learning, approximating algorithms are run over millions of iterations, and merge their outcomes with similar algorithms, the solutions which emerge cannot be tracked or explained by humans. This process has become known as AI, or Artificial Intelligence.
o The processes of AI are emergent processes.
The Boundaries of Chaos
o The phenomenon of bounded chaos has been described by the example of a marble running down one channel of a corrugated roof (James Gleick in an early popularized book on Chaos Theory).
o A marble released on the peak of a corrugation and running from the top of the gable down to the gutter will follow an unpredictable (chaotic) path, but emerge within the same corrugation unless its velocity is excessive. Its path will be different each time. (Strictly, the marble's path would be predictable if the initial starting conditions were identical, but in reality starting conditions which are even infinitesimally different - and they always are - will lead to a multiplication of differences and thus different paths).
o In the marble example, the peaks of the corrugation set the boundaries on the marble's chaotic path. Nature is full of bounded chaos like this.
o No two heartbeats are the same. Each heartbeat varies unpredictably within bounds. If those bounds are exceeded the heart fibrillates and the patient dies.
o No two utterances have the same sound quality. You voice varies unpredictably but within bounds. If those bounds are exceeded you will probably not be understood.
o The precise daily behaviour of an employee is unpredictable within bounds
o Cultures, laws, religions, ideologies and beliefs set bounds within whichEmergentSystems2 individuals are free to vary their behaviour. Such human-made boundaries however are apt to overlap and contradict each other, which can lead to chaos of a different kind.
o The challenges of governance, and of management, revolve around setting those boundaries within which individuals can make decisions and interact freely so that optimum outcomes emerge. If those optimums are of the Mills & Boon formula type, then the boundaries of variation will be narrow. If the optimum outcome is a creative society, or an innovative company, then the boundaries themselves will be wider, and often moved experimentally.
The Philosophy of Agile
o Disclaimer: I have a cheek talking about Agile in administrative spaces because it has not been part of my professional experience. Feel free to contradict me. Nevertheless ...
o There are organizations which appear to be fairly static, simple and predictable, yet function successfully over long periods of time. Think of, say, a small family pickle factory. There are many other organizations which are managed as if they are static, simple and predictable, but beneath the formal structures are not like that at all. (Maybe that is why there have been claims that only 10% of managers are actually effective..)
o The reality of running a government, or many large organizations, is often that there are mountains of intractable and mutually limiting problems. Problems may appear to be insoluble.
o The Agile method, as I understand it, is to approach large, complexes of problems by finding solutions, one by one, to fragments of those complexes, then eventually reach a point of having to integrate a collection of local solutions as opposed to choking on a hopeless, indigestible tangle of problems.
o Agile seems to have been adopted most enthusiastically by IT practitioners who have a natural penchant for organized thinking.
o The social philosophy of Agile is to have non-hierarchical teams working on solvable local problems. Their local solutions will tend to be only probably approximately correct within the context of the ultimate big mess of problems.
o As probably approximately correct local solutions are found, they will be aligned against other adjacent areas of solution. Then there will have to be a new cycle of experiments (or many cycles) until many adjacent areas of solution can gel. Note the similarity to AI.
o When the Agile process succeeds, the final complex outcomes which emerge may not have been thought possible, or predictable or even imaginable at the outset of the process.
o How well does the Agile process actually work? Some of you will have much more informed ideas about that than I do, so its time to pass this whole discussion over to you. My guess is that incipient dictators, egotists and empire builders will want to strangle the whole thing at birth. But maybe we can get past them, sometimes.
Emergent Systems - Part 2 - Doubt & Confusion - Questions to Ask
1. If emergence is found throughout nature, why wasn't it described much earlier?
- Of course we have been watching plants and children growing forever, emerging from seeds into something quite different. And of course, what Darwin noticed about emerging species in evolution had been seen by farmers since there were farms. Novel writers have been around for centuries. The best laid plans of mice and men have always gone astray. What was new? Noticing a system, a system with bounded mathematical variance, and the way that wholes emerge to be more than their parts.
2. What is an example of emergence in knowledge management?
- The Internet is an extraordinary example of emergence, chaotic in growth yet bounded in ways that make it the most powerful tool ever devised. Within the Internet itself there are many sub-systems illustrating emergence. Wikipedia is an excellent example.
3. Are there examples of knowledge management which are handicapped by ignoring the power of emergence?
- Yes, there are many, many examples of crippled knowledge management.
Example 1: Rigid, over-specified curriculums often strangle learning at birth because real learning is an emergent process of progressive reciprocal exchange between students of tutors. I have a special professional dislike of tick-box 'training programs' by 'instructors' in many businesses. You train dogs.
Example 2: When the knowledge existing within a company or a country is vacuumed up by some power holding elite and buried in the name of 'security', innovation and creativity will sooner or later die. Knowledge, like money, has to circulate to do its magic.
4. In actual, day to day management situations, isn't it inefficient to look for emergent solutions?
It can be. The Mills & Boone business model is very successful in its corner. Dictatorships often get things done while democracies meander. Do we actually want to live in an ant nest? The payoff in making space for emergence is often not in the short term. Agile type procedures try to take account of the need of large systems to keep functioning overall by limiting the search for superior emergent solutions to particular bounded problems, and seeking a wider integrated solution later.
5. In the human context of emergent Vs structurally rigid approaches aren't we really talking about different personality types which have always existed?
Yes, probably. Hierarchical cultures have an inbuilt rigidity which has always favoured top down, predictable control, and hence the kind of personalities which thrive within that structure. It doesn't mean that such people and such hierarchies are ultimately the most effective. However, they are easy to understand, have a strong historical legacy, and often a certain populist appeal.
Agile type strategies usually use a team with limited, defined objectives. They are supposed to be non-hierarchical, inquisitive, and ready to experiment with different solutions to see what works best. So far they have mostly been found within well defined, time limited projects such as IT programs. Even there, Agile enthusiasts sometimes admit that they can only function after being given space and protection by someone in senior management.
6,7 ... other questions??
.... Over to you
Emergent Systems – Synopsis
An emergent system is one where ...
a) the final system has properties and effects which exceed the properties and effects of its component parts, and
b) whose final properties and effects cannot be predicted from merely examining the component parts.
The prediction issue becomes critical in dynamic systems where human judgements have to be made but the true emergent form of the system to come is not known.
An example of a potential emergent process would be the politics and economics, then outcomes of trying to introduce, say, a Universal Basic Income of $400 per week for all Australians. (How would that actually restructure the society?).
However emergent systems are also found throughout nature. To take an extreme example, an alien looking at the component chemicals of planet Earth three billion years ago could not have predicted our wayward human presence and transformational impact on the environment. Indeed the prediction of human presence could not have been made in geological time until very recently, although many other complex systems had emerged and often merged into new life forms.
Andy Martin, Andy & Kristian Helmerson (October 1, 2014) "Emergence: the remarkable simplicity of complexity". The Conversation [recommended]
Denning, Steve (August 13, 2016) "What Is Agile?" [recommended] Forbes Magazine
Emma T (Dec 20, 2016) "Proof the agile approach works". South Australian Government
Holman, Peggy (2010) "Engaging Emergence: Turning Upheaval into Opportunity ". [context: change management in business] Kindle ebook from Amazon
Ito, Joi & Jeff How (January 30, 2017) "Emergent Systems Are Changing the Way We Think". Aspen Institute
Klein, JoAnna (May 8, 2018) "How the Father of Computer Science Decoded Nature’s Mysterious Patterns". New York Times
Lean Agile Training (April 15, 2017) "Emergent Leadership". Blog
Mitchell, Melanie. Complexity: A Guided Tour (Kindle Location 65). Oxford University Press. Kindle Edition.
Ockerman, Stephanie (January 2, 2017) "Getting to Done: Balancing Emergence and Delivery" [IT project contexts] Scrum.org website
Rajamani, Venkatesh (April 22, 2018) "Scrum Sprint Goal". [context: IT project management with Agile] Scrum Master Studio Series, Scrum.org website
Taber, David (June 9 2016) "Why political candidates can't do agile". CIO website
Valiant, Leslie. Probably Approximately Correct: Nature's Algorithms for Learning and Prospering in a Complex World (p. 2). Basic Books. Kindle Edition.
Wikipedia "Emergence". Wikipedia
Wikipedia "Interactional Linguistics". Wikipedia
Wikipedia “Mills & Boon” [romance novel publishers] Wikipedia
Thor May - bio for this seminar:
Thor has taught in tertiary institutions in 7 countries of the Asia Pacific region. His PhD was on knowledge worker productivity in educational institutions, drawing on 20 case studies. The productivity notion was examined beyond its typical definition in economics to consider individual notions of productivity, public and private, in various institutional roles. Taken together this multiplicity of personal productivity notions emerges as a status hierarchy in each institution, effectively controlling outcomes, and often contradicting the stated public objectives of the institution. In earlier, other lives (1980s, 1990s) Thor walked away from two PhD candidatures in formal and cognitive linguistics after realizing that existing static models of language systems could not account for the emergent properties of human languages.
Professional bio: Dr Thor May has a core professional interest in cognitive linguistics, at which he has rarely succeeded in making a living. He has also, perhaps fatally in a career sense, cultivated an interest in how things work – people, brains, systems, countries, machines, whatever… In the world of daily employment he has mostly taught English as a foreign language, a stimulating activity though rarely regarded as a profession by the world at large. His PhD dissertation, Language Tangle, dealt with language teaching productivity. Thor has been teaching English to non-native speakers, training teachers and lecturing linguistics, since 1976. This work has taken him to seven countries in Oceania and East Asia, mostly with tertiary students, but with a couple of detours to teach secondary students and young children. He has trained teachers in Australia, Fiji and South Korea. In an earlier life, prior to becoming a teacher, he had a decade of finding his way out of working class origins, through unskilled jobs in Australia, New Zealand and finally England (after backpacking across Asia in 1971-72).
contact: http://thormay.net mail : email@example.com
academic repository: Academia.edu
discussion: Thor's Unwise Ideas
personal site (legacy): http://thormay.net
ideas discussion meetup (bi-weekly): Active Thinkers
Emergent Systems - An Overview © Thor May 2018