Monday, February 12, 2007

Chess and Evolutionary Theory

Today is Darwin Day, which honors Charles Darwin (February 12, 1809 - April 19, 1882). So I thought I'd say a few words on the topic of chess and evolutionary theory.

I began thinking about the subject of this essay a few months back when I read Garry Kasparov's "An Evolutionary Theory of Chess" (November 2006), which reminded me that the term "evolution" is quite common in theoretical histories of the game. Several works even use it in their titles, including Imre König's Chess from Morphy to Botvinnik: A Century of Chess Evolution (1977), Raymond Keene's The Evolution of Chess Opening Theory: From Philidor to Kasparov (1985), and Macon Shibut's Paul Morphy and the Evolution of Chess Theory (1993).

Observing the popularity of the term "evolution" among chess writers gave rise to several questions: Do the authors who use the term intend reference to Darwin's theories? How much is the use of the term "evolution" accurately informed by a post-Darwinian paradigm? And how much might current developments in evolutionary theory (especially views that recognize the importance of symbiosis and lateral genetic transfer) broaden our insight into how chess theory really "evolves"? Some of the answers I found surprised me.

A Problem with the Term Itself
Former World Champion Max Euwe may well have started the trend toward evolutionary thinking in his ground-breaking The Development of Chess Style (1966) where he writes:

The history of chess (under its present rules) is the study of the growth and gradual change of the strategic ideas of leading players of succeeding generations. Taking note of this evolution and throughly grasping it is the very thing which makes for better judgement and an increase in playing strength. The development of a chess player runs parallel with that of chess itself; a study of the history of playing methods therefore has great practical value. (Introduction, p. 8).

People have generally read this passage as saying that chess offers proof of the old notion in biology that "ontogeny recapitulates phylogeny," with the idea that our individual history mirrors the evolution of "the race" because the evolutionary story somehow inheres to our genetic make-up. By reviewing chess evolution, according to Euwe, we can speed up our individual progress by, as it were, skipping ahead in the story.

Implicit in Euwe's treatment of chess "evolution" is the notion that the "march of progress" in chess (both individually and collectively) is ever upward toward greater improvement. However accurate that may be to our theoretically finite game (where Truth is knowable in theory, if often tricky in practice), it does not do justice to a post-Darwinian understanding of biological development. Rather, it reflects both the common use of the term "evolution" and the misunderstanding that the term has helped to cement into our collective consciousness.

In his famous "reverie" on the opening of the New Haydn Planetarium, titled "What does the dreaded 'E' word mean, anyway?" the late Stephen Jay Gould reflected on how the word "evolution," which Darwin himself rarely used and would have thought inaccurate, came to signify Darwin's prefered description of "descent with modification" for describing his theory. As Gould writes:

"Evolution," from the Latin evolvere, literally means "an unrolling"--and clearly implies an unfolding in time of a predictable or prepackaged sequence in an inherently progressive, or at least directional, manner.... The few pre-Darwinian English citations of genealogical change as "evolution" all employ the word as a synonym for predictable progress. ... Thus, on these two fundamental grounds--lack of inherent directionality and lack of predictability--the process regulated by natural selection could scarcely have suggested, to Darwin, the label "evolution," an ordinary English word for sequences of predictable and directional unfolding. We must then, and obviously, ask how "evolution" achieved its coup in becoming the name for Darwin's process--a takeover so complete that the word has now almost (but not quite, as we shall soon see) lost its original English meaning of "unfolding" and has transmuted (or should we say "evolved"?) into an effective synonym for biological change through time.

This interesting shift, despite Darwin's own reticence, occurred primarily because a great majority of his contemporaries, while granting the overwhelming evidence for evolution's factuality, could not accept Darwin's radical views about the causes and patterns of biological change. Most important, they could not bear to surrender the comforting and traditional view that human consciousness must represent a predictable (if not a divinely intended) summit of biological existence. If scientific discoveries enjoined an evolutionary reading of human superiority, then one must bow to the evidence. But Darwin's contemporaries (and many people today as well)would not surrender their traditional view of human domination, and therefore could conceptualize genealogical transmutation only as a process defined by predictable progress toward a human acme--in short, as a process well described by the term "evolution" in its vernacular meaning of "unfolding an inherent potential."

Following Gould, we might say that the notion of "evolution" presented by Euwe is consistent with the ways in which Darwin's theories have been absorbed in the West, even if his ideas about "evolution" are not consistent with some basic assumptions made by Darwin himself. The most important points (all related) that Euwe and others ignore are: (1) that Darwin did not depict biological change as "progress"; (2) that Darwin thought that adaptation happened due to unique local circumstances and not due to the natural movement toward an "ideal"; and (3) that Darwin therefore did not believe that the current state of things was necessarily better than any prior state.

Kasparov's Theories
Kasparov's theories of chess evolution do break from the model offered by Euwe. Though his writings are often rather generalized, they are also consistent with Darwin's own ideas. In his essay "An Evolutionary Theory of Chess," for example, Kasparov writes:

The position of chess in culture is reflected by the way the game has evolved along with society. Every generation has its leading schools of philosophy, art, and, you may be surprised to learn, chess. And every generation of chessplayers has had its influential leaders. We'll look at a few of these leaders, their chess, and how the game changed in sync with the world around it (Kasparov 1).
In other words, Kasparov is suggesting that the style of chess that evolved at any particular time was a local adaptation to larger historical circumstances. This is analogous to Darwin's theories. To give one example, from among many, Kasparov writes:

With Philidor we also see the beginning of an interesting trend that illustrates how cultural and social currents spread and cross, even onto the chessboard. The best chess masters of every epoch have been closely linked with the values of the society in which they lived. The cultural and political background are reflected in the style and ideas of their play. It was no coincidence that chess developed and flourished in Italy and Spain during the Renaissance.

Correspondingly, Philidor developed his concept of positional play during the Enlightenment and the era of rationalism. Looking back we can even find a historical context for Philidor’s most memorable maxim, “the pawns are the soul of the game.” His contemporaries, he believed, failed to understand the strategic importance of the pawns, the weakest but in some ways most complex and essential members of the army. Do not these thoughts of “power to the common man” in some way presage the French Revolution?
What's more, Kasparov does not appear to believe that the current state of chess play is qualitatively superior to the play of the recent past. The players of today may know more, but they don't play better. In an interview, Kasparov had the following nuanced exchange with Mig Greengard:

Mig: In My Great Predecessors you write about the evolution of chess over the decades. As with evolution in nature it’s hard to see up close. When you look at the top few players, the top five or ten, do they play at the level you and Karpov were playing at in the early 1980’s? Has there been an increase in sophistication and quality?

Kasparov: There is increasing sophistication, but as for quality... When I played Karpov there were two of us well ahead of the rest. Today, can you say Anand is so much better than Leko? You have Anand, Kramnik, Leko, Mickey, Topalov...

Mig: Okay, but are they better than you and Karpov were in 1983?

Kasparov: Technically speaking, better because every new generation is better than the previous generation. But is the quality of the chess in the middlegame now better than 20 years ago? Openings are another matter. Any GM today could, technically, play better in the opening than you did then because of databases. I think the quality of the chess we played in Leningrad in 1986 was phenomenally high. I don’t think today’s players could beat that. In my view that was the best match we played and I don’t think they will ever reach it. Today’s players know more even in the middlegame positions because they learned from us, but in terms of the decision-making processing in ’86, no way.

Though he accepts as a matter of course that the enlargement of chess theory will make the current generation of players quantitatively more knowledgeable than those of the past, he does not accept the notion that they are qualitatively better in terms of their "decision-making processing." Mig even presses him on that point, but Kasparov refuses to capitulate to the common notion of evolution as progress, suggesting that the players of the past may well have been better than they are today. Theory may have enlarged, but individuals are no better, while the population has moved toward greater uniformity in its ability. All very analogous to Darwin's theories.

Lateral or Horizontal Transfer
Kasparov's use of the term evolution comes closest to reflecting a Darwinian paradigm of any chess writer, but it also still subscribes to some older and limiting notions of descent. After all, in Kasparov's view, change in chess still happens exclusively in tree-like fashion, descending vertically to the present generation of players from "Great Predecessors." Change is the result of individual genius, not the collective workings of many players who creatively adapt both old and new ideas to novel contexts. Furthermore, in Kasparov's view, evolution happens only through combat and not cooperation. He is, after all, the author of Attacker's Advantage.

A more recent and still emerging paradigm of evolution (supported by genome studies) stresses instead the ways that horizontal gene transfer (especially through viruses, by the same means we use for gene therapy) within populations has a powerful effect on biological change through time, and that evolution is as much driven by cooperation as it is by conflict. Though Kasparov's more traditional notion makes for a good story, it does not present the whole picture.

There is a growing literature on lateral transfer, which offers a powerful explanatory system for understanding the speed with which genes (or, by analogy, ideas or "memes") develop and spread. It is therefore a valuable addition to our understanding of changes in chess theory, especially in this age of the internet with its incredible escalation in our ability to communicate and transfer knowledge "peer-to-peer" rather than waiting upon our forefathers....

I am still compiling examples of the phenomenon, but several instances of lateral transfer of chess ideas can be readily observed. The obvious relationship between the Goring Gambit (1.e4 e5 2.Nf3 Nc6 3.d4 exd4 4.c3 dxc3 5.Nxc3) and the Smith-Morra Gambit (1.e4 c5 2.d4 cxd4 3.c3 dxc3 4.Nxc3 Nc6 5.Nf3) is a simple example which can explain what I mean. Basically, a method or pattern that is shown to be useful in one opening line is adapted to another line by way of analogy. Even Black modes of defense (such as declining either gambit with an early ...d5 or ...Nf6) transfer readily between the two openings and almost certainly had some effect on their initial development.

In other words, chess ideas do not descend from founding fathers (whose names become associated with the lines they are imagined to have originated) along specific lines (such as within "the Orthodox Variation of the Queen's Gambit Declined") but are inspired through a process of cross-fertilization between players and between opening "lineages."

In his wonderful chronological collection, The Evolution of Chess Opening Theory, Raymond Keene notes several examples where developments in one opening helped originate others. He writes, for example, that "Alekhine's Defense, with its deep insight that the avalanche of White centre pawns is, perhaps, not so deadly afer all ... was the precursor of such provocative defences, now commonplace, as the Pirc and first move fianchetto, although, ironically, Alekhine himself disapproved of 1.e4 g6" (75). He also notes that while Frank Marshall adopted the Modern Benoni during New York 1927, the opening "really came into its own as a by-product of the greater understanding of the King's Indian Defense during the 1950s, and in the dynamic hands of Mikhail Tal it was elevated into a fearsome tactical bludgeon" (Keene 171).


Position after 6...h5!?

One good example I have found of lateral transfer involves an early ...h5 advance by Black against White's g3 fianchetto, which I illustrate in "Chess and Evolution: An Example of Lateral Transfer." The examples I show come from various openings -- including the Vienna, the English, and a line of the Sicilian which is essentially the reverse of the English line. What I think you will see if you look through these games is that the essential patterns and ideas -- analogous to genes (or what some theorists call "memes") -- are what transfer, and they do not arise either through descent nor solely through closer consideration of the line itself. Instead, these ideas seem to be transferred from one line to the next.

The idea of lateral transfer (which seems implicit in the writings of John Watson, from whom my main example derives) is very different from the story of evolution in opening theory as told by, say, Imre König's Chess from Morphy to Botvinnik, where the development of theory in the Orthodox Variation of the Queen's Gambit Declined descends in stages as various great players attempt to solve the problem of the Bishop at c8. For König, the story is of one of struggle between succeeding rivals, from Anderssen vs. Steinitz to Capablanca vs. Alekhine (pp. 74-96), with historically successive contributors to the theory along the way. That story is straightforward and fits well with traditional notions of evolution, as I indicate above. But it is neither fully accurate to how chess theory changes (since it leaves out the population of players as a whole) nor is it especially helpful to undestanding the rapidity of change in our own hyper-connected age, where information is exchanged with ever increasing speed. It is also a little different from the views presented by João Dinis de Sousa in his excellent "Chess moves and their memomics: a framework for the evolutionary processes of chess openings," which develops a theory of meme transfer that is more fully theorized than my own simple analogies yet remains confined to specific opening lineages (on the König model) with no accounting for lateral transfer between lines.

The most important new source of chess ideas is the computer chess program, which inspires with analysis and even with ideas (such as its surprising Rook lift against Kramnik). Often, multiple analysts arrive at the same idea at the same time because they are using the same program. This is a bit different than lateral transfer but achieves the same effect.

Evolution by Competition and Cooperation
There is a long tradition of summarizing Darwin's theories as simply saying it is "survival of the fittest" (everyone for himself and God against all), which encourages us to imagine adaptation in the competitive arena of predator and prey, animal and environment, with "nature red in tooth and claw." Such a vision of evolution was especially encouraged by the Social Darwinists of the first Gilded Age in the late 19th Century, and such notions are especially popular today in our own second Gilded Age (especially among the red-meat Republicans).

The notion of evolution by competition in chess is contained in a title like Anthony Saidy's The Battle of Chess Ideas (1972), which suggests a similar survival of the fittest in chess theory. Interestingly, Saidy's book opens with a rather mythic history of chess itself, in which the game is imagined as emerging in conflict, suggesting through this myth of origins that the seeds of evolution through conflict were planted long ago. "One may look for symbolic meaning in the game's attraction for great revolutionaries--Karl Marx, Lenin and Fidel Castro" writes Saidy.

Chess is clearly a war game. It shows us two opposing armies comprised of royal hierarchies and their assembled soldiers. Its aggressive content is evident. World Champion Lasker attributed its unique attraction to the prime human delight in a fight, which he regarded as the essence of chess. To him, chess was an intellectual microcosm of the struggle of all life.... Thus, Lasker left us no new strategy nor a legacy of beautiful games. Rather, his games approach in chess the quality of that mythical being which he postulated in a philosophical writing, the Macheide ('Son of Battle')--a being which, evolving through eons of struggle and natural selection, reached a peak of indomitability (12).
Chess would appear the perfect symbol of competitive evolution, as White and Black struggle always in a zero-sum game of one-upmanship. But this vision of chess ignores the cooperative nature of any human endeavor. New versions of chess might be depicted as "the next level of chess evolution," but it is clear that chess itself evolved through combination of games (see also review by Taylor Kingston and the interesting essay "Is Chess a Hybrid Game?" and this website on The State of Chess Research) rather like the first cells emerged through symbiotic combinations of much simpler organisms.
Ideas are not put into battle only on the competitive stage, in actual contests between players. They are tested and analyzed long in advance, often in anticipation of a contest that never comes, often in small study groups of peers or teachers and students. What is more, after an actual game has ended, the contestants typically share ideas with each other in post mortem analysis sessions. These unrecorded "games" are actually more numerous than their recorded brethren and are as important as singular over-the-board encounters in helping to refine the theory of a line.
Thus, to ignore the cooperative nature of chess evolution is to ignore the most important part of chess.

There is a growing literature on evolution that emphasizes the importance of cooperation between organisms of different species in creating change. I could name a large number of books, but three that I found both useful and readable were Lynn Margulis's Symbiotic Planet: A New Look at Evolution (which argues that the role of symbiosis and incorporation or lateral gene exchange are the chief drivers of evolutionary change); Mark Ridley's The Cooperative Gene (which argues that the evolution of complex life was not possible until selfish genes were tamed through the creation of sex to allow for more productive non-clonal transmission); and Frans De Waal's The Ape and the Sushi Master (which argues that generosity and altruism must have evolved as they were selected for in socially organized animals). I recommend these to anyone who is interested in learning more about this alternative and important model.

To ignore the role of cooperation as well as conflict in shaping biological change is to ignore an equally important part of the story. I think it is also, ultimately, motivated by blind ideology. In order to arrive at a deeper notion of how biological evolution may apply to chess we have to leave ideology behind and look at the science itself.

In the U.S. currently, you will find less consensus on Evolution than you will on Global Warming--and that is only because of a sudden shift in our collective thinking about the latter due to incontrovertible evidence. We are a people who still wonder "which came first, the chicken or the egg?" when a sixth grade biology student elsewhere might be able to answer "the egg, of course, since it was laid by a proto-chicken whose genotype had not stabilized into the current species we know as 'chicken'--as developed through artificial selection not by God but by Frank Perdue (in his own image)."

Why do limiting notions of evolution persist in our everyday discourse, including about chess? Because we are willing to stop far short of real knowledge. If you look online, you are more likely to find jokes about chess and evolution than you are any enlightenment.
There is one intriguing essay titled "Reflections and Debates: The Evolution of Ideas about Chess" at the Chess Theory website (now buried behind PHP code, so I reference the Web Archive version) that tries to use a Kasparovian notion of situated evolution to explain historical developments in the game. Others have tried to use evolution to help develop better models for chess computers. Such attempts, motivated by real science, are rare, mostly because they are difficult. When you are explaining chess, the last thing you want to do is take the time to explain science as well.... It is so much easier to stick to familiar notions that help us dispense with explanation as much as possible. But a more accurate view of evolution can only aid us in understanding the history and the future of our game.


Ashik Uzzaman said...

Great analysis Michael! I liked Kasparov's "An Evolutionary Theory of Chess" as well -

Robert Pearson said...

A wonderful, wide-ranging post, Michael. Two points that I must disagree with you on, though. I'm pretty familiar with Republicans, and I don't think there are really that many today who would would qualify for the 'Social Darwinist' label--whether 'red-meat' or other variety. I haven't heard much talk of letting the poor starve to death as being good for the species around the old campaign headquarters, lately.


In the U.S. currently, you will find less consensus on Evolution than you will on Global Warming--and that is only because of a sudden shift in our collective thinking about the latter due to incontrovertible evidence.

I don't believe in collective thinking, but you could be right on the numbers; I must call you on the use of 'incontrovertible evidence,' though. I don't think that in science there is such a thing.

Anyway, I agree with just about all of your observations on evolution, and chess. Thanks for such a thoughtful, thorough and stimulating essay.

Michael Goeller said...

Thanks for the feedback. Yeah, the "red meat Republicans" remark probably goes too far.... Glad that even those who do not share my politics can still enjoy my posts!

There is no question, however, that the tide of public opinion on global warming has shifted dramatically in recent weeks. The science there has been extremely convincing for at least the past decade. Any disagreement is completely unsupported by objective science. Now we can finally have a debate about what actually to do about it!

Tom Chivers said...

I sometimes attempt to analogize chess history and chess developmental psychology with the development of mathematical concepts, which I studied at university as a mathematics degree. I've never managed to work it up into a coherent theory though.

Anonymous said...

Another exceptional article. Michael, I truly enjoy your long posts.

Kasparov's Amazon short "An Evolutionary Theory of Chess," didn't strike me as a very scholarly work. Everything in his article that could masquerade as an insight came ex post facto. What has he predicted?

Taken as a whole, the books you mentioned on "chess evolution" are an interesting bunch. I once had the opportuniy to purchase Koenig's work for $5 -- an opportuniy lost. There is another book in this genre, I forget the author, but the title is Dynamic Chess Psychology . Kasparov's new book on the openings of the 70's should also add some interesting material.

Howard goldowsky

Blue Devil Knight said...

A great post. I think that many who use the term 'evolution of X' really mean to give a historical account of progress in the field (e.g., the evolution of mathematical thought).

One bit that you might consider is the recent obsession with memes (Dennett, Dawkins). That is, units of cultural selection: little behaviors, habits, and customs that are passed on culturally (e.g., saying 'God bless you' when someone sneezes). They do not follow the same rules of biological inheritance, but work much more rapidly, and can 'reproduce' across people very quickly, even within a single generation. It would be interesting to see a meme analysis of chess ideas, fashions, and the like.

Michael Goeller said...

Howard: Thanks for the note -- especially the heads up regarding Kasparov's "Revolution in the 70s," of which I had not yet heard. It sounds like something I'd like to read. As for the "Dynamic Chess" reference: I assume you mean Beim's "How to play dynamic chess"? Or maybe RN Coles's "Dynamic chess." Both excellent and related to my topic.

Eric (Blue Devil): Concerning "memes": yes, that is basically what I am talking about, but I did not feel like setting forth the theory, especially since I was trying to discuss the evolutionary model directly. The works I cite (such as Kasparov) mainly work directly with evolutionary theory by analogy, so it didn't make sense to dwell on memes, though I make reference to the theory several times. There actually is a pretty good piece online that presents a meme theory of chess, as I mention in the piece, titled Chess moves and their memomics: a framework for the evolutionary processes of chess openings. You should read it. My main criticism is that he does not break from the limited model of Koenig and others which only considers opening memes as functioning vertically, across generations, rather than horizontally or laterally, across openings, which is the most powerful method of meme propogation, as ideas from one opening influence another. I have read some things by Susan Blackmore on memes, which I recommend. She's a good writer.

Michael Goeller said...

One additional point, Blue Devil: you are right to observe that "I think that many who use the term 'evolution of X' really mean to give a historical account of progress in the field." But the use of the term "evolution" in historical accounts of chess theory is very consciously referencing the biological model, as I hope my analysis makes clear. Certainly Kasparov is very conscious in his use of the term, as his essay title alone announces. Others, especially Saidy, are fully explicit in their references to the post-Darwinian meaning.

Blue Devil Knight said...

Michael: I hadn't noticed that meme reference in your piece: excellent. I will check it out. (Note, officially I am suspicious of evolutionary biology being imported into the social sciences, but I find the meme idea fun at the very least, if not a bit fuzzy headed).

Anonymous said...

Michael, I checked the book out at home, and what I was refering to was The Dynamics of Chess Psychology , another study of the "evolution" of chess. The author is Cary Utterberg (1994).

Howard Goldowsky

Anonymous said...

Very interesting idea, that opening development shares characteristics with horizontal gene transfer. Also interesting is the symbiosis such a transfer may cause in the chess world; I would think that players that import an idea from one opening to other are likely to encourage the use of both openings.

Of course, it is all moot, since the theory of intelligent design dictates that Cassia helped to create the openings. :-P

Unknown said...

My Revolutionary Opening Theory is set-up at:-
You can create a link to it on your
site if you want.
Mike Surtees (Alias Snafflevich)

Anonymous said...

This article of yours introduced me to "horizontal gene transfer," and now I hear about that in the New York Times today:

Missing your blog.

Michael Goeller said...

The use of the term "evolution" to describe the development of opening theory in chess goes pretty far back. Consider, for example, William Cook's "The Evolution of the Chess Openings" from 1906 (!):

More currently, there is "Chess Evolution" opening theory. Meanwhile, in the "ontogenic" realm of evolution, there is Arthur Yusupov's training course from Quality Chess:

Anonymous said...

I am reading David Quammen's new book The Tangled Tree: A Radical New History of Life. It presents many of the ideas you cover in your essay on chess and evolutionary theory. You were really ahead of him in many ways, especially considering that he does not consider the way HGT happens in culture as well as nature.