Data-Based Art , Algorithmic Poetry : Geert Mul in Conversation with

The award-winning media artist Geert Mul (the Netherlands, 1965) has been making computerbased artworks for over twenty-five years. A large portion of his oeuvre, and his more recent work in particular, relies heavily on existing images, often sourced online. With the help of image analysis software, Mul reworks the pictures into new combinations, attracted by the unexpected results that algorithmic operations produce, and the revelatory potential they hold. The artist refers to this work as ‘data-based art’ – a term revealing not only of his own process as a maker, but also of his take on how people today engage with the world around them and make sense of it. At the conclusion of a large-scale retrospective of his work, Eef Masson spoke with him about some of the key ingredients of his visual practice and the inextricable relations between them: information, databases and collections; randomness and rules; and crucially, makers and audiences or users. In the course of the conversation, Mul also reflected on how his work ties in with much older traditions of play, in artistic practice, with data and the rules for their recombination.

been attracted by the unexpected results that algorithmic operations produce, and the revelatory potential they hold.The exhibit's curators therefore characterize his work as Rechen-Kunst, or 'calculation art'.Mul's images, because of their dependency on mathematical operations, are marked by randomness.Such randomness, however, does not entail arbitrariness, as the results of the calculations can always be traced back to very precise statistical procedures, programmed into the above-mentioned 'machines'.Importantly, Rechen-Kunst also results in a sum that is much 'more than its parts'. 2For the artist, it is here -in this sum -that meaning emerges.
Mul has produced computer-based art throughout his career, which now spans more than twenty-five years.His work comes in different shapes: printed, but also sculptural; installed and performed; self-contained and interactive.His pieces of the past fifteen years in particular also share a key feature: very often, they make use of existing images, reworked into new combinations.Since the early 2000s, the artist has been compiling databases of pictures; first his own, but later also repurposed ones, and increasingly sourced online.Today, he considers those compilations so central to his practice even, that he no longer thinks of the computer but rather of the database as his main medium.He uses the amassed images as starting points for a process involving what he calls 'pixel statistics': a form of advanced image search that allows him to select and (re)combine pictures on the basis of visual resemblance. 3age 1. Geert Mul, I've Been Watching You, 2006, computer prints on panels, 49.In February of last year, Mul and I took part in an expert meeting organized by Utrecht University (the Netherlands) on artistic and creative forms of data visualization. 4On this occasion, we engaged in a public exchange on the role of collections and databases in his visual art practice over the years and on his approach to data visualization generally.The talk coincided with the conclusion of a large-scale retrospective of his work, entitled Match Maker, at the municipal museum in Schiedam (also in the Netherlands). 5The exhibit established the historical significance of this work, demonstrating how over time, it transformed along with the technological developments that it also responded to.Taken together, the works presented attested to a persistent fascination with the possibilities of (visual) recombination -specifically in a formalized, rule-based manner.The text below is an adapted version of our Utrecht conversation, reflecting on, and integrating fragments from, some of Mul's work of the past one and a half decade.In the second half of the interview, we also reflect on the connections between this work and other, historical practices of artistic data exploration and recombination and the bending of 'rules' they involve.However, this association is also relevant in light of how, and to which end, such accumulations of objects are made accessible.An archive, you have argued, provides access to its holdings -likewise termed 'archives' -in reaction to queries or requests by users with specific needs.

Collections
Records can be consulted either on site, sometimes in their original form, or remotely, as ( digital) copies, and for various reuse purposes.In most of these cases, users make sense of the overarching theme of a given exhibit.He or she can make different selections, or use the same objects but order them differently in order to tell different stories.A database is comparable to a collection in that it contains discrete objects; however, those objects are not physical but consist of sets of features, expressed mathematically.In addition, retrieval and display do not happen in space but on a flat screen.Like curators, administrators of databases can select, order and reorder the objects within them.However, they can do so at a much higher speed (over 10,000 times a second) and on a much greater scale.In addition, they can copy and paste -or in other words: manipulate the composition of the databases' contents.Moreover, forms of interactivity can be built in.In the comparison with museum curation, this entails a very different sort of model for the disclosure, or exhibition, of a given set of data or information (a given I also explore in some of my work). 6 light of this difference in generative properties, ingesting a (selection from) an archive or collection into a database inevitably entails a number of fundamental transformations.But in addition, I also tend to repurpose the data I receive.For instance, objects in museum collections often come with metadata, as retrieved from institutional catalogues.In the interactive installations W4 and Horizons, I used some of those labels in selecting images for recombination and visualization.In the latter case, for instance, I've only pulled those works from the collection that were associated in the museum's database with the keyword 'horizon'.In addition, I also made use of information about the height of this horizon (in a painting's composition), by way of positioning works in relation to each other.In W4, the selection I do is explicated in the work's title, which identifies the fields I used for matching: the 'what', the 'who', the 'when' and the 'where' of the repurposed items.
EM: So how would you say does working with metadata in this case affect your process?
GM: Essentially, of course, data and metadata are made up of the same 'stuff': they're all strings of numbers.Calling them by different names imposes a hierarchy on them that I prefer to ignore.In my work, I approach data and metadata in the same way: as numbers that can be analysed, and used for purposes of comparison and recombination.In doing so, I heavily rely on algorithmic operations, as performed by a computer -but in the process, I also intervene.In the examples you mentioned, I select objects on the basis of (pre-identified) features, designating those as either relevant or irrelevant.But in addition, I also repurpose them, in that I use the metadata in ways they were never intended to, by the people who produced them.

Algorithmic Art
EM: Let's talk a bit further about the role of algorithms, in your art but also in your production practice.You just mentioned that in repurposing existing collections, you can sometimes rely on labels that others have assigned to objects, prior to your reusing them.But you often also source images elsewhere, recording them off air or scraping them online.While such images of course have technical specifications, embedded in their so-called 'metadata', they generally have not been subject to re-interpretation -whether by institutional cataloguers, or social taggers -in the same way that annotated collection objects are.Therefore, they lend themselves particularly well to automated image analysis, comparison, and recombination.These are procedures you often use; for instance, in the installation Match of the Day (2004-present, made up of images recorded from international satellite television channels) or the live performance Big Data Poetry (2013, with sound artist Michel Banabila, that makes use of downloaded images).
Can you explain your process further, zooming in specifically on how works that heavily rely on quantitative operations -calculations, measurements -of purely formal image features, can result in products that seem highly meaningful to human observers?Specifically, I am interested here in your role as a maker: your 'editorial intervention', so to speak, in the 'matching' that the computer does.GM: What I like about Match of the Day is that it clarifies some of the relations in my work between images and data, and databases and visualization.But also, between technology (or automation) and meaning -or how such meaning is attributed.
As you mention, Match consists of televised images.But in the first instance, the work treats those not as pictures but as mere data.The images are randomly recorded from about thirty international satellite television channels.Image recognition software subsequently analyses them in terms of 5,000 features, and compares them to all the others, arranging them according to the strength of the visual similarities between them.The 'top' results this generates are uninteresting, as they are pairs of identical images used by different broadcasters; those at the bottom of the hierarchy, likewise, are unappealing, as they appear to be entirely random combinations (because the numeric matches made somehow elude our visual capacities).But somewhere in the middle range, there is a sort of 'frequency' where images are close yet also different enough that they start to 'resonate'.Here, the analogies made through pairing strike a human observer as beautiful, for instance -or funny, or even politically charged (sexist, or racist).
My intervention here takes different forms.On the one hand, I control the system: I tweak the software's parameters so as to create an optimum chance that 'meaningful' combinations are generated, in the above-mentioned sense.On the other, I also select image pairs for presentation purposes.As a result, only about thirty out of each 10,000 matches the computer makes end up in the installation.I see this intervention as key, as it's the only way I can guarantee that quantitative operations produce qualitative results.But it is important to realize here that I do not censor; for instance, I do not eliminate matches that may be perceived as inappropriate or even offensive.Ultimately, what I seek to do is to explore and visualize a particular cultural landscape -in this case, a televisual one -by seizing upon the possibilities, in terms of significance, that it raises.In other words, I conceive of the meanings that emerge as part of the computer's matchings as products of the particular cultural landscape, or practice, they derive from.A landscape and practice that they also serve to illuminate.
Something else Match of the Day illustrates particularly well, is how compelling images are, when it comes to our human capacity, and urge, to attribute 'meaning'.When confronted with the matches the computer proposes, audiences inevitably feel the need to bestow some kind of sense on them.In light of this, the piece powerfully demonstrates our inability to see without interpreting.

GM:
The poetry idea is one I have entertained for a while -but recently, I've come to find it particularly useful in discussing my work with a broader audience.During the Schiedam retrospective, for instance, I've regularly taken groups of visitors on guided tours through the exhibit and in doing so, I've increasingly relied upon this metaphor to clarify my approach.A key motivation here was that it could help me subvert the notion that because I work with computers, I am concerned exclusively with the recent or 'new': with new images, new events, new ideas or new perceptions.
On the one hand, of course, the database -my main medium -roots my work in the hereand-now, in this particular epoch.For me, the database is not just hardware or software or a combination of those, but a cultural form and a way of seeing things.It is a mode of perceptionand importantly, one that is dominant in this day and age.Throughout history, we've been able to perceive the world by virtue of the media that we had at our disposal and this shaped our perceptions.Depending on the media available, we've conceived of the world as an organism, a machine, or a brain; of late, we've turned it from a machine into a computer or database.Artistic production illustrates this phenomenon in compelling ways.So, if I call my work 'data-based' art, I do so not only because it relies on data, but also because it explores, and at the same time gives expression to, a kind of data-based mode of thinking that is specific to the here and now, and that affects the way we see the world, comprehend it, and render it. 7But on the other hand, my algorithmic method also fits into a lengthy tradition.

EM: Can you expand?
GM: To clarify this, it helps to consider the aforementioned 'data-based' way of thinking from the perspective of art history.Over time, I would argue, artistic practice has consistently involved attempts to escape the conventions, or rules, of other knowledge systems.In doing so, it has taken two different roads; or in other words, it has been marked by two alternative tendencies.
Some artists have elected to simply ignore the rules, avoiding any sort of systematics altogether, in pursuance of the purely sensual.Others however have chosen to play with them: to move their audiences by twisting, bending or rewriting the rules.In the domain of poetry, whose medium is language -by definition, a heavily rule-based system! -we see those tendencies exemplified particularly well.
Arguably, it is the second tendency that points to where my method aligns with poetry-writing as a creative practice.In poetry, of course, the rules to play with are those of grammar: the formal system that governs the combination of sounds, words and phrases that we use to communicate.
What the poet does is to approach conventional grammar -those rules that in daily life, we abide by in order to express ourselves as clearly as possible, with the least possible 'noise' -as a construct that can be manipulated.He or she explores whether it's possible to tweak it (by changing or replacing existing rules) so that 'new' kinds of expression can be produced.Ideally, such utterances, or works of poetry, allow us to perceive our world in different ways, or even imagine different futures to the ones we're most familiar with.A set of formal rules -for instance a rhyme, or a rhythm -that is in itself meaningless, thus serves to create 'content' or 'meaning'.
The step from grammar to software, I would argue, is a modest one.Perhaps we can even describe the latter in the exact same terms as the former: as a formal system that generates expressions.Computer language, unlike natural language, is a construct that lies outside our human capacities (one that we cannot simply master through lots of practice).But it definitely belongs to the category of knowledge systems that we call 'language'.And, as in poetry, the rules of this language can be experimented with, and tweaked.Much like a poet does with conventional grammar, I tinker with algorithms, in order to change the system's generative features.
Similarly, my purpose is to conceive alternative ways of visualizing and in doing so exploring, data or information.This is essentially what my creative method consists of.

Timeliness and Tradition
EM: However much your art, through your data-based approach, is connected to the here-andnow, it also evokes associations with practices that go back many centuries.In previous conversations, you have aligned your work with historical examples of ars combinatoria or 'combinatorial art'.With hindsight, you claim, computational art can be placed in a lineage of practices involving a variety of combinatory systems and permutational procedures, that were considered to have revelatory potential (for instance, in the sense that they might be instrumental in revealing a body of hermetic knowledge). 8In this context, you've referenced the work of the thirteenthcentury Christian mystic and Neoplatonist Ramon Llull (hailing from the Kingdom of Majorca), who developed his own combinatory system consisting of letters and revolving wheels.He believed this system could be applied to all fields of knowledge, but he devised it primarily in order to be able to 'systematically prove the reality of universal Christian truths' (as part of his missionary endeavours). 9But others have devised similar structures before him, since antiq- GM: What my work shares with Llull's, but also other forms of procedural art, is that they all involve practices of recombination, and more specifically, rule-based forms of recombination.
In all of these cases, it is the recombining that is governed by the rules that the artist invents or tweaks, in order to determine a system's generative properties.In all the cases mentioned, a formal construction -whether original, as in Llull's case, or based on existing algorithms or rules, as in mine -pre-exits the 'content' that is produced with it, and that the audience encounters at a particular point in time.
Obviously, technological media lend themselves to this sort of combinatory practice particularly well.One of the key operations that databases enable is the retrieval of the data within it.
Such retrieval involves a (re)combination, in an infinite -or at the very least humanly unknowable -number of aggregations.But databased art also vividly illustrates how closely entwined technology and 'content' are.Considering how much the appearance of a work is determined by the execution of a specific set of rules, which are a function of the technology, the two cannot really be disentangled.This serves to reaffirm my perception that in my work, the database is more than just a tool: it is my medium, in that it shapes every aspect of the ' content' with which it combines to form a 'work'.

EM:
We have now spoken on several occasions of the creative interventions that you, as an artist, make, in 'meddling' with the mathematical operations that the computer performs as it generates image combinations.Interventions, I would argue, that somewhat undermine the idea of a Rechen-Kunst or calculation art, as the Dortmund exhibit called it.What the audience sees, after all, is not purely the result of calculations -nor could it be.One reason for this is that the revelatory potential of algorithmic procedures needs to be recognized as such by a human mediator, in order for it to get actualized.(By yourself in this case, since you identify such potential as a starting point for an act of editorializing.) Let's shift attention for a moment from your own role as an artist or maker, to that of your work's viewer or user.We previously discussed the audience's role in attributing meaning to the image combinations a computational system generates.In this context, you remarked among others that viewers, when confronted with such combinations, are bound to interpret them; in other words, that sense attribution in this case happens almost involuntarily.Some of your work however is also interactive.Arguably, then, the viewers' or users' role here is even more keyand presumably also more 'active'.Would you say that the audience, in those cases, is given more agency?
GM: Among the works we've already mentioned, there's several that require some kind of user intervention in order for image combinations to occur -Horizons, for instance, whose paintings merge into each other in reaction to the visitor's movements, or W4, which aggregates pictures on the basis of commands that the user gives by selecting keywords with a lever and buttons.Another example is God's Browser (2010-present), an interactive installation that generates, in the moment, a film composed entirely of images sourced from the Internet, but whose sequence (and speed) the user can influence by manipulating a theremin.In this work, image analysis is instrumental not only in determining image selection but also as a basis for the production of sounds (which are similar, if they accompany similar images). 11 the one hand, of course, the users' role here is highly productive, in that the appearance of the works changes based on their input.Since the number of image combinations or sequences that these works allow for is pretty much infinite, the movements or choices made may even result in compilations that have never occurred before, and most likely won't occur again.In those instances, the users' role seems particularly decisive.In God's Browser, I would argue, the music further underscores this apparent agency.For even if the audience cannot (directly) determine the quality of the sounds produced, those do serve to render their engagement with information sensory -and in a different way than the images do.
On the other hand, one might argue that within the generative hardware-software set-up that each artwork is, the user is merely one in a much larger series of variables that determine the outcome of a calculation (or what can be seen, and heard, in a particular time and place).A variable, in this case, that can tweak other variables -but a mere variable all the same.Within the dynamics of the system, the user is one of the wheels that make things turn, rather than to feature somewhere at the top of a hierarchical chain.This is true for most of my works, including those where the viewer's role is merely to attribute meaning to the visual or auditory combinations that the system generates.(Because without interpretation, there simply is no 'work'.) But the interactive pieces certainly foreground this circumstance -and in the process, perhaps, raise doubts as to how far in fact the user's agency reaches.
Yet none of this is to deny the very crucial function that such usage, or manipulation, plays.I would even argue that because of its importance, some of my interactive work can only be truly appreciated from what I would call a 'third person's perspective'.For after all, if it is in the interaction between hardware, data, a set of 'rules' for their recombination, and the different 'variables' that play into thoseincluding also the interacting user -that such Clip 5. Geert Mul, God's Browser, 2010, projection screen, computer, interface (artwork © 2010 Geert Mul/Gallery Ron Mandos, Amsterdam; video by LIMA, Amsterdam).
However, your practice also diverges in important ways.Most significantly, I would argue, in terms of how you interpret or value both the products of algorithmic operations, and the data they serve to recombine -or more precisely, the cultural objects that those data in turn reference, or even constitute.In the first place, because the visualizations you produce, while sometimes relying on very similar datasets, were never intended as a basis for making scholarly claims.Like the practitioners of cultural analytics, you may be curious to discover large-scale patterns or developments in art or popular culture -but not in order to make empirical statements about them, using computational techniques. 13In fact, I would say, the different ways in which your work foregrounds the creative intervention of a maker, but also interpretive activity on the users' part, challenge the assumption (which oftentimes informs such ambitions) that computational analysis might generate more unbiased results than traditional humanistic methods.
But in addition, your practice also engages much more profoundly with data as a cultural given.For instance, your work also reflects on the realities -including, to some extent, the history -of data production and circulation; incidentally, dimensions that practitioners of cultural analytics tend to overlook in the data they work with (and quite paradoxically so, in view of their practice's name). 14On the one hand, because your work often provides an 'image of the times', especially in retrospect.Once again, the aforementioned The Library of Babel is a case in point, both in terms of its aesthetics and in terms of the online practices it alludes to.
But on the other, your work also seeks to explore what you call 'sites of cultural information'.
You're referring here not to physical sites, but to the sorts of (conceptual) 'spaces' where people produce, reproduce, or otherwise engage with specific kinds of data.
Can you elaborate on this idea of a site of information, and on your conception of its exploration as some sort of dérive in the psychogeographic sense?Additionally, can you say a little more about how it informs your artistic practice?
GM: As you mentioned, I conceive of my work as a series of explorations of different domains, or 'sites', of cultural information (or 'data').Depending on the type of site I am concerned with in each case -a city, an archive, a database -this information either takes on a material form (and is spatially organized) or is immaterial (and arranged according to some other logic).
In conceptualizing how I explore such sites, I take inspiration from the work of the philosopher Guy Debord, co-founder of the Situationist International [a Paris-based, Marxist-inspired movement, involving primarily avant-garde artists and political theorists, that was active between the late 1950s and early 1970s -EM].The situationists, and Debord in particular, laid out a programme for a so-called 'psycho-geography', which involved procedures for exploring urban environments in playful but also unbiased ways, unaffected by one's pre-existing interests or prior knowledge of the sites in question. 15The dérive, in this context, is a type of walk -literally 'drift' -that is both arbitrary (for instance, in that one can start it at a random point in the city) and highly structured (in that one follows a predetermined pattern, taking specific turns at a given rate).
In my work, I basically apply the logic that structures Debord's city walk to explorations of non-material sites, such as archives and databases.Essentially, the dérive is a very simple algorithm: a mathematical rule, executable also on large accumulations of data, for the purpose of analysing them.Much like in the case of the urban drift, the exploration here generates different outcomes depending on one's particular point of departure within a dataset.Likewise, each outcome is equally valid, as each 'drift' through the data is equally likely to reveal some the specific 'substance' of a particular cultural site -whether this site is a city, or, as in the case of a work like Match of the Day, the domain which, at a specific point in time, we referred to as 'satellite TV' (that is, in the period between 2003 and 2006).But also, one that forces us to let go of some of the conceptions we had of it before. 16've been exploring the possibility for some kind of a 'data drift' for a while now; for instance, in the late nineties, in the eponymously titled film La Dérive (1998), in which I explore a physical site but using a similarly inspired editing logic.Yet in light of what we discussed before, there are very specific reasons why this approach to exploration is so well-suited to immaterial data, and at this particular point in time.Today, people often feel lost among the huge accumulations of data that are constantly available.The dérive, here, offers something to hold on to: a framework that structures one's explorations, while making the world more transparent and allowing us to intervene in it.A framework, moreover, that perfectly aligns with the data-based mode of thinking that we're currently so familiar with.So that ultimately, we can be receptive also to the (flickers of) meaning that confront us, as we encounter the matches that emerge out of a given cultural landscape.
photography in Rotterdam) and Museum Boijmans van Beuningen (a museum in the same city, whose collection covers Western art since the Middle Ages).The bodies of work these images are sourced from, acquired their status as collections by virtue of their association with a given cultural institution, and as part of its specific history.

EM:
In spite of how dependent your practice is on algorithmic procedures, you often refer to it as 'artistic' or 'creative research'.Moreover, you often call the products this research spawns 'poetry' (as in the title of one of the works we already discussed: Big Data Poetry) and refer to what you do as an act of poetry-writing.Can you explain why?
uity.In subsequent centuries, combinatorial art (in the visual, musical and literary domains) would shift in quality from the mystical to the formal, and involve further experimentation with systems for symbolic logic, semantic invention, but also pure process and play.10Can you elucidate once more how your work perpetuates this tradition of combinatorial art, and in particular, what the use of technological media contributes here?Image 5. Ramon Llull, figure from Ars Magna (Ars Generalis Ultima), ca.1305, manuscript illustration (source: Wikimedia Commons).