Complex Network Analysis in Python – 2nd Edition Released

Today is a fine day! I woke up to an email from pragprog.com – a publisher for programming and computer science textbooks – telling me that an updated version of “Complex Network Analysis in Python” by Dmitry Zinoviev has just been released and since I bought the ebook I can download the new version for free! Ok – that’s an update to a textbook, what’s the big deal you might ask… Let me explain:

This summer, I embarked on a journey exploring networks and their analysis. I had decided to work again with materials from the “Medieval German Religious Plays and VR/AR”-project I had developed in 2016, mainly trying to model the stage interactions of characters, but also places and objects in a way a machine can understand (and compute). This being a side project, I cannot invest much time and have to progress slowly. Interactions of characters sounded pretty networky to me, so I looked around for some basic introduction to networks and how to do something meaningful with them computationally. (Another side-project of mine is teaching myself coding with Python – and some other languages, so I didn’t want to stop at using a fancy digital humanities tool to do the network analysis for me, black-box style. but rather test my hypotheses using math done by a machine.) The Google search suggested a book called “Complex Network Analysis in Python”, which sounded very much spot on and looking at the release date January 2018 I had the impression that this was a good first investment: It does network stuff, with Python and is fairly new! What could go wrong?

I must say there’s rarely been textbooks that I liked reading, let alone enjoyed reading and most I put aside pretty quickly and forgot that they existed. However, with a background in humanities and their tendency to verboseness, obscurity, and rhetorics, what was I to expect? Much unlike the horrible textbooks I knew from studying, science textbooks, especially computer science textbooks are awesome! They explain just enough to get the main problems across, but never too much to drown the content in endless strings of letters. They have plenty of examples that you can follow along and execute with an immediate check whether you succeeded or not and most are written in a witty, almost colloquial style. CNA in Python definitely falls into that category. It was easy to read even though the subject matter is tough and the examples looked easy enough to follow along coding in a Jupyter Notebook. I installed all the necessary python packages for doing complex network analysis, especially networkX, using the command line (smirkingly)

pip install networkx

and thought I was good to go.

Not so fast! I had not payed attention to the fact that CNA in Python uses networkX v1. Since the release of the book, networkX had been migrated to v2, including major changes that rendered the examples in the book practically undoable, if the latest version of the networkX package is installed (which happens when you thing you’re smart using the pip command without checking first…). I realised that this wouldn’t work out. But I really liked the book and wanted to follow along teaching myself CNA and get better at programming with Python, too. So I googled a little more and found the documentation for networkX v2 as well as a migration guide explaining the changes from v1 to v2 and ‘translated’ the examples (of the first 5 chapters) into networkX v2 code to make it run in my notebook. I was successful (in almost all examples, some stuff still is beyond my abilities) and could make it work!

However, I stopped reading – and writing out the examples and translating them – after chapter 5 because of lack of time and other things needing my attention. Last week though, I taught a study group networkX for CNA using my examples, the examples from the book and Gephi, building on the material I had created for the Innsbruck Easterplay (blog posts here and here). It went really well, I advertised the book to my study buddies even but made it clear it is hard to use with the new version of networkX, but still worth a recommendation. A study group friend even told me to send the ‘translations’ to the author for the next version of CNA with Python, but I declined, hinting at that there are surely better suited people working on this than me.

And here they are! I’ve already downloaded the upgraded version and am more eager than ever to go through the examples again and see how the author (who is a proper computer scientist!) has done it in v2 and finally proceed with the book beyond chapter 5 where the real fun begins: the actual analysis of complex networks (with math!!). I will assign myself the remainder of the year 2018 to re-do chapters 1–5 and proceed further, applying what I have learned thus far to my example, the character, places and object network of the 1391 Innsbruck Easterplay. And I will offer a follow-up session for the study group on using the python package networkX, too!

If you want to buy to the book, you can do so here: https://pragprog.com/book/dzcnapy/complex-network-analysis-in-python . I am not payed or remunerated by pragprog.com or the author or anyone else to recommend this book, but do it because it is my honest opinion that this textbook is useful, has good examples and is fun to read! Also for non-computer scientists, programmers, and math people 🙂

A Network Visualization of the 1391 Innsbruck Easter Play – Part II

After doing some exploratory diving into complex network analysis for finding out more about the character interactions in the Innsbruck Easter Play and not really being satisfied with the outcome (Part I), I have decided to continue working with this play and will be writing about it in a number of blog posts.

I’ve done some more work, mainly:

Created a .csv file for all ‘interactions’ of characters in the play.

By interactions I mean verbal and non-verbal communication directed to one or more characters (including the audience) as well as movement towards a stage location. I have even included the interaction of characters with objects of Christian symbolism (the shroud Jesus was buried in). Thus, the nodes in my graph can be characters (persons), stage locations (places), and objects (objects). The (…) serve as node attributes. The edges are the directed interactions between nodes. I have classified them into a number of ‘speech interactions’, ‘movement interactions’, and ‘physical interactions’. These serve as edge attributes. I have manually extracted these interactions from the stage directions of the play and in a few cases from the dialogue text. In quite a few cases, though, I had to leave a node blank because it wasn’t clear whom or what the interaction is directed at. I didn’t want to leave this information out so I decided to include it at this stage and create a special node “Ignotus” (unknown) instead. I will try to look deeper into the text to determine who or what is the target of the interaction in the unknown cases.

To get an impression of the interactions, lets take a look at an example from the text of the Innsbruck Easter Play (lines 147-166):

Nuntius recedit et vadit ad Pylatum et dicit
(The messenger steps back and goes to Pilate, and says:)

Edeler konnig Pylate,
dir enpiten vyer ritter drate,
daz sy wollen czu dir komen,
alz von dir haben vornamen.

Pylatus dicit
(Pilate says)

Ge hen vnd heiz sy czu mir komen,
ez wert er ere vnd ouch er frame:
ich muz sy czu noten haben,
wen da ist nymant by dem grabe.

Nuntius dicit ad Pylatum
(The messenger says to Pilate):

Pylate, lyber here myn,
laß dir dy wile nicht lang sin.

Nuntius exit et conducit milites ad Pylatum cantans Judaicum
(The messenger exits and leads the soldiers to Pilate, singing in Hebrew)

[Chodus chados adonay … amel!]

Angeli
(The angels)

‚Siletis’

And here’s how that looks in my spreadsheet, from row 57:

source label target
Nuntius EILEN Pilatus
Nuntius SPRECHEN Pilatus
Pilatus SPRECHEN Nuntius
Nuntius SPRECHEN Pilatus
Nuntius GEHEN Ignotus
Nuntius BRINGEN Primus Miles
Nuntius BRINGEN Secundus Miles
Nuntius BRINGEN Tertius Miles
Nuntius BRINGEN Quartus Miles
Primus Miles GEHEN Pilatus
Secundus Miles GEHEN Pilatus
Tertius Miles GEHEN Pilatus
Quartus Miles GEHEN Pilatus
Primus Miles SINGEN Ignotus
Secundus Miles SINGEN Ignotus
Tertius Miles SINGEN Ignotus
Quartus Miles SINGEN Ignotus
Primus Angelus RUFEN Spectantes
Secundus Angelus RUFEN Spectantes
Tertius Angelus RUFEN Spectantes

As you can see, I have decided to handle groups of characters as individuals. This is mainly because all groups (angels, soldiers, the Marys, and the Jews) also have their members acting individually with their own dialogue and interaction. One could argue that the group makes for an individual node apart from the individuals of the group making a node also, but for this part, I have chosen to list individuals only. You can also see the infamous “ignotus”-node – in the case above it is not entirely clear whom the soldiers sing their song to: it is definitely not directed towards either Pilate or the messenger, nor is it directed towards the audience. You could argue that I might as well skip it, because the action “singing” happens simultaneously with the action “walking”, making it rather a type of walking than two different actions. I haven’t decided on how to handle cases like this one (there’s several), but for this test, I recorded a maximum of interaction and characters.

After collecting the character-interaction-information from the text of the play in this edge list, I have created a graph visualization with the tool Gephi. The result looks like this:

Character network visualization of the Innsbruck Easter Play

The visualization looks quite different from Part I, mainly, because I have run some analytics on the graph (you can do all this with Gephi, it’s quite powerful!) and used directed, parallel edges. What the visualization shows is ‘amount of interaction’ between nodes with the thicker the edge line, the more interaction. And it shows the ‘connectedness’ of characters by node size: the more (different) connections a character has to another, the bigger the node.

However, if you pay close attention to the graph, you will see that I have not separated types of nodes or types of edges: the graph doesn’t differentiate between persons or places, nor does it take the types of interactions into account (I’ve played around with the settings for this but apparently didn’t take a picture – so, “it didn’t happen!”). The other issue is the big “Ignotus” node, that could be anything from an individual person, a group of characters or a place. I don’t like this, but at the moment, there’s not much I can do about it, since it involves a couple of hours of manual text analysis to decide where interactions are directed towards. I will leave it at this. For the next part, I will hopefully have dealt with the issue and gotten the data cleaned so I can finally embark on the analysis part of the character network of the Innsbruck Easter Play.

You can do your own analytics with my data – it’s all (in a bit of a mess) in my GitHub repository InnsbruckEasterplay.

Stay tuned!

A Network Visualization of the 1391 Innsbruck Easter Play – Part I

tl;dr

I found a character network visualization tool for simplified analysis online, https://ezlinavis.dracor.org/, tried it out with data from a medieval religious German play from 1391 and replicated the visualization and analysis in Gephi and deliberated about some constraining factors for future analyses.


I’ve been playing around with https://ezlinavis.dracor.org/ – an easy visualization tool for networks in plays, developed by Frank Fischer and Carsten Milling. It takes a simplified input of roles in a play, with their names by scene and act and then visualizes the interaction by drawing a line between the roles that are active in the scene. So, if there is Faust on the stage as well as Mephistopheles, but no one else, then both characters are nodes and an undirected edge is drawn between them. We got, however, no indication of the quality of their interaction. It might well be that one character has no dialogue or serves as background staffage or has significantly less dialogue than the other. Their ‘presence’ is thus simply them being on the stage at the same time during a scene.

Now, I have been working with medieval religious German plays and one of my favorite plays is the so called Innsbruck or Neustift Easter Play, which has survived in a well used, vertically folded folio manuscript dated 1391. The play is one of the better know German religious plays, mainly due to it featuring an extended burlesque scene around the ‘purchase of ointment’ by the three Marys on their way to Jesus’ grave. It also features an extensive satire of the three estates (‘Ständesatire’) accompanied by a devils’ play (‘Teufelspiel’).

My idea was to try and see how the character network of a medieval religious play might look like – potentially opening for a collective visualization of all a large enough number of plays – something like the Dramaviz poster by Fischer et al. that features >400 German dramatic plays from the modern times. And think about the potential of this as a comparative study where modern German theater (meaning: plays that were created in the classical tradition roughly from 16th century onward) is compared with medieval plays (that were, according to literary scholars, of an entirely different nature).

The first problem I faced was with the EasyLineVis tool’s basic assumption about how a play is structured: you have a number of acts (in classical drama it’s usually 5) and scenes within acts where characters enter (and leave) the stage. These are clearly marked, so it’s relatively easy to extract this kind of information. However, in medieval religious plays, there is no such thing as the common classical/modern notion of acts and scenes. This has – among other things – to do with the staging of these plays on the simultaneous stage (‘Simultanbühne’), where the characters are present ‘on the stage’ at the same time and coming forward and interacting with each other when it’s their turn. So, a simple ‘enters the stage’ or ‘leaves the stage’ cannot be found in the stage directions of the plays. A common method to structure the plays (for analysis or philological treatment) is thus to determine scenes according to the narrative of the salvation history that forms the plot basis for these plays. Alternatively, one could determine scenes by finding indicators for a ‘shift of focus’, that is, when the plot shifts to a different set of characters (e.g. the Jews, Pilate and the Soldiers, the Soldiers and the Angles; the three Marys etc.). Or – and this would be more according to the physical features of the simultaneous stage, they could be determined in accordance with the concrete stage locations, the so-called huts (‘Häuser’). It is, however, often not clearly stated in the stage directions or elsewhere in the plays’ text and paratext, what kind of  huts are on the stage and which characters belong to which hut.

For a first try out, however, I’ve attempted a separation of the Innsbruck Easter Play into scenes using Wikipedia’s suggestion of thematic units, based on Johan Nowé’s “Wir wellen haben ein spil: Zur Geschichte des Dramas im deutschen Mittelalter” (1997), p. 36, and this is what I came up with:

  1. prologue
  2. ordering of the guards
  3. resurrection
  4. decent into hell
  5. plaint of the Marys
  6. peddler’s play
  7. the grave and the apostles’ run
  8. epilogue

So, we have 8 scenes – let’s look at the graph for this!

How to do that:

(1) download the file “InnsOS_characters_scene.txt”  from my GitHub repository,

(2) copy-and-paste its content into the left blank pane of the ezlinavis tool.

(3) the tool will automatically create a network graph in the right pane and provide a .csv-file of your input in the middle pane

(4) which you can download if you want to do your own visualization, with, let’s say, Gephi.

(5) click on ‘Graph’ in the right upper corner and chose ‘ForceAtlas2’ as a layout for your graph.

ezlinavis doesn’t let you download the graph as an image file – it’s an educational tool and serves a different function – so I took the .csv-file and uploaded it into Gephi. After some tweaking of parameters (e.g. visible labels for the nodes and a change in scale) I applied the same layout that ezlinavis uses as default (ForceAtlas2) – et voilà! A first character network visualization of the Innsbruck Easter Play looks like this

Character Network Visualization for the Innsbruck Easter Play (1391) Using 8 Thematically Determined Scenes

I’m not unhappy with the outcome – you can clearly see which characters have the most connections (the Angels and Jesus) and the peddler’s play stands out as separate pretty visibly. Interestingly, the devils’s play – which I am tempted to call a ‘cluster of devils’ (as in: a murder of crows) – is also neatly defined.

However – and this is a big however – this visualization has at least two big flaws which make it unusable outside a mere context of discovery. I.e. you cannot prove or disprove anything with this.

  1. The connections (edges) between the characters are ‘undirected’ – you do not see who talks to whom – and the basic data model is relatively simple: it doesn’t account for actual interaction, nor does it take the characters’ performative actions, movements, or (length of) dialogue, or time spent in a scene into account. This can, however, be helped with modifying the data model and adding the info to the csv-input.
  2. The division of the play into 8 scenes is in my opinion rather superficial. Nowé does determine the scenes thematically and in accordance with salvation history. And not based on the text of the play and the division indications that can be derived from the stage directions and the dialogue.

I will take it from here and fix these issues, starting with the division into scenes. I will then run the same data (who’s on the stage at the same time) with different scene divisors again and see what happens. As a second step, I will account for direction of action and possibly length of dialogue and/or time spent in a scene (likely to be derived by analyzing dialogue length) as well as movement (characters moving from one stage location on to another). But this will have to wait!

So far, I can recommend trying out the ezlinavis tool for simple exploration of character networks in performative genres such as plays – it looks like it could be especially useful in a teaching context! But for more in depth and deliberate analysis of character networks, a tool like Gephi – and more knowledge of the underlying assumptions of modeling networks – will be the most sensible way to proceed!

 

Digital Textology

UPDATE: It’s July 2018 and I am a bit embarrassed – I started this blog more than four years ago and never wrote more than the opening post… So much for bi-weekly updates! This time will be different, I promise!

———————————————-

This blog deals bi-weekly with contemporary humanities research from digital editing to the scientific study of literature and other cultural phenomena. It is meant primarily as a ways of discussing material – articles, monographs, scholarly editions, websites, tools – in the light of critical rationalism. The blog is a means for its author to engage with a mode of doing humanities research and especially literary (or text) studies that recognises the potential of “the digital” for an empirical or scientific approach to literary studies. It is also an opportunity to collect and discuss new material from literary and linguistic studies in the scope of a future research project.