Volodymyr Miz

Wikipedia, COVID-19, and readers' interests across languages

Wed, 05 Aug 2020 13:13:13 +0200

To tell you the truth, I’d better be writing my PhD thesis right now. But the recent findings got me so excited that I cannot wait to share them. Anyways, the thesis can wait; hello ~~darkness~~ procrastination, my old friend.

If you are only interested in the results, feel free to skip to 3. Results.

1. Introduction

A few months before I started writing my thesis, the COVID-19 pandemic had unfolded. At the beginning of the pandemic, strict confinement measures were introduced, first, in China, then in several other countries in Asia, and finally, across Europe and in other countries around the world. These measures globally affected the world, leading to drastic changes in mobility patterns, among many others. Following these restrictions, the interests of Wikipedia readers have also changed (you can read more about it in a recent study by Manoel Horta Ribeiro et al.).

All in all, the striking influence of the pandemic on everyday life inspired us to run another experiment. We decided to investigate the evolution of trending topics throughout the pandemic across different language editions of Wikipedia. We focused on the data over the period from December 2019 until May 2020. The study involves 7 language editions of Wikipedia, including English, French, Russian, Italian, Spanish, German, and Chinese (note that Wikipedia has been banned in China since 23 April 2019).

2. Method

In the previous post, I wrote about an algorithm for detecting and comparing trending topics across different Wikipedia language editions. The trend detection approach we use here is almost the same as in our previous study. The core difference is the classification engine. This time, we didn’t train it. Instead, we adopted a model designed and trained by Wikimedia Foundation researchers, Isaac Johnson, Martin Gerlach, and Diego Saez-Trumper. This model is more versatile than the one we used before because it makes predictions for Wikidata items instead of Wikipedia articles, which makes its application in multilingual settings easier. Read more about their classification model here.

Another difference is the way we label trends. In the previous study, we labeled an entire cluster with the most popular topic, computing popularity of the topic using a complex formula that involves graph-based attributes. This time, we decided to keep it simple. Each cluster has multiple topic labels that give a more fine-grained outlook on each trend. As we can see in the figure below, when we use the Wikidata-based approach for topic classification, we obtain heterogeneous clusters. Left: general trends with mixed topic labels; right: the same general trends decomposed into sub-clusters of articles on the same topic.

Hence, each cluster corresponds to one trend, which covers multiple topics. These topics are implicitly related to trending events that triggered the creation of each cluster. To understand better, let us look at several examples in more detail. The majority of pages in Sports-related clusters (highlighted in blue) are labeled as Sports. However, depending on the nature of sports, we can also see smaller sub-clusters of pages related to other topics, such as Media, Education, Healthcare, and Engineering inside of the main cluster. Needless to say that media and education are inalienable parts of many popular sports these days.

We can also see that trends related to politics (highlighted in red) are also very diverse. If we look at the cluster that emerged after the death of John McCain, we can see that the majority of pages, indeed, belong to politics. What is interesting is that we can also see fairly big sub-clusters that comprise articles on topics such as Society, Military and Warfare, History, and Business. All these areas are inevitably involved in political careers and processes.

Clusters created as a result of the readers’ interest in natural disasters (yellow) are also heterogeneous. In the cluster that emerged after the Hurricane Florence, we can see that it is comprised of pages that cover a wide range of topics including STEM (mostly articles on Earth & Environment), Politics (articles about politicians involved into solving the crisis), and History (articles related to previous disasters and their causalities).

Finally, art-related topics (green) are also diverse. We can see that the cluster that emerged due to the anniversary of Aretha Franklin’s death includes articles on different topics associated with various artistries, such as Media, Music, and Visual Arts.

To get a better view of the heterogeneity of the clusters classified using the Wikidata-based approach, let’s look at another plot, which illustrates more concrete examples.

We can see that clusters related to sports, such as NCAA (American football), US Open (Tennis), and Belgian Grand Prix (Auto racing, Formula~1), have the least diverse set of topics among others. Naturally, most of the pages are classified as Sports. Media, Business, and Politics are also among common topics in the sports-related clusters. However, it is interesting to see that secondary topics in these clusters reflect specific features of different sports. Engineering advancements in the automotive industry play a significant role in Formula~1, which is reflected by the presence of topics Transportation and Engineering in its cluster. NCAA is an organization that regulates student athletes from North American institutions, so we can see Education among the most represented secondary topics in that cluster. We can also notice a similar effect in the clusters related to politics and show business where secondary topics give us a wider perspective on the specific nature of the occurred events.

3. Results

3.1 COVID-19 and trends

Now, let’s focus on general trends over the first months of the pandemic. On the plot below, we can see that the distribution of trending topics looks similar to what we saw in the previous experiments.

Sports and Media are the leading trends, followed by Music, Films, and Politics & Government. We also noticed an emergence of two new clusters that were not captured before, namely Medicine & Health, and Biology, which was triggered by the increased interest of the readers in the articles related to viruses, influenza, and the COVID-19 pandemic itself.

The popularity of Society is 20-25% higher in Chinese and Russian editions compared to other languages that we analyzed. After a qualitative analysis of the classification results, we have discovered that there is a significant overlap between the topics Politics and Society in Chinese and Russian editions. We found that Wikidata items, related to local political figures and elections in the regions where the majority of people speak Russian and Chinese, often belong to the topic Society. This finding is thought-provoking and requires more research.

3.2 Evolution of trends during COVID-19

The evolution of trends during the pandemic is even more interesting. To capture the dynamics, we aggregated trending topics bi-weekly; each data point represents the popularity of a topic during a selected two-weeks period. In this study, we focus on the short-term dynamics, which reflects change points in the trends in a moving time-window. This allows us to get a live picture of how users shifted their attention from one topic to another. In the stacked chart below, you can see a dynamic view of the changing popularity of some of the most popular topics. We normalized the popularity of each topic in each language between 0 and 1. The more drastic the attention shift, the thicker the line on the plot.

We can see that the COVID-19-related topics, such as Biology and Medicine & Health, have an attention spike in January. Then, after a short-term drop, these topics develop a steady momentum starting from February. In Chinese Wikipedia, we observe the most significant increase in attention to these topics in January. The interest in the topics remains consistent throughout the entire period and only slightly diminishes in April. Looking at other language editions, we observe that at the beginning of February, the interest of the readers to Biology and Medicine & Health drops. However, soon after that, the topic Biology regains popularity among Italian-speaking readers, followed by English- and German-speaking audience. Attention to Medicine & Health also bounces back, first, in Italian and French editions, and then in German and English. Russian-speaking readers develop an interest in both topics closer to the end of March. All in all, most of these observations reflect the COVID-19 development timeline in the locations where these languages are spoken primarily, however, it is still hard to align geographically the results for English, French, and Spanish language editions because of their global adoption in different regions of the world.

Sports is the most popular topic in all languages at the beginning of the pandemic. However, we can notice an abrupt change of attention levels across all languages. The readers become indifferent to this topic starting from March. One of the possible explanations is that the pandemic resulted in the cancellation of the majority of sports events around the world.

Attention to Media, Films, and Music is mostly uniform across all languages during the pandemic. The spike in the topic Films can be explained by the worldwide popularity 92nd Academy Awards ceremony, which took place on February 9. When we look at Music, we can see slight shifts towards indifference among Italian-speaking readers, which happens in the second half of February. This can be attributed to the strict lock-down measures that were introduced on March 9, however, this is just an observational hypothesis.

Finally, let’s compare our results to the ones reported in the study by Manoel Horta Ribeiro et al. The main difference between the two approaches is the comparison strategy. In our approach, we focused on the live short-term attention shifts or change-points, while the authors of the study compared attention levels to the previous year, reporting long-term changes. Even though we used a different approach, it is exciting to see that our findings confirm some of the observations reported in that study. During the first months of the pandemic, Wikipedia articles on Biology and Medicine & Health gained a lot of interest from readers across all language editions, while Sports-related articles lost a significant share of their audience. Nonetheless, there are a few discrepancies. For instance, we did not notice similar short-term changes in the attention to the topics Media, Films, and Music. We found that these topics retained the same level of short-term attention throughout the studied period.

4. Conclusions

The COVID-19 pandemic has affected pretty much every aspect of everyday life, including collective behavior and interests of digital users worldwide. This global crisis highlighted the utter importance of Wikipedia. We can see that people reached out to the encyclopedia looking for answers to pandemic-related questions. It is also exciting to see that, despite the ban, Chinese Wikipedia is still very active.

The difference between long- and short-term changes is interesting too. Attention to Biology, Medicine & Health, and Sports changed the most in the long- and short-term. Other topics, mostly related to entertainment, such as Media, Films, and Music, gained more interest compared to the previous year. However, throughout the pandemic, the attention level to these topics did not fluctuate.

Also, we noticed that trends in Wikipedia’s viewership are more complex than we previously thought. Each trend that we detected contains a set of articles on various topics. This gives an interesting perspective on the collective perception of trends across languages. We can see which topics are associated to which trends in every language edition.

From a technical point of view, the new topic classification model gives really promising results and inspires us to develop the project further. Now, the master plan is to ~~finish my thesis~~ scale the study to even more languages, make it more automated, and create web-based service reporting trends in different languages editions of Wikipedia.

5. Code, data, results

Everything related to the COVID-19 case study is on GitHub. If you are interested in the trend detection algorithm, take a look at another blog post explaining it in more detail. If you want to run similar experiments, take a look at the distributed graph-based framework for Wikipedia data processing, which we use in all experiments.

Interactive visualizations (similar to this one) are coming soon. Stay tuned!

6. Acknowledgments

This project would not have happened without a great effort by Etienne Chalot, who did a summer internship (and the actual work on the project) in LTS2. I would like to thank Isaac Johnson for sharing the topic classification model and for providing advice on how to deploy it locally. Also, kudos to Nicolas Aspert and Benjamin Ricaud for coming up with interesting ideas that shaped this project. Special thanks to Nicolas for helping us with technical issues and taming our greed for computing power. Lastly, I hope my advisor, Pierre Vandergheynst, is not reading this, but just in case he is, I am grateful to him for letting me spend the entire morning not writing my thesis and writing this blog post instead :D

7. Questions?

If you have any questions or suggestions, leave a comment below or contact me directly.

What's trending on Wikipedia?

Sun, 07 Jun 2020 13:13:13 +0200

There are as many opinions as there are people, they say. After a recent study we have done in LTS2, I can paraphrase it. There are as many opinions as there are languages on Wikipedia. In our case though it was more about people’s interests rather than opinions.

In a recent study, we found that interests of Wikipedia readers largely depend on the language in which they read Wikipedia. We compared English, French, and Russian language editions and found that certain topics are more popular in one language than in the others. For instance, topic Sports dominates among users reading Wikipedia in English, Francophone readers prefer Wikipedia articles about Movies, and Russian-speaking audience is obsessed with Wikipedia articles related to Science. Also, we found that topics related to pop-culture (e.g. release of a movie by Marvell) and global dramatic events (September 11) are popular across all languages that we have studied.

So, what’s trending on Wikipedia? It depends. Specifically, it depends on the language you are interested in. If you want to learn more, watch our presentation at Wikipedia Workshop 2020 and read a review of our study by Isaac Johnson in Wikipedia Research Newsletter. If you are interested in the technical details, our method, and the dataset, take a look at the paper.

I worked on this project with Joëlle Hanna, Nicolas Aspert, Benjamin Ricaud, and Pierre Vandergheynst. You can find this and many other Wikipedia-related projects and resources by our lab on Wikipedia Insights project website, for instance, an anomaly detection algorithm and a graph-based dataset with pre-processing tools that we used to detect trending topics on Wikipeida.

If you are interested in the graph visualization tools we used in this project, take a look at the following tutorials:

Stay tuned for more!

Gephi tutorial. Publishing interactive graphs online

Sun, 05 Jan 2020 13:13:13 +0100

Ever wondered how to publish interactive graph visualizations online? This tutorial is for you then, the second one in the series of Gephi tutorials. If you want to learn more about layouts and attributes in Gephi, check out the first tutorial.

As a result of this tutorial, you will create an interactive visualization that will look similar to this one.

In this tutorial we will learn:

How to prepare a graph layout before publishing it online.
How to export SigmaJS template and customize it.
How to publish your interactive graph visualization online using GitHub pages.

If you don’t feel like reading the instructions, watch a short step-by-step walk-through screencast to get an idea of what we are going to do. The video reproduces all the steps listed below.

1. Install plugins

Install all necessary plugins before starting this tutorial (unless they are already installed in your verstion of Gephi):

Multigravity Force Atlas 2
Circle Pack
Label Adjust
SigmaJS exporter.

2. Import graph

Open GEXF file in Gephi (File->Open...). In this example, we are going to work with a subsample of English Wikipedia. The graph contains pages that had anomalous visitor activity over the period 16-31 August 2018. You can find corresponding .gexf file in this folder.

3. Compute layout

Let’s start with Multigravity Force Atlas 2. Take a look at this tutorial for inspiration if you want to use another layout.

Parameters:

Scaling: 10
Edge Weight Influence: 0
Dissuade hubs: True

4. Export SigmaJS template

Export SigmaJS template File->Export->SigmaJS template and fill in all required fields.

5. Test locally

Go to the exported folder and start a simple Python HTTP server to test your visualization. Depending on Python version, in the terminal, type the following command.

Python 3.X python -m http.server
Python 2.7 python -m SimpleHTTPServer

Now, you can interact with your graph using your web-browser (works best in Chrome). Go to http://localhost:8000/ and play with it.

That is all we need to do to publish our visualization online. Although, as we can see, the graph looks quite raw and it is hard to interact with it. Let’s compute more attributes and make the graph look nicer and more user-friendly.

6. Compute attributes.

We are going to use attributed layouts to enhance readability. Before doing that, we need to compute attributes.

Modularity (Use weights: False)
Average Degree

7. Color nodes according to their community

Color nodes according to their modularity class and make their size correspond to their degree.

8. Circle Pack layout

Use Circle Pack layout to rearrange nodes according to attributes. Use modularity and degree as parameters.

9. Scale and labels

Adjust the scale and labels. Use Expansion layout to increase the scale of the layout. Display labels, reduce the font size and use Label Adjust layout to prevent overlapping node labels.

10. Export SigmaJS template once again and test it locally

Our graph looks more readable now and it’s much easier to interact with it. Export SigmaJS template once again and check it on localhost (steps 4 and 5).

11. Publish your graph on GitHub pages.

Now, we can publish everything to GitHub pages.

Create a GitHub repository.
Go to the settings of your repository and find GitHub pages section. Specify master branch as a source.
Clone the repository.
Copy the exported SigmaJS template that we have prepared to the cloned folder.
Push the files to the repository.
Check the website with your interactive visualization https://[YOUR-GITHUB-USER-NAME].github.io/[VIS-REPOSITORY-NAME]/network/
Et voila. You have just published your interactive graph visualization online.

12. Optional. Play with the config file

You can adjust properties of the visualization using the config.json file that you can find in the folder with our SigmaJS template. Play with SigmaJS config file:

Adjust node sizes using minNodeSize and maxNodeSize.
Adjust label thresholds using labelThreshold.

13. Optional. Customize HTML/CSS

If you are familiar with HTML/CSS, you can customize the style of the web page. All HTML and CSS files are in the generated foloder. With just a few adjustments, you could create something like this. I have customized this template for one of our projects (see more details and examples here)

Gephi tutorial. Layouts

Sun, 05 Jan 2020 12:13:13 +0100

Colleagues and students often ask me whether it is easy to create a nice graph visualization. My answer is always positive but such conversations often end up where they started for multiple reasons. First, it is hard to understand what people want to do with their visualization. Second, it is hard to summarize everything in one 3-5 minutes step by step explanation. Finally, even if we have managed to resolve first two issues, I still feel like my verbal explanations sound obscure so they feel discouraged and don’t even try. Although, after I show them how to do it, things get much easier. As we all know, it is “better to see something once than to hear about it a hundred times”.

I have decided to create a series of tutorials on Gephi graph visualization. Each tutorial has a short ~3min long walk-through videos. These videos demonstrate that creating interpretable, interactive and nice looking graph data visualizations can be fast and easy. Moreover, it doesn’t require any coding skills, so it can be used by people who just need to get quick insights from their graph-structured data.

I also gave these tutorials after my lecture on data visualization for A Network Tour of Data Science course at EPFL.

This tutorial is about some of the most popular layouts in Gephi. In the next one, I’m going to show how to make your graph visualization interactive and publish it online in a few clicks.

Layouts. Graph attributes. Data exploration

In this tutorial, we will try the most popular tools for graph exploration in Gephi. We will learn how to get visual insights by spatializing and highlighting important attributes of graphs. As an example, we will use a subnetwork of Wikipedia pages that had anomalous visitor activity over the period 15-31 October 2018. You can find the corresponding .gexf in this repository.

Before starting this tutorial, install the following plugins (unless they are already installed in your version of Gephi):

Multigravity Force Atlas 2
Circle Pack layout
Leiden algorithm
Bridging centrality
Clustering coefficient

As promised, here is a short step-by-step walk-through screencast to give you an idea of what we are going to do in this tutorial. Also, if you don’t feel like reading the tutorial, just follow the steps in this video. The video reproduces all the steps listed below.

1. Spatialize your graph with force-directed layouts.

To start with, let’s spatialize our graph. To do that, we can use force-directed layouts. They don’t require any attributes and quite easy to setup. We will try four layouts: Multigravity Force Atlas 2, Yuifan Hu, Fruchterman-Reingold, Open Ord. You can find all these layouts in the “Layout” pane. Here are some tips related to the parameters of each algorithm:

Multigravity Force Atlas 2
- Scaling. Control scale of the expansion of the graph.
- Dissuade hubs. Apply stronger repulsive forces to hubs.
- Prevent overlap. Prevent nodes from overlapping.
Yifan Hu
- Step ratio. High ratio improves quality (at the expense of speed)
- Optimal distance. Controls distance between nodes
- Theta. Smaller Theta gives leads to more accurate results (slower)
Fruchterman-Reingold (expensive)
- Gravity. Attraction strength.
- Speed. A tradeoff between speed and accuracy. Higher values lead to faster but less accurate results
Open Ord
- Edge Cut (0 to 1). Higher values lead to more clustered results.
- Num Iterations. Higher values lead to larger expansion.

In the following examples, colors correspond to modularity.

Multigravity Force Atlas 2	Yifan Hu	Fruchterman-Reingold	Open Ord

2. Highlight attributes

With Gephi, we can highlight attributes of the nodes using different colors and sizes, reflecting the scale of the attributes. You can do it using “Appearance” pane. Before using the attributes, we should compute them using “Statistics” pane. Some examples of attributes that can be used for sizing and coloring are listed below.

Node and label size

We can use the size to highlight local attributes of nodes.

Degree. Connectivity of a node
Centrality
- Betweenness. Number of random walks passing through the node
- Bridging. A measure of bi-partisanship of a node. Nodes that connect multiple communities (serve as bridges) have high bridging centrality
Page rank. Importance of a node
Local clustering coefficient. Determines how close are neighbors of a node to a complete graph

Color

Coloring different communities of the graph is a allows to explore the structure of your graph and make it more visually appealing. To do that, we need to run a community detection algorithm. Once you compute communities, you can color the graph using “Appearance” pane.

Community detection algorithms
- Louvain modularity (Modularity in Gephi)
- Leiden algorithm

3. Use attributed layouts

Gephi also provides a set of attributed layouts. You can use them to spatialize nodes according to their attributes. Here are a few examples of attributed layouts:

Circular. We can use it to show the distribution of nodes and their links and order nodes by an attribute and draw them on a circle.
Radial Axis. This layout allows studying homophily by showing distributions of nodes inside groups. Axes radiate from the central circle. The layout groups nodes by an attribute and draw them in axes.
Network splitter 3D. Use this if you want to unfold the graph in layers based on attributes. However, there is a trick. To use it, you need to copy a column with an attribute of interest and add [Z] in the end of the name of the column. Z-maximum level parameter should be equal to the number of layers you want to get, e.g. number of communities if you use communities as a splitting attribute.
Circle pack. This layout allows to group nodes by attribute(s) and plotting them in circles.

In the examples below, we use modularity (community ID) as an attribute.

Circular	Radial Axis	Network splitter 3D	Circle pack

4. Enhance readability with spatial transformation layouts

Normally, after all the transformations we have performed, it’s hard to read labels. To enhance readability and interpretability of your graph, use the following layouts.

Expansion/Contraction. This layout simply changes the scale of the graph.
Noverlap. Use it to prevent overlapping nodes.
Label adjust. This layout spatializes labels so that they are easier to read. Before running it, you should display labels.

5. Additional resources

If you want to learn how to publish Gephi graphs online, take a look at this tutorial. For more information, check out official Gephi tutorials:

Gephi tutorial on layouts.
Other Gephi tutorials.

Wikipedia Graph Dataset

Wed, 05 Jun 2019 13:13:13 +0200

Needless to say that Wikipedia is an invaluable source of free knowledge. In addition to that, Wikipedia weblogs data is a great resource for the research in many different fields such as collective behavior, data mining or network science. After all the projects and hours spent on wikipedia data, we got really tired of the data pre-processing. Even though Wikipedia data dumps are quite organized and clean, it takes time to pre-process data to represent and handle it as a graph data structure. Wikimedia Foundation has already done a great job making this data quite well structured and publicly available (consider donating). We wanted to make this data even more accessible to researchers and tech-savvy Wikipedia enthusiasts interested in studying network-related and socio-dynamic aspects of this knowledge base.

We are focusing on two aspects of the data dumps, Wikipedia web network and pagecounts (number of visits per Wiki page per hour). While working on multiple Wikipedia-related projects, we realized that the life of researchers interested in getting insights from Wikipedia web network structure would have been much easier if the data was natively represented as a graph. Also, we thought that it would be nicer and more convenient to have pagecounts stored in a database. This would allow us to ask questions not only about the static network structure but also about the dynamic aspects of Wikipedia, such as visitors’ interests over time, anomalies in viewership activity, or any other questions related to collective behavior.

All in all, we provide tools to pre-process two types of Wikipedia dumps, page graph (pages as nodes, hyperlinks as edges) and pagecounts. We store the graph in Neo4J graph database. To use the graph dataset you need to deploy Neo4J database on your local computer or a remote server (preferably Debian-based distribution). Pagecounts are stored in .parquet so you can use the files with any compatible framework. You can use the datasets together as well as independently. If you decide to use the datasets together, they can be connected using a key PAGE_ID (as shown in the igure). Take a look at the detailed deployment instructions at the end of this post.

To incentivize you to start using these databases, we will show a few use cases and queries that should give you an idea of what you potentially could do with them in your Wikipedia-related research projects.

Queries and use cases

Pretty much everyone is crazy about Game of Thrones these days so let’s use it as a demonstration example just to show courtesy to pop culture trends. Let’s take a look at the subnetwork of GoT characters and the activity of Wikipedia visitors over time. To do that, we will need to perform the following steps.

1. Graph database. Neo4J.

We will start with a simple Cypher query. To which Wikipedia categories does Daenerys_Targaryen page belong to?

MATCH (page:Page {title: "Daenerys_Targaryen"})-[:BELONGS_TO]->(category:Category) RETURN category.title

This query returns 15 categories to which Daenerys Targaryen BELOGNS_TO. For example, this list includes Fictional_princesses, Fictional_victims_of_child_abuse, and Female_characters_in_television. Taking this into account, we may assume that all GoT characters belong to two large categories, Female_characters_in_television and Male_characters_in_television. Let’s keep this in mind until we write the final query.

Now, we are interested in a sub-network around category GoT. To get this subnetwork we need to submit the following query:

MATCH (page)-[relationship*1..2]->(category:Category { title: 'Game_of_Thrones'}) RETURN page,relationship,category

Technical note: in this query, we set depth of the neighborhood to 2, which means we want a sub-network that includes nodes not further than 2 hops from the GoT category page.

The response is too large to visualize in Neo4j browser. I would recommend either storing the resulting graph in a file and then visualizing it elsewhere or limiting the size of the response with LIMIT keyword at the end of the query. For example, we can save the response in GraphML format and use Gephi to visualize it. To do that, we need to use APOC procedures and the query would look as follows.

CALL apoc.export.graphml.query("MATCH (page)-[relationship*1..2]->(category:Category { title: 'Game_of_Thrones'}) RETURN page,relationship,category", '/tmp/got-graph.graphml', {useTypes:true, storeNodeIds:false})

Once you get the file and apply some Gephi-fu, you will get a graph similar to the one shown on the image below. To take a closer look on this graph, open this project file in Gephi. You can also click on the image and explore the network in an interactive environment.

In this graph, you can find Wikipedia pages related to GoT such as actors, characters, seasons, episodes, and other related pages. Note that Wikipedia structure can be confusing due to two types of Wikipedia articles: pages and categories. You should take this into account when working with the dataset. For example, in this graph, you will find both types of nodes. Do not get confused when you find two nodes with title Game_of_Thrones. One of them (green) is a category, another one (pink) is a page.

Finally, to extract a subnetwork of GoT characters, we need all Wikipedia pages that BELONGS_TO categories Game_of_Thrones, Female_characters_in_television and Male_characters_in_television. To do this trick, we need a bit more sophisticated yet very concise and self-descriptive query. Combining the two previous queries, we get something like this:

MATCH (c1:Category)<-[:BELONGS_TO]-(p)-[r*1..2]->(c2:Category {title:"Game_of_Thrones"}) WHERE c1.title = "Female_characters_in_television" OR c1.title = "Male_characters_in_television" RETURN DISTINCT p.id, p.title;

This query returns characters and corresponding page IDs. If we wanted to get the full network of characters with edges between them, we would have to extend the query above in the following way:

MATCH (c1:Category)<-[:BELONGS_TO]-(p)-[r*1..2]->(c2:Category {title:'Game_of_Thrones'}) WHERE c1.title = 'Female_characters_in_television' OR c1.title = 'Male_characters_in_television' WITH COLLECT(distinct p) AS pages UNWIND pages AS p1 MATCH (p1)-[rel]->(p2) WHERE p2 IN pages RETURN pages, rel;

It is a bit longer than the previous one. We have added the part that fetches edges between all pages that we have extracted in the previous query. Click on the image below to interact with the network.

Once we have extracted the subnetwork of GoT characters, we can look at the popularity of the characters on Wikipedia according to visitor interests. To do that, we need to query another part of our dataset that is stored in .parquet files.

2. Pagecouts dataset

In this section, we will take a look at the viewership activity on a couple of pages.

To take a look at the popularity of GoT characters over time on Wikipedia, we need IDs of the pages we are interested in. We can extract the IDs from the previous query (we can export them in a CSV file) and use them to query tables stored in parquet files.

In the image below, you can see a partial result of such query. We can notice the day and night fluctuations in the time-series. Here, we display the activity of only two pages to avoid the clutter.

There are many ways to use activity information. For example, take a look at one of our latest projects on Wikipedia viewership anomaly detection.

More details and the team

If you are interested in other languages (apart from English), our pre-processing tools allow you to work with dumps of any language edition of Wikipedia. More details in the deployment instructions below. GitHub repo with the deployment instructions is available here. A step-by-step deployment tutorial is available here.

If you have any problems related to deployment or usage of the dataset, we encourage you to create an issue on GitHub. This is the most efficient way of communication. You can also leave a comment below or contact me directly.

This dataset is a joint effort of Nicolas Aspert, Volodymyr Miz, Benjamin Ricaud, and Pierre Vandergheynst (EPFL, LTS2). For more details, take a look at the paper we have recently published in the proceedings of the Wiki Workshop 2019 held at The Web Conference 2019.

Acknowledgements

Kudos to Kirell Benzi for the idea of the GoT case study.

How to install Apache Spark on Ubuntu using Apache Bigtop

Thu, 04 Apr 2019 17:07:13 +0200

Has this ever happened to you? A new version of Spark is coming out and you want to try it out. To do that, you have to remove the previous version, download and extract a new one and hope that everything still works. Or, sometimes, you are just getting started with a Spark project and want the installation process to be seamless and easy. A one-liner command would be nice, wouldn’t it? Making this process more organized and user-friendly is one of the goals of Apache Bigtop. This post is for ML and infrastructure engineers, data scientists, and those who are just willing to try Spark out.

Apache Bigtop is aimed at providing ML engineers, infrastructure engineers, and data scientists with a convenient tool for packaging, deployment, and integration of Hadoop-related projects such as HDFS, MapReduce, Pig, Hive, HBase, ZooKeeper, Spark, and many others.

In this tutorial, I will show how to install Apache Bigtop and how to use it to install Apache Spark. Here, I will focus on Ubuntu. For other distributions, check out this link.

Bigtop installation

This tutorial is for Bigtop version 1.3.0. If you want to isntall other versions, change the version in the commands below accordingly.

Make sure that you have the latest JDK installed on your system (so far, JDK 8 works well).
Install the Apache Bigtop GPG key.

wget -O- http://archive.apache.org/dist/bigtop/bigtop-1.3.0/repos/GPG-KEY-bigtop | sudo apt-key add -

Make sure to grab the repo file.

sudo wget -O /etc/apt/sources.list.d/bigtop-1.3.0.list http://archive.apache.org/dist/bigtop/bigtop-1.3.0/repos/ubuntu16.04/bigtop.list

Update the apt cache.

sudo apt-get update

Browse through the artifacts.

apt-cache search mahout

Install bigtop-utils.

sudo apt-get install bigtop-utils

Now you can install Spark and other Hadoop-related projects.

Spark installation

Install Spark.

sudo apt-get install spark\*

Take a look at the Wiki of the Bigtop project for more information concerning other Hadoop-related projects.

If you are looking for an easier way to try out Spark, check out another tutorial on how to create Spark Scala project in Intellij IDEA. This way you do not have to install anything except Intellij IDEA.

Anomaly detection in the dynamics of Web and social networks

Wed, 13 Feb 2019 12:13:13 +0100

Imagine, you have a large network. Say, a social one. Every day, users of this network (aka nodes) generate massive amounts of likes, messages, and clicks (aka activity logs). Also, these users are connected through some links (aka edges), say, follow or befriend each other. There is nothing special about daily activity in this network. But, at some point, something unexpected happens in the network dynamics (e.g. an attack). Since the network is huge and the activity logs are massive, it is hard to explain WHAT happened, WHERE it happened, and, what is more important, WHY it happened. And the more active your users and the larger your network, the more complicated these questions become to answer.

This social network is just one example of a large dynamic network where anomaly detection, inspection, and analysis may be required. Apart from that, good candidates are Web networks, email networks, and brain networks. You can think of any other domain that could fit this data model. There are only three requirements for the data. Entities in the dataset 1) could be represented as nodes, 2) have some relations among each other, and 3) generate some activity over time.

Introduction

In this post, I am going to present our recent work with Benjamin Ricaud, Kirell Benzi, and Pierre Vandergheynst (EPFL, LTS2). It is about anomaly detection in the dynamics of social and Web networks (preprint is available on ArXiV). This is yet another application of our algorithm that I already described in the previous blog posts. So far, we were able to detect events and study collective memories. This time, we realized that we can use the same (but slightly modified) algorithm to detect anomalies in the dynamics of communication networks. Moreover, I will show you how we can use it to investigate network attacks and recover traces of an attack if the intruders tried to hide or erase them.

Take a look at the related research and interactive demos on the website of our Wikipedia Insights project.

Method

We define an anomaly in the network dynamics as an unexpected spike of activity in a cluster of nodes. So, the objective is to find those clusters. As I said before, our networks are very dense, large, and the corresponding activity recordings are very noisy. This makes it hard to localize the anomalous clusters. This is where we came up with an idea of the algorithm that would help us to keep only those parts of the network that could potentially be related to anomalies in the network.

The core of the algorithm is a modified Hebbian learning rule, a theoretical assumption made by neuroscientists to explain the learning function of the brain. The simplified original rule implies that if two neurons (nodes) are active together, they tend to increase the strength of the connecting synapse (edge weight) between them. We modify this rule to fit our dynamic network data model. First, we do not care about the causality of activations and focus on simultaneous activation of the nodes in the network. Second, we introduce a convenient update function that allows us to forget about such problems as normalization and weight thresholds during learning.

Speaking of our data model, the algorithm consists of three main steps and works as follows. First, we select “spiky” nodes that have unexpected bursts of activity. We simply use mean and standard deviation to assess the level of the bursts. Once we identify the spiky nodes, we remove the ones with uniform activity. Second, we compute the strength of the connections (weights of the edges) between the spiky nodes using the modified Hebbian learning rule. In other words, if two nodes are active at the same time, the weight of an edge connecting them increases, otherwise, it either remains unchanged or decreases when the forgetting parameter is on. When the forgetting parameter is on, we want the algorithm to forget about previous anomalies. If it is off, the algorithm accumulates anomalous events and stores them as a memory. Finally, when the weights are computed, we remove low weight edges so the remainder of the initial graph contains clusters of the nodes with anomalous activity.

As you can see, all computations are local on the graph. This means that the weight updates of adjacent edges of a node depend on the node’s attributes and the attributes of its neighbors. This allowed us to fit the algorithm in GraphX framework and implement it in a distributed fashion. See the code on GitHub. If you are new to Scala and Spark, take a look at this tutorial.

UPDATE 0: We have a new enhanced and more readable version of the code available here. To run it, you need to deploy Neo4J and Apache Cassandra databases. You can find deployment instructions in the same repo on GitHub. See details here.

UPDATE 1: If Scala code is confusing, I wrote a minimal Python code example with experiments on random data. Check out this GitHub repo for more details. It should provide a quick but good understanding of our approach. Since the algorithm relies on the structure on the graph, the results with random graph are not very impressive. However, even though the structure of random graph is very dense we have managed to get decent results (see images below below).

Anomalous time-series		Initial random graph		Detected anomalous cluster (bold edges)

Experiments

We conduct experiments on two datasets: Wikipedia web network and Enron email communication network. The datasets are publicly available (check out the links).

Wikipedia web network

In this experiment, we studied Wikipedia web network. The aim was to detect anomalous behavior of Wikipedia visitors. Here, we use Wikipedia pages as nodes and hyperlinks connecting the pages as edges of the graph. Every node has a time-series attribute corresponding to the number of visits per hour. Wikimedia Foundation makes viewership activity publicly available so we used this data to run the experiment. The pre-processed dataset is on Zenodo (see links in the previous paragraph).

This experiment is quite interesting because it is hard to validate the results quantitatively or present a numerical measure that would reflect the success rate of the algorithm. As a workaround, we decided to validate its results using another anomaly detection framework as ground truth. In this experiment, we use Google Trends API as a benchmark.

As you see in the figures below, the results are quite impressive. The algorithm has managed to detect multiple anomalies in Wikipedia viewership activity. These anomalies correlate with the trending keywords on Google Trends.

Besides, there is another detail I want you to focus on. Look at the figure reflecting anomaly related to Germanwings 9525 crash. There is a smaller spike of activity at the end of December 2014. The fact is that there was another airplane crash at the end of December 2014, Indonesia AirAsia Flight 8501. We detected this event not because of a bug or a noisy activity pattern. This is a feature of our approach that reflects an aspect of collective behavior of Wikipedia visitors, which is called Collective Memory. The phenomenon of collective memory suggests that social groups recall past traumatic events when another similar traumatic event happens at present. This is exactly what our algorithm detected. When Germanwings crash happened, people started searching for other related airplane accidents. Google Trends has not reflected this since it focuses on particular keywords whereas our approach looks at more general picture due to the graph representation of the data. This means that our algorithm detects complex anomalies in the viewership dynamics that can only be spotted if one looks at the group of concepts rather than a particular keyword.

Ferguson unrest		Charlie Hebdo attack		Germanwings 9525 crash

Enron email network

We also investigated Enron email dataset and ran another anomaly detection experiment on it. We built a communication network based on the emails sent among employees. Nodes of the network correspond to employees; we drew an edge between two employees if they have exchanged at least a few emails. Activity on the nodes is the number of emails sent by corresponding employees.

In this experiment, we had ground truth. We knew when the anomalies in the dynamics happened. This gave us a chance to verify the accuracy of our algorithm. There are four main events related to the Enron scandal (see details in the paper). We found that the real-world events related to the scandal triggered the anomalies in the communication dynamics of the corporate email network. The image below shows the result of the anomaly detection algorithm.

Attack investigation. Data recovery

We got to the final part of the blog post. Imagine, someone attacked your network trying to wipe network activity data and this resulted in an anomalous spike of activity on the nodes of your network. Or, simply, you lost information about network activity due to an unexpected shutdown of the hosting server. In any case, the main problem when this happens is to recover missing information about the network activity. Our algorithm can help.

Below, you see an example of the recovery of damaged parts of the data. Here, we wipe 20% of viewership activity data in a cluster of Wikipedia pages and recover it using our algorithm. To memorize and recover missing data we used Hopfield neural network, a basic model of artificial associative memory. Incidentally, Hopfield nets also use the Hebbian learning rule.

We use prior activity of the network to learn collective patterns in the network dynamics. The patterns correspond to clusters of pages with similar anomalous behavior. We then use these clusters to recover the missing data. We found that the algorithm allows recovering missing data for the period when the anomaly has happened (in this example, it is bounded by the red vertical lines). It is also possible to recover more information outside these bounds but here we focused on the information that is most interesting for us, the missing information related to the moment of the attack.

As you can see, when recovering the data we heavily rely on the neighboring nodes in the cluster. This means, if we had the entire cluster wiped by the intruders, we would not be able to recover the original data. To recover missing information in the cluster we need to have at least 50% of nodes with valid undamaged observations. This is a limitation of our approach.

Read more about the memory properties and the data recovery feature of our algorithm in my previous blog post.

Conclusions

Networks are everywhere. We are surrounded by networks in pretty much all aspects of our lives explicitly or implicitly. It is important to develop a stable and robust toolkit for monitoring these networks and protect us from unexpected accidents related to them. Here, we introduced a new method for anomaly detection and missing information recovery in the large networks. We showed that the network structure carries a lot of useful information in itself, hence, it is very important to use it when working with the dynamic or time-varying data that have spatial or structural component. All things considered, I believe that in the following years, we will see more and more applications related to spatio-temporal data mining (for more details about this emerging field, take a look at this recent survey).

Movie genres classification using character interaction networks

Mon, 09 Jul 2018 13:06:13 +0200

Social interactions are among the most important needs of everyday life. Whenever we communicate with relatives, colleagues, friends or acquaintances, we create invisible social links adding those people into our personal social network (here, I mean the real one, not 500+ Facebook friends and LinkedIn connections). Sometimes, we look at interactions in our networks and compare our lives to movies and TV series trying to find similarities with real life. This made me think of a couple of interesting questions. Can the structure of our social network tell us anything about ourselves? What if it defines the genre of our life? Let’s try to look at this question from a data scientist’s perspective.

To do that, first, we would need a labeled dataset reflecting “social network”=>”genre” relation. Clearly, we do not have access to real-life social networks or even if we were able to extract them from, say, Facebook, first, that would be considered a privacy violation, and second, we would not have labels or “genres” characterizing each network. Although, the good news is that we have got movies and it is possible to extract networks of character interactions. Also, we know the genres of each movie. Fortunately, the guys from Moviegalaxies have already captured quite a few of these networks and published them on their website.

All in all, we found a suitably labeled dataset containing a “character network of a movie”=>”genre” relation. Now, we can build a machine learning model to find out whether the structure of character networks has anything to do with genres of corresponding movies.

Data

Moviegalaxies is a collection of social graphs in movies. A team of researchers from RWTH, Cologne University, and MIT (Jermain Kaminski, Michael Schober, Raymond Albaladejo, Oleksandr Zastupailo, and Cesar Hidalgo) has collected and published social network interactions among characters in 774 movies (and counting). They used “same-scene appearance” approach to build these social networks. This means, for example, if Padme and Anakin appear in the same scene in Star Wars: Episode II Attack of Clones, they are connected in the social network of the movie. On the other hand, Amidala and Yoda have never appeared together, therefore there is no edge linking them in the graph.

Although, it is not that easy to get genre information for each movie from Moviegalaxies website. In order to obtain this information, I used Open Movie Database (OMDB) API. The service provides 1000 requests per day for free which are enough to get more information about each movie in the Moviegalaxies collection.

For each movie, we consider only the main genre and omit the secondary ones. Here and in all experiments below, we keep only the top 7 most popular genres. As we can see in the chart below, the dataset is quite imbalanced.

Note: Visualizations work best with Chrome.

Features

We want to find out whether the structure of character networks is enough to define movie genres. To do that, we are going to look at the topological properties of graphs, corresponding to the character networks. In particular, we are interested in assortativity, transitivity, degree distribution, clustering coefficient, and modularity. Also, we consider the number of nodes and edges in social graphs as features.

Apart from having fancy names, topological properties of graphs can tell us interesting stories about social networks. For example, clustering coefficient, transitivity, and modularity reflect to what extent members of a network tend to connect or cluster together. High assortativity coefficient indicates that people in the network tend to socialize with peers that have similar properties (popularity, for instance).

After removing outliers and normalizing, let’s take a look at pairwise relationships of the features in our dataset. To reduce clutter, you can choose genres using check-boxes below. I recommend comparing genres that have a similar number of samples, e.g. Horror vs Adventure.

Note: you may need to wait a second after you press a checkbox. Works best with Chrome.

Comedy Drama Action Crime Biography Adventure Horror

As we can see, our dataset is quite complicated since we have overlapping classes. I had an assumption that the features may have non-linear relationships that would help to separate the classes. To verify that, I applied manifold learning to the dataset in order to learn non-linear features, although this had not improved the classification results. Check out this Jupyter notebook for more details.

Even though the dataset is far from being perfect, we see that feature distributions of some genres are slightly different. Take a look at Horror, Biography, and Adventure, for example. This means we could try to use these features to classify movie genres.

Classification

We want an interpretable model since we need to know which topological properties are common to a certain genre. Hence, Decision tree algorithm is a good candidate.

Our dataset is imbalanced. Fighting imbalanced problems is quite difficult especially when we have a very limited number of samples. Also, this is not the purpose of this experiment, so we go for a workaround. To simplify our task and the model itself, let’s split the data into two groups to have more or less the same number of samples in each group: 1) Action, Comedy, Drama and 2) Biography, Adventure, Horror. Also, let’s put aside Crime movies for now.

Below, you can see the results of classification using Decision tree model that we trained using our data. We built decision trees for each pair of genres in our groups. We can see that when we try to classify only two genres (binary classification), the accuracy of the models is quite high, reaching 79% for Horror vs Biography pair, for example. However, when we have more classes in our model, the results are only slightly better than a random guess for Horror vs Adventure vs Biography (56%) and even worse than that for Action vs Comedy vs Drama (45%). This implies that some genres have similar topological properties of character networks.

To read and interpret the trees below, assume that the upper branches correspond to True conditions and the lower ones are False.

Let’s take look at the actual output of the model. Below, you can see classification results for each genre in every group. We can see that the majority of the movies are classified correctly in all cases. Although, the number of misclassified movies grows with the number of classes in our model.

Interpretation of the results

By looking at the decision trees we built, we can say which features characterize each genre. For example, if we compare Horror vs Biography, we can see that Biography movies tend to have lower modularity and higher assortativity measures. In other words, characters in Biography movies tend to socialize with those who have a similar level of centrality and importance in a movie, but, at the same time, they do not form strong communities. Reminds of real life, doesn’t it? On the other hand, characters in Horror movies socialize with everybody but tend to interact in groups (to me, this sounds like networking at a conference). Take a look at the decision trees above and try to interpret the results explaining particular features of each genre.

Conclusions and possible extensions

I always feel like static networks lack an important component – temporal information. In this project, we ignored the time-series of character co-occurrences. It would be interesting to look at the dynamics of character interactions in movies. If we had time-series of character interactions, we would be able to learn weights between nodes in graphs. That would give us insights into the strength and importance of connections in social networks in the movies. One way to learn the weights would be to use the approach we developed for Wikipedia graph mining.

Besides, this time, we chose to avoid the problem of imbalanced classes. It would be interesting to apply weighted versions of Random trees or XGBoost algorithms to compare classification results for all genres in the dataset.

Due to the low number of data samples in the Moviegalaxies dataset, it was more of a fun project rather than a serious research. I am looking forward to the next data release by Moviegalaxies team. Nonetheless, the results are quite interesting and we have actually managed to find a few insights about social networks in movies. In general, the idea of using topological properties of graphs for classification problems still excites me. If we had more data related to this problem, it would be interesting to build another model at a larger scale.

How are Web Networks similar to the brain?

Wed, 14 Feb 2018 10:59:13 +0100

Web networks resemble the brain if we think of web pages as neurons. Indeed, interconnections have complicated structure, while nodes produce time-series of activations (visits of web pages and spike-trains of neurons). A detailed and very exciting comparison available on ExplainThatStuff.com. Here, we focus on memory properties of the Web and show in what way it is similar to human memory.

Wikipedia Web network as a global memory

In my previous post, I showed how we can identify collective interests of people over time using Wikipedia Web network and its page view statistics. We called these collective interests Collective memories. We demonstrated that the structures, which we extracted from the Wikipedia Web network, comprise pages related to certain events. Clearly, if we look carefully at the structures of the learned graphs we will see that they have associative nature. This insight led us to another interesting question. Are these structures similar to artificial models of human memory?

To answer this question, we performed a few experiments. We took one of the most popular models of the associative memory, Hopfield network, and modeled the recall process. Hopfield network is an artificial recurrent neural network model that serves as an associative memory. This model was also used for understanding of human memory, so we decided to use it to test our assumption.

Method

Instead of learning the weights of the neural network in a conventional way, we took the weighted adjacency matrices of the graph structures that correspond to the detected collective memories. As you remember from the previous post, we learned them using a modified Hebbian learning rule. Then, we applied these weight matrices to the time-series of the Wikipedia page views (for more details, read preprint on arXiv).

Experiments

In the first experiment, we used the collective memory graphs that we learned for each month. In the images below, you can see the recall results. The model re-enforced activity levels for every month in our dataset. Red areas show the periods when the memory is inactive, while green areas correspond to the moments of memory activation.

October		November		December		January		February		March		April

After that, we tested if our model is able to reconstruct missing collective memories. Here, to give an example, we show the second wave of Ferguson unrest collective memory. The unrest occurred on the 24 of November 2014. We remove 20% of activations in the cluster of the collective memory and apply Hopfield recall model. The image below illustrates a recall from a partial pattern.

As you can see, in this case, the model behavior is quite interesting. The model does reconstruct missing memories, but also it acts as a filter. It keeps only those parts of the signal that are relevant to the event. Indeed, the behavior reminds of the way our memory works. We memorize better if we find an association with something. We focus our memory on a certain period of time when an event occurred and forget scattered unassociated memories.

Check out the latest version of our paper on ArXiV to get the technical details of the experiments. The code is available on GitHub. If you want to try to reproduce the experiments but are new to Spark Scala projects, take a look at this tutorial.

Besides, we realized that we can use the aforementioned memory properties to recover missing or corrupted records reflecting the dynamics of social and web networks. Take a look at another blog post for more details.

Wikipedia graph mining: dynamic structure of collective memory

Mon, 14 Aug 2017 10:06:13 +0200

This is the accompanying blogpost for our upcoming research paper (read preprint on arXiv); joint work with Kirell Benzi, Benjamin Ricaud, and Pierre Vandergheynst (EPFL, LTS2). Here, we focus on the results, omitting the details of the algorithm and the implementation.

Intro

Wikipedia is a great source for data analysis due to its outstanding scale and the graph structure. Tens of millions of visitors surf it daily, leaving their footprint on the Web. The combination of the Wikipedia graph structure and visitor activity on the pages gives us the dynamic graph – the graph with time-series signals on the nodes. The dynamic nature of the graph makes the large-scale analysis problem complicated.

In the original paper we analyze the Wikipedia graph. The aim is to detect collective memories using the activity of the Wikipedia visitors. We use graph-based approach to build our model. The computational model is inspired by the synaptic plasticity and Hebbian theory.

Not surprisingly, we could not fit all the results into the paper. Apart from that, PDF is a poor format for communicating the research findings. The aim of this post is to show the results in an interactive way. While reading the paper and this post, we encourage you to open the graphs, appearing everywhere in this post, and play with them: zoom, click, move, search, and select. This is by far the funniest way to plunge into the main results of our work.

Graphs are interactive

Click on any graph in this post to open it in a new window.
Zoom, click on the nodes, search pages by name, highlight clusters by color.
- When you click on a node – you select all the neighbors.
- When you select a cluster – you select all the nodes in this cluster.
- The list of selected nodes appears on the right.

Works best in the latest version of Chrome. DO NOT try to open the graphs on a smartphone. The graphs are too large and it might take forever to render.

Dataset

The original datasets are publicly available on the Wikimedia website. We took the SQL dumps of the English Wikipedia articles to create the graph. The visitor activity is the number of visits per page per hour. We consider the period from 02:00, 23 September 2014 to 23:00, 30 April 2015. Pre-processing details are described in our paper in the Dataset section.

Dynamics of the Wikipedia network

In the paper we make an assumption that the dynamics of the graph can affect its structure. We apply the update rule, based on the signal on the nodes, to observe this effect. Here we show that the Wikipedia graph can self-organize into the sets of meaningful communities of the nodes, if we take into account the visitor activity dynamics of the graph. Click on the graph on the right and explore the result by yourself.

This graph is the result of 7-month dynamics of visitor activity on Wikipedia. Here you can find main events that had taken place over the considered period. The stable or scheduled events, like tournaments, awards ceremonies, contests, and most popular holidays form big clusters. The unstable or unexpected events, like incidents and accidents are grouped in small clusters. Even though, this graph provides good summary of the dynamic patterns, we can see only the final result. What is more important, is to get insights on the dynamics of the graph over time. How the clusters emerge, evolve, and disappear? To answer this question, we pick one particular event and look at its dynamics in details.

Dynamics of an event: NFL championship

In order to understand the dynamics of evolution of the graph, we pick one of the most popular events, highlighted in English Wikipedia – the National Football League (NFL) championship. We consider the season 2014-2015. The plot is on the right (click to enlarge). For the plot interpretability we extracted 30 NFL teams out of 485 pages in the original cluster. The timeline shows the overall cluster activity over the period of 7 months. The dynamics timeline of the graph and the NFL cluster evolution is illustrated on the top row. It reflects the interest of the NFL fans in the championship. The cluster is small and sparse at the beginning of the championship and becomes denser and bigger, approaching the final game date. The behavior of the Wikipedia visitors during the day of the final game Super Bowl is outstanding. The activity of the NFL fans is much higher, comparing to the activity of other Wikipedia users. It makes an analogy to the real life, when during the finals the fans become the most active people on the streets.

The NFL championship is just an example of a detected event and its evolution. You can dig into the graphs of monthly activity and check out other detected event-clusters. The total number of the detected events is 172. Click on the graphs below to open an interactive version and explore by yourself.

October		November		December		January		February		March		April

The NFL cluster is a good example of a stable event, represented as one of the biggest clusters in the resulting graph. What about the unscheduled events, like attacks and other accidents?

Collective memory

Traumatic events, like terrorist attacks, flight crashes, wars, and conflicts, often remind us of the past. These memories are often common for most people in a social group. That is the reason why they are called Collective memories. Our approach allows to detect these memories and serves as a general model for collective memory emergence. We provide the examples of 3 events, detected among the others.

The table below contains examples of collective memories. To show the details of the detected collective memories, we pick 3 particular events among the others detected: Ferguson unrest (second wave - November 24, 2014), Charlie Hebdo attack (January 7, 2015), Germanwings flight 9525 airplane crash (March 24, 2015). Top row contains the extracted clusters of collective memories for each of the discussed events. Bottom row shows the detailed activity of each page in the clusters.

Ferguson unrest		Charlie Hebdo attack		Germanwings 9525 crash

We see that the core events trigger relevant memories. Ferguson unrest reminds us of other riots and other shootings of innocent people of color. Charlie Hebdo shooting has links to the other terrorist attacks, bloodshed, and law enforcement agencies. Germanwings crash is surrounded by the dense group of the other airplane crashes, indicating that the flight accidents are thoroughly structured on Wikipedia.

Although, we can see slightly different behavior in the clusters. For example, the activity during the Ferguson unrest has two distinctive spikes that indicate beginning and end dates of the riot. In case of the Germanwings, the activity of the cluster has a spike in December. The reason is another airplane crash that occurred in December and for this reason got attached to the main cluster.

The short-term events and their collective memories can be found on the monthly dynamic graphs, presented in the previous section in the timeline table. Check the graphs out and search for the events of your interest.

Conclusions

Wikipedia can tell us more than is written on its pages. It is a great source of data for the collective human behavior research. Nonetheless, the dynamic nature of the graph-structured data leads to new challenges for data science and machine learning. In the paper we proposed a new method for collective memory modeling and understanding. We applied the method to the Wikipedia datasets. We detected collective memories using the combination of the Wikipedia Web network graph and the its visitor activity history.

We assume that the learned graphs have associative memory properties. To verify this assumption, we model the recall process using Hopfield networks. The results of these experiments are in my blog post describing these memory properties.

Besides, we noticed that the events we detected were triggered by the anomalous activity of Wikipedia users. This inspired an idea of an anomaly detection algorithm that I described in another blog post. Take a look at the interactive demos of the results on the website of our Wikipedia Insights project.

Tools and code

We make all the experiments using Apache Spark GraphX. The code is written in Scala and available on GitHub. If you feel like playing with the code but new to Scala and Spark, take a look at this tutorial.

Acknowledgments

I would like to thank Michaël Defferrard and Andreas Loukas for fruitful discussions and useful suggestions.