{"id":2245,"date":"2018-11-20T14:30:52","date_gmt":"2018-11-20T18:30:52","guid":{"rendered":"https:\/\/www.danielpradilla.info\/blog\/?p=2245"},"modified":"2018-12-04T22:49:14","modified_gmt":"2018-12-04T22:49:14","slug":"recommender-system-for-finding-subject-matter-experts-using-the-enron-email-corpus","status":"publish","type":"post","link":"https:\/\/www.danielpradilla.info\/blog\/recommender-system-for-finding-subject-matter-experts-using-the-enron-email-corpus\/","title":{"rendered":"Recommender system for finding subject matter experts using the Enron email corpus"},"content":{"rendered":"

\"\"<\/a><\/p>\n

This is a little project to create a recommender system to find mentors inside an organization, using Natural Language Processing. It started as an excuse to build a data visualization I had in mind: an interactive word cloud that did something. When I started, I didn’t know anything about Topic Modeling, Topic Extraction, or Natural Language Processing; and fell head first into a rabbit hole.<\/p>\n

TL;DR:<\/h2>\n

Topic extraction is deep and potentially rewarding. Sanitize properly. SpaCy<\/a> and Gensim<\/a> are your friends.\u00a0Search YouTube<\/a> for knowledge. This is related to “Topic Extraction from Scientific Literature for Competency Management<\/a>” and “The Author-Topic Model for Authors and Documents<\/a>“. Get the code of this project at\u00a0https:\/\/github.com\/danielpradilla\/enron-playground<\/a><\/p>\n

<\/p>\n

General Description<\/h2>\n

Imagine we would like to know who is the best person to ask about a subject inside a company \u2013a potential mentor. One way would be to infer each person’s speciality from their main body of work: emails.<\/p>\n

If I lived in another world in which privacy is not an obvious concern \u2013or if I worked in Google\u2013 reading other people’s email would be totally kosher. In the normal, privacy-complaint world, this remains a purely academic exercise.<\/p>\n

However we do have access to a publicly-released corpus of emails to work with: the Enron email dataset<\/a>.<\/p>\n

My first idea was to use a named entity recognizer (NER), because if one were designing a recommender system for an energy company, one of the use cases would be to suggest whom to ask about a very specific technical issue. At the time, I found SpaCy<\/a> to have a nice NER for python.<\/p>\n

To identify the mentors, I assumed that whomever wrote an email about a subject, knew something about it. I’m not\u00a0that<\/em>\u00a0naive, I know that’s not always the case, but hey, that’s what I had! I could create a distribution subject-person and argue that the ones at the top of each subject knew something about it. Looking at the corpus, it seemed I had to extract the “from” field and the body of the email to build this distribution. I could use the email<\/a> module and Beautiful Soup<\/a> for HTML emails. With a little bit of text mining, I could transform this bunch of files \u2013over 600,000!\u2013 into a structured dataset.<\/p>\n

Before I dived in, I had to clean the text. I thought that a couple of regular expressions would suffice but these emails had some surprises in store:<\/p>\n