SEDTWik: Event Detection from Tweets

Implementation of my paper: SEDTWik: Segmentation-based Event Detection from Tweets using Wikipedia which is available here

SEDTWik Architecture

The process of Event detection can be divided into 4 parts:

1. Tweet Segmentation

Split a given tweet into non-overlapping meaningful segments, giving more weight to hashtags (𝐻). Filter out words not present as a Wikipedia page title.

Tweet	Segmentation (with 𝐻 = 3)
Joe Biden and Paul Ryan will be seated at the debate tonight #VpDebate	[joe biden], [paul ryan], [seated], [debate], [tonight], [vp debate]x3
Amanda Todd took her own life due to cyber bullying #RipAmandaTodd #NoMoreBullying	[amanda todd], [cyber bullying], [rip amanda todd]x3, [no more bullying]x3

2. Bursty Segment Extraction

Score segments based on their bursty probability (𝑃_𝑏), and follower count (𝑓𝑐), retweet count (𝑟𝑐), and count of unique users using them (𝑢). Select top 𝐾=√(𝑁_𝑡 ) segments based on 𝑆𝑐𝑜𝑟𝑒 (𝑁_𝑡 = total number of tweets in current time window).

𝑃_𝑏(s) measures how frequent a segment is occurring compared to its expected probability of occurrence.

Score_s = 𝑃_b(𝑠) × log⁡(𝑢_s) × log(rc_s) × log⁡(log⁡(𝑓𝑐_s)).

3. Bursty Segment Clustering

Variation of Jarvis-Patrick Clustering algorithm.

Segments considered as nodes in a graph and 2 segments belong to same cluster if both are in 𝑘-NN of each other.

Segment similarity: 𝑡𝑓−𝑖𝑑𝑓 similarity between contents of tweets containing the segment.

After creating candidate Event clusters, discard those that have newsworthiness value beyond a threshold.

4. Event Summarization

Use all tweets containing segments of the event cluster and apply any text summarization algorithm to them to obtain a summary of the event.

Event Summarization in itself is a big research area and many sophisticated methods are available to summarize text.

A simple way to do this is by using LexRank (Extractive Text Sumsmarization) algorithm.

We leave this part to the user to use any appropriate Summarization method.

Cite

@inproceedings{morabia-etal-2019-sedtwik,
    title = "{SEDTW}ik: Segmentation-based Event Detection from Tweets Using {W}ikipedia",
    author = "Morabia, Keval  and
      Bhanu Murthy, Neti Lalita  and
      Malapati, Aruna  and
      Samant, Surender",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Student Research Workshop",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/N19-3011",
    pages = "77--85",
}

Abstract:
Event Detection has been one of the research areas in Text Mining that has attracted attention during this decade due to the widespread availability of social media data specifically twitter data. Twitter has become a major source for information about real-world events because of the use of hashtags and the small word limit of Twitter that ensures concise presentation of events. Previous works on event detection from tweets are either applicable to detect localized events or breaking news only or miss out on many important events. This paper presents the problems associated with event detection from tweets and a tweet-segmentation based system for event detection called SEDTWik, an extension to a previous work, that is able to detect newsworthy events occurring at different locations of the world from a wide range of categories. The main idea is to split each tweet and hash-tag into segments, extract bursty segments, cluster them, and summarize them. We evaluated our results on the well-known Events2012 corpus and achieved state-of-the-art results