Stance Detection in Tweets: Yes or No?
This is the project page for SemEval 2016 task 6: Detecting Stance in Tweets.
DATA and Task:
- A topic, a piece of text, mentions and hashtags are given. The question is whether the tweet is in favour, against or neutral about the topic.
227 Atheism The Son of God became man so that we might become God. ~Athanasius #SemST AGAINST
123 Atheism 'Pop may throw in the towel second half.. #SemST NONE
375 Atheism @GregAbbott_TX which god? Yours? not mine. oh wait i don't have one. #LoveWins #SemST FAVOR
The distribution of labels is shown here. The AGAINST tweets are twice the number of FAVOR or NONE (neutral) tweets.
The distribution of topics is shown here. The distribution of tweets among topics seems to be almost uniform.
- The number of training and development samples are 2815 and 100 respectively.
Models
- Baseline 1: Train a text-based classifier over all training samples regardless of their topic and test it on development data. The cost function is hinge loss (SVM) and Elastic Net (50% l1 and 50% l2) regularisation. 10-fold cross validation has been used for tuning the regularisation coefficient.
`
_ precision recall f1-score support
AGAINST 0.68 0.85 0.76 53
FAVOR 0.70 0.64 0.67 22
NONE 0.71 0.40 0.51 25
avg 0.69 0.69 0.68 100
`
Evaluation
- Task is evaluated with the average of f-scores for FAVOR and AGAINST labels.