Newsaccess Study - Automated vs Manual Sentiment Analysis
When reviewing press coverage how does automated sentiment analysis compare with manual sentiment analysis?
Newsaccess has recently carried out research to gain a greater understanding of the impact automated software can have on the outcome of sentiment analysis.
We know automatic analysis is fast but is it accurate?
To examine this issue, events considered to be of national importance were monitored on Twitter. The tweets were analysed by automated sentiment software and by an experienced member of our evaluation team.
The results were fascinating.
Differences between the two methods were clearly split between three distinct categories:
- The type language used in the tweet
- The use of organic hashtags
- Slang
- Context
Language
Maximum Media’s Niall McGarry shows (just a little) sarcasm in his 140 characters, but the automated analysis didn’t register it. The words ‘successfully’ and ‘profit’ meant this tweet was deemed to have positive sentiment. Our manual evaluation read the context of this tweet and immediately deemed it to reflect negatively on the subject of the story – Rehab/Angela Kerins.
Hashtags
One of the biggest challenges facing many companies today is creating hashtags that will actually catch-on with twitter users. There are countless examples of organic hashtags wiping out the attempts of digital teams to put their own stamp on tweets. But how does our automated software cope when users dictate exactly what they are tweeting?
The above tweet spills positivity about BOD as emotions ran high on his final Lansdowne Road performance. The ‘#’ symbol and the ‘no gaps’ rule for hashtags means the software simply couldn’t distinguish the true sentiment of the tweet – deeming it to be neutral. Newsaccess rugby-loving evaluator immediately saw that this was a positive tweet.
Slang
The single biggest challenge for automated sentiment analysis is understanding colloquial language.
The dynamic nature of social media means that it’s almost impossible for automation to keep up with the constantly changing slang that seems to appear on a daily basis. Our analysis picked this up in the classiest way possible, with a tweet which could have been directly taken from an episode of The Only Way Is Essex.
Automated analysis deemed this tweet to reflect negatively on the subject. Our evaluator correctly identified the complete opposite.
Context
Twitter is about conversation. But what happens if the context of a conversation is misunderstood. Automated analysis is at a far greater risk of assigning an inccorect sentiment, as it cannot take in to account the posts prior to the one being analysed. Identifying context in manual analysis is far easier, allowing the reader to assign the correct sentiment.
The result
Tonality and Sentiment analysis is subjective in nature. On the positive side automated sentiment analysis is far faster than the manual process. However our research found significant error rates when compared with human analysis. The Brian O'Driscoll coverage had an 87% error rate between manual and automatic results deemed to be Negative. Neutral results had a 38% error rate while positive results had a 34% error rate.
As part of a simple test this may not seem too important, but in the context of a planned media campaign, the tonality of coverage can have a very significant consequences.
This research reaffirms our belief that that manual sentiment analysis is the only way to achieve accurate results when evaluating media coverage.
Paul Moriarty is Head of Insight at Newsaccess Media Intelligence. Newsaccess is the leading provider of media monitoring and media analysis service in Ireland. For more information on Newsaccess click here.
