Sentiment Accuracy vs. Sentiment Accuracy
** Posted as a comment to this post at lexalytics **
Great post, we’ve also found it pretty easy to build a system that gets 70-80% accuracy in almost no time at all.
Sentiment accuracy is also interesting and measuring it becomes all the more difficult when you consider it as a spectrum rather than simple agree/disagree.
For example, when dealing with financial sentiment measurements as we do, you can be a little wrong, or you can be REALLY wrong.
Getting the right polarity 80% of the time is great, but you also need to consider what 20% you missed. Humans who disagree will usually have agreement on the highly polarized articles. We expect people to disagree on the more intricate cases.
In our experience, even if you are getting the same % agreement overall human-human or human-computer, computers are much more likely to throw articles humans would all agree on into the wrong bucket.
