Chasing 80%: Peripheral Sentiment

Chasing 80%: Peripheral Sentiment

I recall that academic research performed in the past has demonstrated that determination of polarity in text (what we have come, perhaps incorrectly, to call “sentiment”) has an accuracy ceiling of about 80%. That is, given a body of statements, a team of human beings will not agree on the sentiment in those statements more than 80% of the time. That’s because, well, people have opinions.

So, the best we can hope to achieve in automated sentiment analysis would be that 80%. A lofty goal, indeed.

In an earlier post, I discussed the importance of World Knowledge in making good decisions: that knowledge we’ve gained from lining in our world that biases, hopefully correctly, the calls we make. Computers are not good at world knowledge, although we’re always trying to get better.

At Essencient, we have recently developed a concept that we call Peripheral Sentiment. Since our objective is not to make decisions for you, but to make sure that the texts you see are worthy of human evaluation, PS is a very useful thing.

PS says that for a given topic, if there is no directly measurable sentiment for or against it, but there is sentiment of some kind at the level of the entire text, some of that latter sentiment might be ascribed to the topic. Therefore, the text is probably important enough for human evaluation.

Here’s an example: “I stayed at Hotel Roger last night. The food was terrible.” World knowledge tells us that a hotel can be judged by its food. Therefore, this text could be considered negative about Hotel Roger, although nothing directly critical of that topic was said. Peripheral sentiment score: negative. Human, please take a look.

It’s not perfect. Consider this one: “Drove the Benz to dinner last night. The food was terrible.” You get it, of course: this says nothing bad about the car, since food and cars have no world knowledge relationship. In this case, we’d put this in front of a human, who would probably consider it unimportant (about Benz, that is).

Still, even with imperfections, it does make it more likely that texts with only indirect sentiment towards your brand will at least end up getting reviewed. That’s why PS takes us a step closer to the 80% mark. Expect to see more about PS going forward.

About the Author