Callum Borchers' Blog

Sociologically significant sports (and class assignments)

NewsTrust rating system adaptable but imperfect

with one comment

My initial impression of NewsTrust was that it is very serious. As its name implies, the organization’s purpose is to gauge how strongly readers can trust the news they read — a pretty serious mission. So, given the freedom to post stories of any kind for a class assignment, I was curious to find out how NewsTrust would handle a topic slightly lighter than the Haitian cholera outbreak.

I led off with Dan Shaughnessy’s column about yesterday’s Patriots-Colts classic, and was impressed by the way NewsTrust adapts its rating system to this genre. Instead of asking, “Is this story factual?” and “Is it fair?” as NewsTrust does about hard news stories, the site asked, “Is this story informative?” and “Is it insightful?”

In this way, NewsTrust passed a big test in my estimation. The concept of a point-based rating system is a good one because it allows for easy comparisons between articles. However, such a system only works if it fits every story, and NewsTrust has made an effort to tailor its system in that manner.

The system is imperfect, however. While I like the tiered rating forms — short, quick, full, advanced — I’m not a fan of the review forms, also available in short, quick, full and advanced. It’s great that NewsTrust caters to people with two seconds or two minutes to spare, but eight different ways to evaluate a story is just too many.

Also, it seems odd to differentiate between a review and a rating. In NewsTrust world, the difference is that review questions are black-and-white — a story is either fair or unfair, factual or not factual — and rating questions are more nuanced, scored on a five-point scale. However, reviews are converted to the five-point rating scale to help measure a story’s overall score. For example, a reader using the short review who called a story factual and fair and said he would recommend the piece produces a score of 4.0. So, on the short review form, there is no way to award a story a perfect 5.0.

Since I used the full rating form to evaluate two other stories, Boston Globe pieces about holiday shopping projections and the UMass presidential search process, I managed to produce more thorough scores. But, in some cases, I think NewsTrust’s rating system sacrifices accuracy for convenience.


Written by callumborchers

November 22, 2010 at 9:04 am

One Response

Subscribe to comments with RSS.

  1. […] Callum Borchers: “It’s great that NewsTrust caters to people with two seconds or two minutes to spare, but eight different ways to evaluate a story is just too many.” […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: