Rating distribution

Triggered by the shoutbox discussion I made a short check about how I use the four main ratings "Buy", "Rent", "Watch" and "Avoid" in my reviews. My impression was that "Buy" and "Avoid" should be relatively small subsets whereas "Rent" and "Watch" should be the majority. I have my ratings data in a spreadsheet so I can easily make some stats about them.

These stats made me aware that I've been too lenient in the "Buy-" area. So I now raised the minimum score for a "Buy-" from 82% to 83% which made quite a difference (9 of 250 rated anime at that time). Now I like the distribution of my votes much better than before.

The following list contains the number of anime I gave the corresponding rating, with the first number being complete works (series resp. movie), the second number being add-ons (specials/OVAs etc.).

• Buy+ (100-90%): 10 + 0 = 10 (3,39%)
• Buy (89-86%): 15 + 1 = 16 (5,42%)
• Buy- (85-83%): 26 + 5 = 31 (10,51%); all "Buy" = 19,32%

• Rent+ (82-79%): 27 + 8 = 35 (11,86%)
• Rent (78-76%): 34 + 7 = 41 (13,90%)
• Rent- (75-73%): 21 + 11 = 32 (10,85%); all "Rent" = 36,61%

• Watch+ (72-70%): 24 + 11 = 35 (11,86%)
• Watch (69-58%): 34 + 20 = 54 (18,31%)
• Watch- (57-50%): 11 + 8 = 19 (6,44%); all "Watch" = 36,61%

• Avoid (49-0%): 14 + 8 = 22 (7,6%); all "Avoid" = 7,55%

Edited on 08/10/2011 12:56 PM.

I was thinking that myself...

I just took a look at the http://anime.mikomi.org/reviewer.html page. How exactly does the CAR software calculate the "average rating" for Jan-Chan with a value of "Avoid"? (Arithmetic means of all values, including the "unevaluated" ratings?)

Yes, it's just a simple mean of the review's. Which worked great before we had "unevaluated" as a category. =p

Could we perhaps add another rating that might read "don't want to rate" or perhaps "your call on this one". Or even "Unrated".

I have posted a few reviews in which I intentionally left a series unrated (unevaluated) because I had some mixed feeling about the title, which I explain in detail in my review.

Edited on 07/01/2010 03:54 PM.

So if the software would just do a "sum" over all ratings that are NOT EQUAL "unevaluated", the result would be as expected? (I would still prefer a percentage value to an overall "Watch" or "Rent" though.)

How about a new rating Currently Under Review and make it the default value/setting for any new review postings?

This would address the issue that many of us have when trying to rate a new series that we are following while it is still airing.

Edited on 07/01/2010 04:04 PM.

For calculation purposes, "Currently Under Review" and "Unevaluated" are equivalent.

Having two values for them would be beneficial if we had a feature to group reviews by this rating. CAR has a "By Rating" access path but only for total ratings of an anime, not for just one reviewer (where it could easily list all "Currently Under Review" entries for this particular reviewer, thus implementing their "to do" list).

The downside of "Currently Under Review" might be that these reviews can be forgotten and the value might remain forever, then being even semantically equivalent to "Unevaluated".

For the sake of clarity I am all for two different values though.

I understand wanting to have a "Couldn't Say" or "No Recommendation" rating, and I'll look at adding that.

Can you explain to me how exactly "Currently Under Review" and "Unevaluated" differ? I read those to mean essentially the same thing. Or do we just want to rename "Unevaluated" to "Currently Under Review" as we'll have a "No Recommendation" rating?

I remember one understandable candidate for "Unevaluated" here.

Semantically the difference would be to be able to automatically separate between complete and incomplete reviews. This would be a concept not really explored at CAR as this portal gives low priority to numerical scores and thus doesn't sport anything like a list of anime reviews sorted by rating of one reviewer. Still, doing the same separation according to the overall rating (much like http://anime.mikomi.org/by-rating.html) would appear easy to implement and serve as "unevaluated" filter at the same time.

Edited on 07/03/2010 05:02 PM.

Ok then, how does "Unevaluated" in that example differ from "No Recommendation"?

I propose that I do this:

• Rename the existing "Unevaluated" to "No Recommendation"
• Add a new "Currently Under Review" rating
• Make the default rating be "Currently Under Review"
• Ignore both in the calculation of mean ratings for both reviewers and series.

The proposal sounds good.

I'm okay with the granularity for ratings of an individual review; it could be better for reviewer & anime averages though (an average of "Watch" or "Rent" for 500 reviews of one reviewer doesn't really give useful information, some decimal or percentage value would be better).

Thinking a bit more about the current ratings, CAR has:

• Watch: I'm not willing to pay money to see it, but if it's being shown I'll take the time to watch it.
• Avoid: If someone puts it on, I'll leave the room.

I don't think there is a need to make or change the system at all. Regardless of the site's meanings, I use the watch rating. If it has to be changed, then leave watch as a positive rating and make the not recommended rating separate. If I use the watch rating, it usually means not worth the money, but certainly worth the time.

For me, there is no real avoid rating as every title should be seen for your own take on it. I don't use Rent either because if I open my wallet to buy something, I'm not giving it back. So I only use watch and Buy. If we can split watch (positive) and (negative), it would give readers a better understanding overall.

Here's how I see it:

Buy - best rating. A first day purchase.
Rent - don't care for and will never use.
Watch (positive meaning) - should stay. we need a buffer for good anime that is worth the time spent watching.
new rating - a negative representation. Something higher than avoid.
Avoid - should stay but I'll never use.

If 'Buy' means 'worth spending a good deal of money on', and 'Rent' means 'worth spending a little money on', then different versions of 'Watch' might mean 'worth spending certain amounts of concentration/brainpower on'. Well, I thought I was on to something for a minute, but maybe this doesn't make sense...

That's exactly it, Stretch. It makes perfect sense. Watch means "worth the time spent". We just need to split watch into two divisions. One left the way it is and one being a more negative term, but still better than avoid.

One thing I often notice (and perhaps something that might divide 'Watch' shows into two categories) is that some shows are so confusing and overcomplicated that I decide that I won't even try to fully understand the entire story. This happens when the impression I get is that the overall quality isn't great enough to justify the concentration and close attention that would be required to catch every detail and completely understand the plot. Still, the show is worth watching. I'll just watch largely for the jokes and action without trying to tie each and every plot strand together. You might say I sort of treat the episodes as filler episodes even though they weren't meant to be that way. Maybe this would be a definition of Watch-: Worth watching, if it can be had for free, but not worth taking the trouble to completely understand it.

One more thing: A Watch- show would most certainly only be watched once, and probably soon forgotten.

Edited on 07/10/2010 11:00 AM.

As long as we have rating categories that some reviewers will never use there's something not right either in the concept of those categories or in the way they're seen by the reviewers. How should any visitor compare a "Rent" rating from some other reviewer against a rating from a reviewer who doesn't use the "Rent" category? How can the averages calculation lead to any meaningful results when some reviewers understand certain categories different than others?

And what about the differences between reviewers who have money to burn compared to those who are just getting by? Isn't a one-word rating nothing more than an extremely quick summary? If you want to be certain that you agree, you need to read the review in it's entirety (and even that might not make clear why a particular rating was chosen).

A "Buy" rating doesn't imply that everyone should buy this anime regardless of their taste and economic abilities. (Or would you buy every hardcore hentai just because it's rated as "Buy"?) But it should imply this if neither taste nor economic abilities are an obstacle.

If you don't address a "standard audience" because you want to take into consideration the economic potential of every single visitor here then why don't you give different ratings of the same anime for every single visitor? This wouldn't be very practical.

A rating should describe the quality of the anime - and thus must be independent of the visitor's particular abilities. It's not the visitor who's rated.

I have always taken the approach that I will rate something a 'Buy' if I would buy it--not because I think the average viewer would or should do the same. I guess the idea was that if, having read one or more of my reviews the reader thinks that I seem like somebody with tastes similar to his- or her own, they might want to buy the show as well. I sort of assumed that this was the way everybody was reviewing things... have I been doing something wrong?

@Xenoknight in 4225 wrote: Avoid - should stay but I'll never use.

Don't Bother works for me.... How about Waste of My Time or Meh - Waste of Time

Edited on 07/11/2010 06:18 PM.

I agree with you completely, Rebecca.

If I do feel it to be an "avoid" type of anime, then I give the reader my personal rating typically at the end of my reviews. Most that read mine know it to look like this:

avoid, AVOID, AVOID..... something along those lines.

Regardless, I still believe every title is worth viewing and a title that is crap for me, may be the best title ever for another person, so I don't want the reader at the start of my review to see a big fat avoid on the rating and not give the title a chance. Unfortunately, this is the reason why Watch ratings are what dominate my overall review average.

I don't have any problem with an "Avoid" rating because I rarely ever will be the only reviewer of an anime so that my rating will just contribute to some average. I gave an "Avoid" for Gantz that got 4 "Buy" ratings (including Forbin and Dreamer) and 2 "Rent" (Stretch and Jan-Chan) while KBanger1 gave a "Watch". Some anime are just not for everyone. Actually, I don't use "Avoid" for expressing "I want my time back" but to give a strict warning that the show in question is beyond what I consider acceptable in certain aspects (much like what Stretch did here).

If I am to write for people with a similar taste then I feel obliged to use all available rating values but at the same time ignore any economical aspects. Even a reader with my taste who wouldn't have money to buy a single anime (but watch them on TV or streaming sites or YouTube or fansub) would probably want me to differentiate between "Buy", "Rent" and "Watch" animes.

Edited on 07/12/2010 07:39 PM.

Would it be fine if we had a completely revamped rating system? We all use the Avoid - Buy system which may work if it was all domestic titles only. Since this is a complete collaboration of domestic and foreign anime reviews, it would be hard to put a "Buy" rating especially if the anime is hard to obtain. It's probably easier to "Avoid" since most of the stuff we do see are non-licensed originals straight from Japan. Plus, since this is a purely user based review site, would it be more beneficial to have a more linear or simplistic system (5 star, letter grade, etc.) so there not only make the rating system easier but still maintain a running average? I know it would be a hassle to redo the rating system itself so I'd like to suggest adding (as a test) either a new box where a user can manually add their own rating or a new drop down box with a 5 point rating system. IMHO, I think it would be a good move to at least try to move ahead on a new rating system.

I'm not keen on having a number based system... but perhaps a revamped scale of rating words? Suggestions?

The essence of the discussion above was that we don't even agree on what these rating words are meant to express.

• For Jan-Chan, "Buy" means "I'll buy it as soon as it gets an R1 release according to my own financial abilities", i. e. what he would do.
• For me, "Buy" means "This is a good show with high rewatchability value so (assuming money is no obstacle) it is worth buying as opposed to merely renting", i. e. what I assume any anime fan with matching taste to mine should do if possible.
• For Xenoknight, his intention is to simply not use certain values (without giving a reason for this).

• ANN has 11 levels with words (Masterpiece, Excellent, Very Good, Good, Decent, So-So, Not really good, Weak, Bad, Awful, Worst ever) expressing some degree of quality experience about the anime; I decided that my Buy+ / Buy / Buy- / Rent+ / Rent / Rent- / Watch+ / Watch / Watch- / Avoid scale that I use here at CAR is good enough to simply map it 1:1 to the ANN system (splitting my "Avoid" amongst "Awful" and "Worst Ever").
• aniDB's percentage values (in 5% steps via GUI, in 0.1% steps via URL hacking) show a similar distribution to mine, so simply using these values over there works reasonably well.
• For anisearch I need the percentage values to get "spread" more evenly within the [0,100] interval (here at CAR I use scores below 50% for "Avoid", at anisearch it's scores below 20%) so I rescale them by something like "Y = 1.8 * (X-40)".

Edited on 08/12/2011 09:56 PM.