Rating distribution


Back to topics list
[post:462#4153]
Devil Doll

06/25/2010 11:03 AM

Reviews: 365
Posts: 1574

Triggered by the shoutbox discussion I made a short check about how I use the four main ratings "Buy", "Rent", "Watch" and "Avoid" in my reviews. My impression was that "Buy" and "Avoid" should be relatively small subsets whereas "Rent" and "Watch" should be the majority. I have my ratings data in a spreadsheet so I can easily make some stats about them.

These stats made me aware that I've been too lenient in the "Buy-" area. So I now raised the minimum score for a "Buy-" from 82% to 83% which made quite a difference (9 of 250 rated anime at that time). Now I like the distribution of my votes much better than before.

The following list contains the number of anime I gave the corresponding rating, with the first number being complete works (series resp. movie), the second number being add-ons (specials/OVAs etc.).

  • Buy+ (100-90%): 10 + 0 = 10 (3,39%)
  • Buy (89-86%): 15 + 1 = 16 (5,42%)
  • Buy- (85-83%): 26 + 5 = 31 (10,51%); all "Buy" = 19,32%
  • Rent+ (82-79%): 27 + 8 = 35 (11,86%)
  • Rent (78-76%): 34 + 7 = 41 (13,90%)
  • Rent- (75-73%): 21 + 11 = 32 (10,85%); all "Rent" = 36,61%
  • Watch+ (72-70%): 24 + 11 = 35 (11,86%)
  • Watch (69-58%): 34 + 20 = 54 (18,31%)
  • Watch- (57-50%): 11 + 8 = 19 (6,44%); all "Watch" = 36,61%
  • Avoid (49-0%): 14 + 8 = 22 (7,6%); all "Avoid" = 7,55%
It might perhaps be a cool additional feature if the CAR software would be able to show these four percentage values on the reviewer's page. (Rebecca, what do you think?)

EDIT: Numbers updated after 300 reviews. I also raised minimum score for a "Buy" from 85% to 86% which now makes a full "Buy" a lot more meaningful and easier to tell apart from some "Buy-" ratings that I gave out of personal sympathy.
(The numbers of ratings add up to only 295 because other sites have a different granularity of anime entries.)

Edited on 08/10/2011 12:56 PM.

[post:462#4160]
Rebecca

06/26/2010 07:14 PM

Reviews: 23
Posts: 771

I was thinking that myself...

[post:462#4161]
Devil Doll

06/26/2010 10:18 PM

Reviews: 365
Posts: 1574

I just took a look at the http://anime.mikomi.org/reviewer.html page. How exactly does the CAR software calculate the "average rating" for Jan-Chan with a value of "Avoid"? (Arithmetic means of all values, including the "unevaluated" ratings?)

[post:462#4193]
Rebecca

07/01/2010 02:10 PM

Reviews: 23
Posts: 771

Yes, it's just a simple mean of the review's. Which worked great before we had "unevaluated" as a category. =p

[post:462#4194]
Jan-Chan

07/01/2010 02:21 PM

Reviews: 599
Posts: 593

Could we perhaps add another rating that might read "don't want to rate" or perhaps "your call on this one". Or even "Unrated".

I have posted a few reviews in which I intentionally left a series unrated (unevaluated) because I had some mixed feeling about the title, which I explain in detail in my review.

Edited on 07/01/2010 03:54 PM.

[post:462#4196]
Devil Doll

07/01/2010 03:36 PM

Reviews: 365
Posts: 1574

So if the software would just do a "sum" over all ratings that are NOT EQUAL "unevaluated", the result would be as expected? (I would still prefer a percentage value to an overall "Watch" or "Rent" though.)

[post:462#4198]
Jan-Chan

07/01/2010 04:01 PM

Reviews: 599
Posts: 593

How about a new rating Currently Under Review and make it the default value/setting for any new review postings?

This would address the issue that many of us have when trying to rate a new series that we are following while it is still airing.

Edited on 07/01/2010 04:04 PM.

[post:462#4199]
Devil Doll

07/01/2010 04:24 PM

Reviews: 365
Posts: 1574

For calculation purposes, "Currently Under Review" and "Unevaluated" are equivalent.

Having two values for them would be beneficial if we had a feature to group reviews by this rating. CAR has a "By Rating" access path but only for total ratings of an anime, not for just one reviewer (where it could easily list all "Currently Under Review" entries for this particular reviewer, thus implementing their "to do" list).

The downside of "Currently Under Review" might be that these reviews can be forgotten and the value might remain forever, then being even semantically equivalent to "Unevaluated".

For the sake of clarity I am all for two different values though.

[post:462#4213]
Rebecca

07/03/2010 04:43 PM

Reviews: 23
Posts: 771

I understand wanting to have a "Couldn't Say" or "No Recommendation" rating, and I'll look at adding that.

Can you explain to me how exactly "Currently Under Review" and "Unevaluated" differ? I read those to mean essentially the same thing. Or do we just want to rename "Unevaluated" to "Currently Under Review" as we'll have a "No Recommendation" rating?

[post:462#4214]
Devil Doll

07/03/2010 04:51 PM

Reviews: 365
Posts: 1574

I remember one understandable candidate for "Unevaluated" here.

Semantically the difference would be to be able to automatically separate between complete and incomplete reviews. This would be a concept not really explored at CAR as this portal gives low priority to numerical scores and thus doesn't sport anything like a list of anime reviews sorted by rating of one reviewer. Still, doing the same separation according to the overall rating (much like http://anime.mikomi.org/by-rating.html) would appear easy to implement and serve as "unevaluated" filter at the same time.

Edited on 07/03/2010 05:02 PM.

[post:462#4218]
Rebecca

07/06/2010 09:04 PM

Reviews: 23
Posts: 771

Ok then, how does "Unevaluated" in that example differ from "No Recommendation"?

I propose that I do this:

  • Rename the existing "Unevaluated" to "No Recommendation"
  • Add a new "Currently Under Review" rating
  • Make the default rating be "Currently Under Review"
  • Ignore both in the calculation of mean ratings for both reviewers and series.
I know there's been some calls for more granularity in ratings, but I'd like to keep them focused on direct practical advice. (And beyond that, they're supposed to be describing what you would recommend yourself do, if you could go back in time and give the advice to yourself before you watched it.)

So what that means is that I don't want to add Buy- or Rent+ ratings. However, if you have suggestions for ratings that can be stated concretely as advice, then I'd be interested in considering them for inclusion.

[post:462#4220]
Devil Doll

07/07/2010 12:49 AM

Reviews: 365
Posts: 1574

The proposal sounds good.

I'm okay with the granularity for ratings of an individual review; it could be better for reviewer & anime averages though (an average of "Watch" or "Rent" for 500 reviews of one reviewer doesn't really give useful information, some decimal or percentage value would be better).

[post:462#4223]
Devil Doll

07/08/2010 12:53 PM

Reviews: 365
Posts: 1574

Thinking a bit more about the current ratings, CAR has:

  • Watch: I'm not willing to pay money to see it, but if it's being shown I'll take the time to watch it.
  • Avoid: If someone puts it on, I'll leave the room.
I'm not sure whether "Watch" is actually a recommendation to watch this show. While it is less than "Rent" (i. e. "invest more than time into this show") it covers both "nice watch but nothing you can't miss" (currently "Watch+") and "waste of time but not really horrible" (currently "Watch-"), the latter not being a reason to "leave the room".

So what about splitting "Watch" into two categories? (Such as "Not recommended" being inserted between "Watch" and "Avoid". Only that "not recommended" not sounds awfully similar to "no recommendation"... perhaps "Don't bother" would be easier to understand?)

[post:462#4225]
Xenoknight

07/08/2010 02:16 PM

Reviews: 244
Posts: 514

I don't think there is a need to make or change the system at all. Regardless of the site's meanings, I use the watch rating. If it has to be changed, then leave watch as a positive rating and make the not recommended rating separate. If I use the watch rating, it usually means not worth the money, but certainly worth the time.

For me, there is no real avoid rating as every title should be seen for your own take on it. I don't use Rent either because if I open my wallet to buy something, I'm not giving it back. So I only use watch and Buy. If we can split watch (positive) and (negative), it would give readers a better understanding overall.

Here's how I see it:

Buy - best rating. A first day purchase.
Rent - don't care for and will never use.
Watch (positive meaning) - should stay. we need a buffer for good anime that is worth the time spent watching.
new rating - a negative representation. Something higher than avoid.
Avoid - should stay but I'll never use.

[post:462#4227]
Stretch

07/09/2010 10:47 AM

Reviews: 1918
Posts: 1313

If 'Buy' means 'worth spending a good deal of money on', and 'Rent' means 'worth spending a little money on', then different versions of 'Watch' might mean 'worth spending certain amounts of concentration/brainpower on'. Well, I thought I was on to something for a minute, but maybe this doesn't make sense...

[post:462#4228]
Xenoknight

07/10/2010 09:36 AM

Reviews: 244
Posts: 514

That's exactly it, Stretch. It makes perfect sense. Watch means "worth the time spent". We just need to split watch into two divisions. One left the way it is and one being a more negative term, but still better than avoid.

[post:462#4229]
Stretch

07/10/2010 10:54 AM

Reviews: 1918
Posts: 1313

One thing I often notice (and perhaps something that might divide 'Watch' shows into two categories) is that some shows are so confusing and overcomplicated that I decide that I won't even try to fully understand the entire story. This happens when the impression I get is that the overall quality isn't great enough to justify the concentration and close attention that would be required to catch every detail and completely understand the plot. Still, the show is worth watching. I'll just watch largely for the jokes and action without trying to tie each and every plot strand together. You might say I sort of treat the episodes as filler episodes even though they weren't meant to be that way. Maybe this would be a definition of Watch-: Worth watching, if it can be had for free, but not worth taking the trouble to completely understand it.

One more thing: A Watch- show would most certainly only be watched once, and probably soon forgotten.

Edited on 07/10/2010 11:00 AM.

[post:462#4230]
Devil Doll

07/10/2010 12:52 PM

Reviews: 365
Posts: 1574

As long as we have rating categories that some reviewers will never use there's something not right either in the concept of those categories or in the way they're seen by the reviewers. How should any visitor compare a "Rent" rating from some other reviewer against a rating from a reviewer who doesn't use the "Rent" category? How can the averages calculation lead to any meaningful results when some reviewers understand certain categories different than others?

[post:462#4231]
Stretch

07/10/2010 01:31 PM

Reviews: 1918
Posts: 1313

And what about the differences between reviewers who have money to burn compared to those who are just getting by? Isn't a one-word rating nothing more than an extremely quick summary? If you want to be certain that you agree, you need to read the review in it's entirety (and even that might not make clear why a particular rating was chosen).

[post:462#4232]
Devil Doll

07/11/2010 12:56 AM

Reviews: 365
Posts: 1574

A "Buy" rating doesn't imply that everyone should buy this anime regardless of their taste and economic abilities. (Or would you buy every hardcore hentai just because it's rated as "Buy"?) But it should imply this if neither taste nor economic abilities are an obstacle.

If you don't address a "standard audience" because you want to take into consideration the economic potential of every single visitor here then why don't you give different ratings of the same anime for every single visitor? This wouldn't be very practical.

A rating should describe the quality of the anime - and thus must be independent of the visitor's particular abilities. It's not the visitor who's rated.

[post:462#4233]
Stretch

07/11/2010 10:18 AM

Reviews: 1918
Posts: 1313

I have always taken the approach that I will rate something a 'Buy' if I would buy it--not because I think the average viewer would or should do the same. I guess the idea was that if, having read one or more of my reviews the reader thinks that I seem like somebody with tastes similar to his- or her own, they might want to buy the show as well. I sort of assumed that this was the way everybody was reviewing things... have I been doing something wrong?

[post:462#4234]
Rebecca

07/11/2010 05:48 PM

Reviews: 23
Posts: 771

@Xenoknight in 4225 wrote: Avoid - should stay but I'll never use.

Really? You've never watched something and gone "wow, I wish I could have those 30 minutes back"? I mean, that doesn't happen every day, but sometimes I do see something so insipid or poorly produced that I really wish I hadn't spent my time on it.

Remember, it's as Stretch says, it's supposed to be what would you do, not what would some theoretical average reader do. They're not (supposed to be) exactly recommendations, but rather, summaries of how you feel about it for yourself. The theory being that people can find reviewers who feel the same as them and make decisions based on that.

The other way we could go, is the Roger Ebert method, where he gives thumbs up/down on the basis of someone who likes that genre. So even if it's not a genre he enjoys, if he thinks fans of the genre would like it, he'll give it a thumbs up. I don't like this approach as much because I think our ability to get in the head of someone who likes things we hate is dubious at best. =p

I do think it sounds like we need something between Watch and Avoid. It always seemed like a bit steep of a jump there. We need, like, "Watch" and "Meh". But I think Devil Doll's "Don't Bother" is probably a better way of putting it. Not worth your attention, but not actively awful.

[post:462#4236]
Jan-Chan

07/11/2010 06:08 PM

Reviews: 599
Posts: 593

[post:462#4237]
Jan-Chan

07/11/2010 06:16 PM

Reviews: 599
Posts: 593

Don't Bother works for me.... How about Waste of My Time or Meh - Waste of Time

Edited on 07/11/2010 06:18 PM.

[post:462#4238]
Xenoknight

07/11/2010 07:49 PM

Reviews: 244
Posts: 514

I agree with you completely, Rebecca.

If I do feel it to be an "avoid" type of anime, then I give the reader my personal rating typically at the end of my reviews. Most that read mine know it to look like this:

avoid, AVOID, AVOID..... something along those lines.

Regardless, I still believe every title is worth viewing and a title that is crap for me, may be the best title ever for another person, so I don't want the reader at the start of my review to see a big fat avoid on the rating and not give the title a chance. Unfortunately, this is the reason why Watch ratings are what dominate my overall review average.

[post:462#4239]
Devil Doll

07/11/2010 09:55 PM

Reviews: 365
Posts: 1574

I don't have any problem with an "Avoid" rating because I rarely ever will be the only reviewer of an anime so that my rating will just contribute to some average. I gave an "Avoid" for Gantz that got 4 "Buy" ratings (including Forbin and Dreamer) and 2 "Rent" (Stretch and Jan-Chan) while KBanger1 gave a "Watch". Some anime are just not for everyone. Actually, I don't use "Avoid" for expressing "I want my time back" but to give a strict warning that the show in question is beyond what I consider acceptable in certain aspects (much like what Stretch did here).

If I am to write for people with a similar taste then I feel obliged to use all available rating values but at the same time ignore any economical aspects. Even a reader with my taste who wouldn't have money to buy a single anime (but watch them on TV or streaming sites or YouTube or fansub) would probably want me to differentiate between "Buy", "Rent" and "Watch" animes.

Edited on 07/12/2010 07:39 PM.

[post:462#4266]
KBanger1

07/21/2010 11:06 PM

Reviews: 118
Posts: 70

Would it be fine if we had a completely revamped rating system? We all use the Avoid - Buy system which may work if it was all domestic titles only. Since this is a complete collaboration of domestic and foreign anime reviews, it would be hard to put a "Buy" rating especially if the anime is hard to obtain. It's probably easier to "Avoid" since most of the stuff we do see are non-licensed originals straight from Japan. Plus, since this is a purely user based review site, would it be more beneficial to have a more linear or simplistic system (5 star, letter grade, etc.) so there not only make the rating system easier but still maintain a running average? I know it would be a hassle to redo the rating system itself so I'd like to suggest adding (as a test) either a new box where a user can manually add their own rating or a new drop down box with a 5 point rating system. IMHO, I think it would be a good move to at least try to move ahead on a new rating system.

[post:462#5333]
Rebecca

08/10/2011 06:57 PM

Reviews: 23
Posts: 771

I'm not keen on having a number based system... but perhaps a revamped scale of rating words? Suggestions?

[post:462#5334]
Devil Doll

08/11/2011 12:29 AM

Reviews: 365
Posts: 1574

The essence of the discussion above was that we don't even agree on what these rating words are meant to express.

  • For Jan-Chan, "Buy" means "I'll buy it as soon as it gets an R1 release according to my own financial abilities", i. e. what he would do.
  • For me, "Buy" means "This is a good show with high rewatchability value so (assuming money is no obstacle) it is worth buying as opposed to merely renting", i. e. what I assume any anime fan with matching taste to mine should do if possible.
  • For Xenoknight, his intention is to simply not use certain values (without giving a reason for this).
My idea of a rating (still) is to rate the anime, not the audience. (And even more so, I would never want to comment on the actual price of a release, most notably because this price will vary in different countries. Am I to rate American DVD shops, being no American to begin with, or am I to rate anime?)
If I were to take the audience's financial abilities into account I could give no rating at all because it would always be wrong for some readers. So I use "Buy" to express "significantly better than Rent, regardless whether your 'effort' to get this show is money or time or whatever". Imagine me to get poorer or richer (for whatever reason completely unrelated to the anime world); am I then to edit all my ratings from 10 years back, simply to reflect my new economic capabilities? And whenever an anime gets licensed in the U.S. and a "Buy" suddenly becomes a valid choice in real life, am I again to change my rating here?

I have my ratings (not my reviews) also available in ANN (which uses a scale of 11 values), aniDB (which uses some percentage scale) and anisearch (which uses a different percentage scale), the idea being that I want to contribute to the average scores on these sites (which makes little sense on CAR when there are merely four possible values and no option to reasonably sort anime by these values). Still, the numerical ratings here at CAR are my source; I have mappings to each of the three other sites that derive the value that I assume appropriate for their system. So whatever happens here, I'll still internally use my percentage scores, and if necessary, map them (via an OpenCalc spreadsheet) to whatever system the site in question is using. Which means: Do as you please, I'll be able to cope with anything. It's just that a mere 4 levels of value aren't sufficient for me so I'll certainly create some "+" or "-" variant of them regardless how you name them. (Triple A anime, anyone? ;-)

The reason why I love number-based systems is exactly what I wrote above - I can map them to most anything:
  • ANN has 11 levels with words (Masterpiece, Excellent, Very Good, Good, Decent, So-So, Not really good, Weak, Bad, Awful, Worst ever) expressing some degree of quality experience about the anime; I decided that my Buy+ / Buy / Buy- / Rent+ / Rent / Rent- / Watch+ / Watch / Watch- / Avoid scale that I use here at CAR is good enough to simply map it 1:1 to the ANN system (splitting my "Avoid" amongst "Awful" and "Worst Ever").
  • aniDB's percentage values (in 5% steps via GUI, in 0.1% steps via URL hacking) show a similar distribution to mine, so simply using these values over there works reasonably well.
  • For anisearch I need the percentage values to get "spread" more evenly within the [0,100] interval (here at CAR I use scores below 50% for "Avoid", at anisearch it's scores below 20%) so I rescale them by something like "Y = 1.8 * (X-40)".
anisearch uses two layers of ratings in parallel, meaning that every 10% interval has a "name" (which sounds much like the ANN values), and the users are free to use the full percentage values or limit themselves to these intervals. But again, the "name" layer is simply a mapping of the percentage values. (Their system is currently under review as well - I expect something like "0 to 10 stars" to be implemented, i. e. using the 11 levels of ANN but without names for them. Which would not change anything significant for me, I just have to round() my scores to their "buckets", thus making my ratings less precise - and less sortable.)

Why do I want these values to be sortable? Because I consider an anime site a tool for recommending anime. My idea of the visitor coping with 2000+ anime reviews at CAR is that they will be trying to find quality anime according to their taste. How can the site respond to this request? By filtering by taste and sorting by quality. Filtering allows to include/exclude categories (thus I love the cascading category filter of CAR, and applying it to a reviewer even allows to compute Ggultra2764's drama movies, just not compute his top choices from the resulting 41 anime which would then be this user's auto-generated and maintenance-free recommendation list for the given category query); the result currently is a set of anime.
Now if the result were a list of anime sorted by rating (instead of sorting by name) then each and every request for anime of a certain group of keywords (or to put it more technically: Each result of a category query) could be offered in the form of a recommendation list by merely giving the URL with the filter query and the sorted result. (anisearch, being a community with hundreds of active users and frequent forum postings like "Please tell me a good school romance anime", will implement this filter/sort feature and then hopefully each forum question can be responded to by an URL, thus directing the audience to the tool and encourage them to formulate their own queries.)

The core idea of all the above would be to design a rating system to the benefit of the assumed visitors' behavior by offering an access path to recommendation lists. Which certainly will not the the only way of looking for anime, and perhaps not even the most likely one. Visitors may well ask "I love anime X, please show me other anime that are similar to X", which would then call for a specification of the term "similarity". The category filter is a step into this direction as well but here at CAR I have already seen a better method: The "Forbin Cube". In this approach, quality (and rating) is completely out of the equation, it's all about themes and elements. And as long as CAR doesn't have any similarity tool I'm stuck with manually maintaining my lists of links to similar anime within most of my reviews.

EDIT (Afterthought): Currently, the "Overall" rating and the six individual ratings (Art, Animation, Character Design, Music, Series Story, Episode Story) are completely unrelated. And I'm not even suggesting to change this. Instead, there might be an additional numeric score for people who want to express such a score.
If I were to write my list of wishes for Christmas then it would contain a feature at CAR to automatically compute the overall score from these individual scores. For me, the formula is: (Art + Animation + 3 * Character Design + 2 * Music + 2 * Series Story + Episode Story)/100"; for other reviewers the formula might be different (if they use any formula in the first place).
So I would suggest to extend the "reviewer" profile by a 6-tuple of integers for these individual scores which in my case would then be set as (1,1,3,2,2,1), or anything semantically compatible to this (such as multiplying everything by 10 in order to get even finer granularity of scores). And in order to make use of these data, an anime review would then show an "overall score" as percentage value computed by this formula if and only if all six individual ratings have been set to a non-empty value, meaning: Reviewers who don't set the individual ratings can safely ignore everything I just wrote, and others would get a more precise sortability feature for the anime they reviewed (plus the Category filter could then show its result list ordered by the pair of (percentage score, overall rating) instead of the current alphabetical order, at least optionally i. e. using a check-box to enable/disable this order).

Edited on 08/12/2011 09:56 PM.

Reply to this topic Start a new topic
Back to topics list

Community Anime Reviews

anime mikomi org