Please refrain from posting animated GIFs, memes, joke videos and so on in discussions other than those in the off topic area.

Dismiss this message to confirm your acceptance of this additional forum term of use.
You must be 16 or over to participate in the Brickset Forum. Please read the announcements and rules before you join.

Set ratings

SMCSMC UKMember Posts: 1,103

I was thinking about all my sets which ones were the best and how I would like to list each set from one theme in order of how I rate them.

I was thinking about how I wound love an IMDB site for Lego and then how brickset has the thing you need most a database.

I was thinking about how I would like the list my sets by how other people rate them so I came on brickset and sure enough you can already do this.

The two main problems are that there are not a lot of ratings for sets by users and when you list them they just come up 5 star for a page then 4 star and so on.

I think it would be great if we could rate sets without doing reviews, rate them out of 10 maybe the stop the number of 5 star ratings and then give the results with decimal places.

It wound also be good to add 'my rating' to the 'sort by' drop down menu so that we could list our favourite sets in order.

I would love after I build a set to come on brickset and rate the set it would add to the fun.

I am sure this has been talked about before but not by me :)

TarDomo

Comments

  • TarDomoTarDomo FinlandMember Posts: 515
    Sounds such an interesting and fun idea for me. :)
    SMC
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 5,753
    It's on the to-do list already, just got to figure out how best to do it...
    SMCDaraghdougtscatwranglergraphite
  • OldfanOldfan Chicagoland, IL, USAMember Posts: 561
    Cheers Huw, the ratings and short comment abilities for individual sets was one of my favorite features of the old LUGnet site.  I'd love to transfer all those thoughts to Brickset eventually...
  • SMCSMC UKMember Posts: 1,103
    Yep thanks Huw I thought it would have been considered already, good to hear it could happen as I think it would be a really fun feature.
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 5,753
    It's *still* on my to-do list :)

    It'll probably have to wait now until the next big development cycle in the autumn/winter.
    dougtsSMC
  • dougtsdougts Oregon, USAMember Posts: 4,086
    can't wait.  I think most people don't do reviews because it's too laborious.  If I could just rate sets 1 through 5 on 4 or 5 different metrics (overall, value, playability, build, etc.), then I would do hundreds of them.  if you get hundreds of people rating each set, then the ratings are going to be pretty meaningful really.

    I know we discussed this idea (a kind of "short review") a couple of years back.  glad to see it is resurfacing again.

    @SMC thanks for bringing this to light again.
    SMCTheLoneTensor
  • dougtsdougts Oregon, USAMember Posts: 4,086
    PS - most well designed rating systems have either 5 or 7 options.  10 doesn't work well (there's a lot of science around this - google it), for multiple reasons: First, humans aren't good at accurately distinguishing between that small of increments in a scale. What's the difference between 7 and 8?  it's largely random and arbitrary. Second, and relatedly, each person's scale is different.  on a 1 to 10 scale, person A will give out 10s like candy, and person B will never give one out, always leaving room for "something that could top the thing I gave a 9 to".  Also, person C's 7 could be the same as person D's 8.  it adds too much noise to the results.  Thirdly, there is no true neutral rating on a 1 to 10 scale.  Making it 0 to 10, with 5 as a true neutral only exacerbates the first point, and people don't think well naturally with 0.

    In the case of LEGO sets, a 5 point system (something like we have now), is probably better than a 7-point system:
    5 Excellent
    4 Above Average
    3 Average
    2 Below Average
    1 Poor

    https://www.surveymonkey.com/mp/likert-scale/
    catwranglerPaperballpark
  • SMCSMC UKMember Posts: 1,103
    Huw said:
    It's *still* on my to-do list :)

    It'll probably have to wait now until the next big development cycle in the autumn/winter.

    Thanks for the update, can I request an edit feature as I would hate my rating to be set in stone.

    dougts said:

    @SMC thanks for bringing this to light again.

    Your welcome but now you have opened up another can of worms and given me flashbacks of when my company refused to rate people out of 5 because managers will always go for the midpoint.

    Although you make good points about each person's scale being different. A 5 point system would leave me giving most sets a 4 on the basis that I bought the set so I think its above average but as you know I don't think anything is as good at it can be, so it cant be a 5.

    So to compromise how about a 5 point system where you can give half marks?

    Or if you want to be clever (and I do) let each member choose there own rating system that is divisible by 5 then divide their score accordingly so their score is out of 5.

    So

    @dougts marks a set 4/5 = 4

    @SMC marks a set 7/10 = 3.5

    @Huw marks a set 43/50  = 4.3

    Simple (kind of)

    I think when it comes to a numbering system for rating Lego sets I would pick one that allows more fun rather than accuracy. The larger the number of options the more you can differentiate between the sets you own. Maybe more like the PA/CA system on football manager, the player rating on fifa or the handicap rating of a racehorse.

    You will always face problems like if people only rate sets they own, you will find say a bad chima set being rated higher than a D2C because only die hard chima fans bought that set and everyone bought the D2C.

    The best measure of a set might be number of people that bought it divided by the number of people that had the opportunity to buy it. But that's not as fun now is it.

  • CCCCCC UKMember Posts: 14,680
    You could even go further than that. You could scale based on all the average scores a particular user gives. So if A consistently votes 5/5 for all sets, then they are over-rating sets. So all their scores should be scaled down. Whereas if B uses the full range, so some less than average (1 or 2), some average (3) and some above average (4-5), then their scores would be less scaled.

    It's funny to see scores like this on eurobricks for the CMF series. Many people vote above average for every series. Which is meaningless, since they clearly have the wrong definition of average* in their mind when they vote.

    *If average is defined over all CMF series only. Of course, if it is average taking into account other lego sets, then they are implying other lego sets are worse overall than CMF.
    SMC
  • SMCSMC UKMember Posts: 1,103

    CCC said:
    You could even go further than that. You could scale based on all the average scores a particular user gives.

    Great idea, is @Huw up to the challenge.

    @CCC I am one of these people, when I see a 1-5 rating I think of it as:

    5 Excellent
    4 Good
    3 OK
    2 Bad
    1 Terrible

    So I give a lot of fours.

    To be fair eurobricks often use a system like the one @dougts talk about:

    5 Excellent
    4 Above Average
    3 Average
    2 Below Average
    1 Poor

    Which is a mix, if you love every set you own a set could be both average and excellent.

    The other thing to consider is people might be comparing their sets with ones they don't own, you don't have to own a set to know you like the minifigure selection better or the look of the final product. The only thing you can't compare is the build experience.

    I still don't like a 1-5 system if you have hundreds of sets you don't just want to put them into 5 groups, you want to be able to decide which ones are truly great not just which ones are top 20%.

  • PaperballparkPaperballpark UKMember Posts: 2,633
    I think a 1-5 system is better than a 1-10 system, but @SMC makes a good point that many people don't want to just put sets into 5 groups such as:

    5 Excellent
    4 Good
    3 OK
    2 Bad
    1 Terrible

    For example, let's take 3 modular sets - Parisian Restaurant, Grand Emporium and Green Grocer. The PR is the best of the three IMHO, then the GG and then the GE. All are at least 'Good', but there's clearly a step between them. I wouldn't rank GG the same as PR, and I wouldn't rank GE the same as GG, but I wouldn't rank any of them as just 'OK'. So what to do?

    Have half-points. Still have the 1-5 system, clearly labeled as above, but have half points inbetween them (unlabeled), so that people have extra options if they feel a set is, for example, better than 'good', but not quite 'excellent'.

    Whilst it's technically almost the same as a 10-point system, the way it'd be done would most likely make people think a bit differently about how they score the sets.

    Just a thought. :)

  • PaperballparkPaperballpark UKMember Posts: 2,633
    edited June 2016
    Regarding the Likert scale, it's interesting, but a thought occurs to me about it.

    I suspect it's mainly useful for measuring customer satisfaction and experiences, rather than rating or ranking something.

    The question "Was your experience of building GG..." is much more suitable for the Likert scale than "Do you think the Green Grocer set is..."

    Asking a question like "Do you think the Green Grocer set is..." is fine in isolation, but if you're then given a chance to rank all the sets you've done that for, you'll quickly find that some of your isolated answers clash when you compare them.

    So for example, if I'd ranked PR as 5, GE as 4 and GG as 5, all at different times, I might then go back and compare them later, and wonder why I ranked GG the same as PR. I might then change GG, and realise that I've ranked it the same as GE. Etc. etc.

    That's fine if you don't plan to let people compare their rankings, but I suspect they'd want to.

    Look at Amazon, which has a 1-5 scale. They ask you what you think of a book in isolation, rather than asking you to rank your books in order.

    Again, just a thought :)
  • MattsWhatMattsWhat Studley, UKMember Posts: 1,639
    edited June 2016
    ^when lots of people rate the set the average will iron out the order nicely, so no need for a more complicated system.

    It's all largely irrelevant anyway, I wouldn't recommend the use of any review on the internet that doesn't have a decent length of words and photos - people are too trigger happy with 0 and 5 (on this scale) and peoples opinions are largely irrelevant in comparison to an expert review (like the user reviews vs. the brickset official reviews for example).  People giving a set 5/5 when they have only built that one, or giving TBBT set 5/5 because they like the TV show* - who wants an average that includes all of that drivel when they are choosing a set?  1 or 2 expert reviews with some pictures or videos is the way forward and brickset already does that well enough.

    If you simply want to get an idea of what the current opinion of sets is, then simply sort the whole of brickset by the sets with the most 'wants'.  That will essentially give you the same order as any rating system will anyway.

    *or to add more examples: the confirmational bias of people reviewing sets they have bought with their own money; people jumping on the band wagon and rating (for example) AoH 1 star without owning it (or 5 stars for #10179 of course); the bias created by the fact the only people using the system like LEGO; the comparison of price to set (or value) being an important factor when the price is different in different countries and different retailers; etc. etc.
    Lots of these issues are removed when I see a review on Amazon.co.uk for example, but even then I (and hopefully most shoppers) actually read the reviews rather than just go "4.5? Must be good".
    cheshirecatcatwranglerdrdavewatford
  • FauchFauch FranceMember Posts: 1,679
    CCC said:
    You could even go further than that. You could scale based on all the average scores a particular user gives. So if A consistently votes 5/5 for all sets, then they are over-rating sets. So all their scores should be scaled down. Whereas if B uses the full range, so some less than average (1 or 2), some average (3) and some above average (4-5), then their scores would be less scaled.
    I almost only give 4 and 5 ratings, because, you know, I try to buy good sets. so am I over-rating if I avoid sets that would be average or under average?
    someone who only buy modulars and rate them all 5 is over-rating? should he give ratings going from 1 to 5?

    now personally, I don't really think of under / above average when rating. I don't even know what the average should be. I think how satisfying is the set and how many flaws it has. it's kind of a gut feeling. if I feel there is nothing bothering me with the model, it is likely to get a 5.
    MattsWhatcatwrangler
  • CCCCCC UKMember Posts: 14,680
    ^ That is the problem of using terms like "average" in a star system. If you use the term average for three stars, then it is confusing when the average (mean) rating across everything is for example 4.2 stars. As everything is well above the pre-defined "average".
    MattDawson
  • cheshirecatcheshirecat Member Posts: 5,319
    I actually think a 100% scale would be the best if picking 1 overall rating. A 1 to 5 rating with or without half marks would work fine if there are multiple ratings such as value, playability, build etc. On a single rating there would be little to no differentiation between most sets. For example, in comparison to the entire LEGO range you would expect all modulars to be 4.5-5 star. And does that really help at all? Which also leads to @MattsWhat point - I would apply zero weight to most ratings as they'd be flawed for so many reasons, not actually owning the set, liking the licence etc.
  • SMCSMC UKMember Posts: 1,103
    I actually think a 100% scale would be the best if picking 1 overall rating.

    I agree some people have hundreds or even thousands of sets and a 100% scale would give them the most opportunity to differentiate between sets. 

    I would be reluctant to have multiple ratings because it would disadvantage certain set, an Architecture set would never be able to achieve top marks (no matter how good it was) as it would score low on playability.

  • CCCCCC UKMember Posts: 14,680
    SMC said:

    I would be reluctant to have multiple ratings because it would disadvantage certain set, an Architecture set would never be able to achieve top marks (no matter how good it was) as it would score low on playability.

    And so it should, because they are not very playable and so they will not be ranked highly in general across all categories. Architecture sets are awful for a child, but better for adults. So if you then allow the user to select the categories via tickboxes that they want to rank sets across (so build, value, playability, minifigs, etc) then they could excluded playability if that is not important to them.
    catwranglerdougts
  • FauchFauch FranceMember Posts: 1,679
    makes me think there is no rating for minifigs. of course not all sets have minifigs, but as we've just said, not all sets offer playability either.
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 5,753
    Thanks for resurrecting this thread.

    Is a 1-5 scale enough or should it be 1-10?

    Consider general usability as well as personal preference when answering, i.e. I don't want to complicate things!
    SMCkiki180703
  • OldfanOldfan Chicagoland, IL, USAMember Posts: 561
    I'd prefer 1-10 ratings.  1-5 results in a big jump when you move from "5-Excellent" to "4-Great", and I'd prefer my ratings to have a little more nuance.  I don't see a need for decimal-fraction ratings regardless of which scale is used, unless it's maybe 0.5 for the 1-5 rating scale (which gives me the same scale as 1-10...).

    I know that the "5-star rating" is really popular on the 'net right now thanks to Amazon, ebay, etc.; but a bar-graphic or simply "7/10" graphic would work just fine for this site I'd imagine, if a 10-star graphic does not display well on the site.

    Would the review ratings be the same scale as the set ratings, and would they automatically count toward the set ratings?

    I would request that the ratings scale does not change once you determine what it should be.  If we all put in ratings now, and the scale changes next year, then what happens to all those old ratings?  Are they ported into the new scale somehow, or would all the ratings suddenly skew toward an unwanted value?  (Admittedly, this is only important if someone is using the ratings averages in order to make a decision, or if you're really OCD about your ratings like I am...)
    catwranglerSMC
  • catwranglercatwrangler Northern IrelandMember Posts: 1,523
    I agree with @Oldfan - I've noticed that any time I try to rate sets out of five, there's a glut of fours and fives, and I'd appreciate a little more granularity. I mean, within that system a four implies near-perfection, yet within the sets I'd rate as fours, there are varying degrees... some will be more fun to build, some will be repetitive builds that nevertheless look great on display, some sets work great both as builds and as parts packs... 

    I feel a 1 - 10 scale could be helpful not just for people choosing what set to buy, but for people looking through their own ratings of their collection if, say, they're trying to liquidate some sets and need more insight into which ones they should keep to enjoy again, or trying to rec sets to friends in more detail than just "if you like Space, these twenty are all Space sets I rated at five stars"...
    SMC
  • dougtsdougts Oregon, USAMember Posts: 4,086
    I think a 1 to 5 scale is the best option, but could see 1 to 7 working also.  1-10 doesn't provide for a true "absolute middle" choice, which is a big problem.  People LIKE having a neutral option

    most well designed rating systems have either 5 or 7 options.  10 doesn't work well (there's a lot of science around this - google it), for multiple reasons:

    First, humans aren't good at accurately distinguishing between that small of increments in a scale. What's the difference between 7 and 8?  results become largely random and arbitrary, and in the end, when collated, (8,9,10) all get counted as equivalent, as do (1,2,3) and (4,5,6,7)

    Second, and relatedly, each person's scale is different.  on a 1 to 10 scale, person A will give out 10s like candy, and person B will never give one out, always leaving room for "something that could top the thing I gave a 9 to".  Also, person C's 6 could be the same as person D's 8.  it adds too much noise to the results.  Thirdly, there is no true neutral rating on a 1 to 10 scale.  Making it 0 to 10, with 5 as a true neutral only exacerbates the first point, and people don't think well naturally with 0.

    https://www.surveymonkey.com/mp/likert-scale/
    catwranglerstlux
  • SMCSMC UKMember Posts: 1,103
    edited October 2016

    Agree with both @Oldfan & @catwrangler. Personal preference would be a 100% scale so for general usability 1-10 would be a good a compromise.

    I do like the aesthetic of the five stars you have added and don't think it would look as good with 10. If you could have a 1-10 system but displayed as five stars with half stars then people wouldn't think of it as a 1-10 system (best of both worlds?).

    Maybe under the stars you could display the numerical value of the sets average rating.

  • dougtsdougts Oregon, USAMember Posts: 4,086
    I think the biggest problem with the current 5 star system is that very few people use the full range.  1 and 2 are almost never used, and 3 very infrequently.  Thus, people are self-selecting into using a 2-star system (4 or 5) and then feel frustrated that they can't distinguish between a "strong 4" and an "weak 4"  I dunno - how about giving one of them a 3 then?
  • SMCSMC UKMember Posts: 1,103

    ^ Because a 1-5 system works like this:

    Very Good

    Good

    OK

    Bad

    Very Bad

    You have brought a Lego set that you liked the look of, that you have an interest in, now rate it. You are not being asked to rate 5 sets top to bottom, you are being asked to rate one set out of 5. So it should be no surprised to find that most are 4 or 5.

  • dougtsdougts Oregon, USAMember Posts: 4,086
    edited October 2016
    Only if you decide to think of it that way. Of course NO lego set is "Very bad" in a general sense related too everything out there.  

    But if you give your rating relative to "other LEGO sets" or "other LEGO sets of this theme or size", then there should indeed be as many 1s and 2s as their are 4s and 5s.

    For example.  If I was rating ANY modular building set, it would have to be a 5, or at worst a 4, in a general sense.  however, If I was rating any particular Modular building RELATIVE TO ALL OTHER MODULAR BUILDINGS, then I would have two 5s, two 4s, four 3s, two 2s and a 1.

    http://www.troll.me/images2/syndrome/everyone-can-be-super-and-when-everyones-super-no-one-will-be.jpg
  • FauchFauch FranceMember Posts: 1,679
    edited October 2016
    I mostly rate between 4 and 5 so a scale from 1 to 10 would be better. or 1 to 11 for very outstanding sets :-p
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 5,753
    A very interesting discussion! I like @dougts POV in his first comment as it pretty much matches my own. People like to choose the middle option, so often in surveys where you don't want them to be able to, you provide an even number of options. Here, though, that is not a requirement and a neutral score is useful. 

    I also thought that if you provide too much granularity people may not be able to decide whether a set is a 7 or an 8 and thus not bother to rate sets at all, or forget how they've applied particular ratings in the past.

    But, I'm gradually coming round to a 1-10 scale, so will try it out on my local PC and see what it looks and works like. If OK, I'll go live Sunday morning.

    Ratings you give are tied to your review, if you've written one, and changing one will change the other. I don't intend to change review scores to 1-10 but of course can just double or halve the number as needed.

    And, should I change from 1-5 to 1-10, I will double all existing scores.
    catwranglerPeteMSMCMooreFX
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 5,753
    Not yet -- I mucked about with other code at the weekend.
    MooreFX
  • davee123davee123 USAMember Posts: 791
    1-5 stars is probably better where an aggregate rank is determined, as the "x/5" rating system is so widely used for things like movies, Amazon reviews, etc.  So, if the intention is to (please-please-please!) use it for aggregated ratings by members, then 1-5 makes sense, although it might be handy to allow half-stars to allow greater granularity.

    1-10 is probably better for personal ratings, because it gives you more options, and nobody cares about it except the person who can see it.  In other words, you don't have to worry about people being intimidated by more options, and thus skipping the rating, because you're never going to share that rating anyway-- it was only for their personal use.  Similarly, you don't have to worry about being accurate, and making sure people can discriminate between a 6 and a 7, because you're never going to use it for anything practical-- it's just whatever that person decides.

    If it's a personal rating, you could even decide to do wacky things like rate each theme on its own scale.  You could (for instance) decide to rate each of the modular buildings in terms of how they compare to OTHER modular buildings.  You can decide, and nobody ever has to care, because it's just for you.

    When it's publicly aggregated, then you have to think about how to combine votes.  For instance, I once pulled apart rankings (I forget what for) based on how frequently people voted for 1s and 2s, etc.  That is, if someone only ever used 3-10, but never assigned any 1s or 2s, I "stretched" their votes out.  Or you could determine the mean and standard deviation, and work out how to balance on "each end" from the middle, and re-tool the middle at (say) a 5 rather than their typical score, which might be a 7 (for instance).

    You could even do some really fancy stuff like determining the "weight" of a person's vote.  If they rarely vote, or if they only vote 4s and 5s, you could lower their rate, compared to someone that more widely distributed and who has rated more sets.  And maybe raise/lower the weight depending on if they own the set.  Or (if you combined with Set Reviews), add extra weight if it's coming from a Review (or even a "helpful" Review, at that, or a person who's done more Reviews).

    Anyway, I love to see stats on this sort of thing, so I'm all for it being publicly aggregated.  It doesn't even have to interfere with the ratings from set reviews, it could be a totally separate rating, or a totally separate set of ratings (sliced in various different ways as above).  As a personal rating, I don't really care as much.  It might be handy occasionally, but I'm not as enthused.

    DaveE
  • SMCSMC UKMember Posts: 1,103
    davee123 said:
    Anyway, I love to see stats on this sort of thing, so I'm all for it being publicly aggregated.  It doesn't even have to interfere with the ratings from set reviews, it could be a totally separate rating, or a totally separate set of ratings (sliced in various different ways as above).  As a personal rating, I don't really care as much.  It might be handy occasionally, but I'm not as enthused.

    I agree with this, I am a lot less likely to rate my sets if it doesn't count towards an overall rating. I also don't see any problem with it being different from set review ratings.

    Huw said:
    Not yet -- I mucked about with other code at the weekend.
     Keep up the good work, you can tell you have been busy, I am reading the capitol review now.
  • 77ncaachamps77ncaachamps Aspiring Time Traveler Stuck in the WestMember Posts: 1,870

    Dave, I don't understand the difference between 1-5 with 1/2 stars and 1-10 with no 1/2 stars. They are practically the same just a different visual meter.

    I'm for 1-10, BTW. I think I can tell the difference between a 6 and an 8. Using a 1-5 scale, a score of 3 is so ambiguous.
  • davee123davee123 USAMember Posts: 791

    Dave, I don't understand the difference between 1-5 with 1/2 stars and 1-10 with no 1/2 stars. They are practically the same just a different visual meter.

    I'm for 1-10, BTW. I think I can tell the difference between a 6 and an 8. Using a 1-5 scale, a score of 3 is so ambiguous.
    Purely aesthetic.  There's a couple things going on there, though.  If (say) BrickSet is seen by LEGO buyers at large (normal moms & dads, etc) as a LEGO set rating system, then a 1-5 scale is more in line with product reviews from most other websites.  A typical rating system is 1-5 stars.

    Furthermore, there's a certain amount of psychological intimidation behind having many, many choices.  If (for example) it were a scale of 1-100,000, sure, it'd be more granular, but it just "feels" like a lot more thinking, and can be off-putting to people, hence discouraging precious, precious data.

    So, the compromise (I think) is a system that lets people do 1-5 stars, but with half-stars as an option.  You still get the granularity of a 1-10 system for all those people (like me) who want to differentiate more finely, but it's not quite so intimidating so as to discourage anyone.

    On the other hand, if the data won't ever be aggregated, and won't be publicly visible to anyone (IE, won't ever be used for moms & dads and the public at large), then I'd just make it useful to those that want it.  There's no fear of discouraging people from using it, because you actually don't WANT their data-- you're not using it for anything, you're just providing bonus functionality.  So in that case, I'd vote for a 1-10 system, since it's slightly more straightforward than a 1-5 score with half-points.

    DaveE
    SMC77ncaachamps
  • plasmodiumplasmodium UKMember Posts: 1,883
    ^ I actually really like your argument for a 5-with-halves system. I couldn't see the point before you put it that way, but that's pretty good.
  • 77ncaachamps77ncaachamps Aspiring Time Traveler Stuck in the WestMember Posts: 1,870
    Thanks, Dave. You pretty much restated things, I believe. But somehow, it was even clearer for me this time.

    lol
  • HushedHushed Member Posts: 11
    Is there a way to reset a rating back to zero? I find when I browse on my phone, I end up hitting the rating button, when I do not intend to. I have ratings for 7 sets, yet I never intended to rate them!
  • HuwHuw Brickset Towers, Hampshire, UKAdministrator Posts: 5,753
    Yes. If you've rated a set 3 then click on the 3rd star.

    Might be easier said than done on a phone, though...
    Hushed
Sign In or Register to comment.
Recent discussions Categories Privacy Policy