I'd like to make a couple comments.First, I miss the ratings. It helped to further differentiated you from other home theater publications. And the ratings of individual products were the very first thing I looked at when I opened your magazine.Second, if you do bring it back, you should consider a formula system for value. I realize that this was done for the overall score, but a formula could definately be used to tabulate value as well. If you keep a databases of the products evaluated that maintains their scores and prices, this wouldn't be difficult at all. It would also help to make it somewhat more objective.Third, having a tiered system was STUPID. Having a three different systems rate a 92 in performance even though none were equal to the other since they were in three different classes completely ruined the rating scale. It was confusing, senseless, and pointless. If you do bring back the rating scale, please just lump everything together.4th, I am no longer a subscri
- REVIEWS
Displays Electronics 
Speakers Sources Other Gear Software - HOW TO
How To Buy How To Use Tech 101
The Great HT Ratings Debate
First and foremost, with complete support of HT’s staff I dropped the ratings because I firmly believed that the ratings system results were artificially high, allowing mediocre components to hide behind ratings that looked good numerically. As I noted in the Letters section of the July issue, of the 160+ components reviewed and rated from January 2007 through April of 2008, only a handful received overall ratings below 84 pts, indicating below average. A whopping 75% of the components were rated at 90 pts or greater, and the average overall rating was 91 pts.
For those of you still insisting that advertiser pressure was the reason behind this move, please read the above again. Manufacturers were not complaining about their ratings. At best the ratings showed some distinction between better and best. But in every instance in which a product was rated as truly exceptional that was abundantly clear reading the review.
And I don’t think this was the fault of the reviewers not being tough enough, but faulty architecture in the system and the way the performance descriptors corresponded to the numbers in place. While the ratings scale ostensibly went to 100, it wasn’t a full 100-point scale. It was really a 74-100 scale, which bunched up a lot of scores at 90 points. It was more like having something that looks like a full 5 point scale, but was really a 3-5 point scale, with of the ratings at 4.5 points and above!
I think some of these issues echo something a reader said in lauding the ratings removal- if you have to explain it every other month then something’s wrong. Frankly, it’s hard to come up with a single ratings system that really works without more explanation than necessary, and then explanations to follow explanations.
Also, while the numbers lent an air of objectivity, they were every bit as subjective as the written review. And with different reviewers, the scale was never really the same across the board as hard numbers inherently imply. And if it’s not, then what is the real value in the numbers? Don’t you have to rely on the review text and the reviewers’ personality to discern these nuances anyway?
Another factor is that with HT’s current design (which will be made over with our October issue) the graphics supporting the ratings used a lot of real estate on the page. So much that a 3 page review was limited to as little as 1400 words. That’s not a lot of space to offer meaningful analysis today’s complex products. Eliminating the ratings allowed us an immediate increase in word count over 25%. So, I saw removing the ratings as mothballing a flawed system, and providing more space for analysis and more authoritative reviews. I feel strongly that the strength of this magazine over its competitors is the knowledge and experience of our writers and their ability to convey in-depth opinion and analysis on the printed page. And I thought that removing the ratings would make this more apparent. But some of you clearly don’t agree.
Now, one significant new feature in HT that I want to ping you on debuts with the August issue, hitting newsstands and subscribers right about now. Every issue of HT will feature two pages that delineate our Top Picks in every product category, broken down by Entry Level, Midrange and High End price points. In every issue the products reviewed that make the grade will appear in Top Picks, and starting in October a Top Picks graphic will be on the opening page of the review, showing you at a glance that the product made the grade. We’re being very selective about these Picks, so they’re not just a list of what we’ve reviewed. And in October, a set of highlights will also be on the opening page of each review, telling you in a capsule what’s hot and what’s not about each product.
So, my question is, do you acknowledge what’s written above and agree? Are these new and upcoming features enough to let you say goodbye to the ratings? Or, do you feel we need to come up with a new ratings system? If a new system is desired, how do you make it meaningful and accurate given the problems I’ve brought to light above?
As I said in my first prologue, Home Theater really is your magazine. Let me know what you think.
- Log in or register to post comments
COMMENTS
Although the rating system was a marginally useful scoring mechanism for comparison reviews in the same article, I question its value over time. As well, a rating scale converging on a narrow band close to 90 points is a waste of effort. I always found myself gravitating to the narrative to weed out the weaker component. I have to disagree with Javan about keeping historical data. Home theater technologies are evolving at such a rapid pace, any attempt to compare ratings from a 2 year old projector to one currently available is simply not practical. Shane makes a good point about space. I would rather see textual interpretation from an experienced and knowledgeable reviewer about features, abilities, and flaws, that I can use as a litmus test to include or discount potential candidates for my short list of equipment. Drop the ratings and lets try a new approach.
Steve and Javan hit on some of the points I didn't even flesh out as fully. At this point in time, even a year after something is out, the scale is re-defined. And frankly, there isn't much objectivity to it, nor can there be. Not only that, with multiple reviewers, the scale simply wasn't consistent by nature. In addition, no weight whatsoever was ever given to the measured performance of products on the test bench. And really, how could it be?
Shane, Your feelings on the rating system mirror mine 100%. As a 10+ year subscriber to all the AV magazines, I had little or no confidence in the ratings, and I found it misleading to have a $500 unit rated as highly as a $5000. pre-pro. It only served the reader that didn't read the whole article, or to reassure the recent purchaser of new equipment. Over the last ten years, it also seemed to be an upward moving standard. The best standard would be to have the state of the art at top, but even then, you can't compare the best from 5 years ago to today. Therefore is is too arbitrary.
And BTW, I have gotten more than a few responses from people who admitted to either not reading the reviews, or only reading them if the ratings looked good. It's hard to get my head around tailoring our content around people who say straight up that they don't actually want to read the magazine!
In my experience it is hard to quantify a number to performance. It usually rewards the mediocre for having one more audio input than another. Having a review that makes you read what the review finds is much more desirable. Numbers are easier to fudge and if the metrics are known it is easy to design for the metrics, then you get an inferior product.
Right on Shane! I agree 100%. I can understand why some people like a number at the end of the review, it makes it easier to flip to the highest ratings and only read those reviews, or worse, just look at the number! They don't serve to clarify anything! For a long time I considered HT mag to be a "poor man's Stereophile", harsh I know, but one of the biggest distinctions between the two publications was the numbers system in HT. Frankly, it dumbed the magazine down. I personally appreciate your efforts to improve the magazine, and I say stick to your guns. If people are still complaining a year from now, then re-evaluate the situation then. While I'm glad the scores are gone, I think a ongoing awards section, or recommended components section would be a nice middle ground. You could then give the most elevated products the attention they deserve, and help the consumers narrow their choices down so they can make the best decision.
When forced to express yourself with numbers, the only way to signal improvement is with bigger numbers. In the end, the rating system seems to have suffered, not expectantly, from subconscious grade inflation. This is best explained by Lake Effect. Specifically, the Lake Wobegon effect, where "all the women are strong, all the men are good looking, and all the children are above average."
Eric- My understanding is that it's been over ten years, two or three acquisitions and I don't even know how many publishers since you worked for Home Theater. Things have changed. I would not presume to speak of how things operated when you were here, since I was not here. The fact that you don't behave with similar restraint doesn't lend credence to the idea that I've overestimated my intelligence.These are the facts: Since I took over, not a single advertiser or manufacturer has complained about the ratings, nor were any complaints passed on from the previous regime. The decision to remove the ratings was made entirely within the editorial staff, with no input or suggestion from the publishing group or anyone associated with ad sales, or any other part of this company. No one on the editorial staff of this magazine was in favor of keeping the ratings system.to be continued...
Going further, no one in our publishing group was consulted on this decision, nor was anyone aware this change was even made until the blue lines came out, including the Senior VP I report to.Those in the publishing group who have spoken to me about this were not in agreement with the decision and would probably bring the ratings back if the choice was theirs. I've told them to give some time for the other format changes, the Top Picks, and some other wrinkles to take hold and see if the ratings are still missed and then re-evaluate.I have no issue with Eric or anyone else disagreeing with this decision and making suggestions. I invited that. But whether it's someone off the street or a former employee, insinuations and false accusations about motives are out of bounds. Especially when they're simply false.PS- I am a wine enthusiast with many friends in that industry. Wine Spectator's ratings are a joke to anyone serious about the hobby.
Shane,I've told you this privately and I'll share it in public. I cancelled my membership to Home Theater about two years ago because it had gone so far down hill. The last 3 issues have been outstanding, with or without the ratings. Keep up the good work!Regards,David
I agree with the removal of the ratings. I think they encouraged too may people to simply compare products by their numerical rating, rather than which would better suit their needs and systems. You can never say that product x is better than product y because x scored a 93 and y scored a 92. Odds are, they're both good products, but you need to read the whole review to understand the subtler points. Does it sounds bright? How will that match with your speakers? Does a pre-pro have the necessary I/O for your system? Etc. And, as Shane pointed out, there is going to be some variance in each reviewer's absolute point scale. That alone makes it impossible to compare products based on a rating.
I agree with the current editor in eliminating the ratings. Keeping them invites comparisons and, worse, comparisons over time, neither of which the subjective ratings can provide accurately, reliably, or validly. If readers want a quick review, the editors can write capsule reviews-- concise language in a callout box next to the full review. But ditch the meaningless numbers.
I liked the rating system and will add my 2 grains to Shane's issues. One message that may come across is that I believe your readers are more "educated" and "smarter" and can understand the ratings for what they were.1. Artificially High: As a 4 year old reader I knew the limitations of the ratings. They helped me COMPARE, products in the same category, it did not really matter what the scale was.2. Architecture: It felt esoteric and could be definitely improved. You do not have to invent the wheel as there are many systems out there that seem to work. What ever the system, it will always have limitations that readers have to recognize.3. Subjectivity: Again, we avid readers knew the personality of the reviewer, his/her likes and dislikes. We read the whole article and past reviews. But the number still helped. I know what a number from Mark compared to one from Geoff. and by the way, also with comments from reviewers from other publications.4. Much more, ran out of
What wrong with Comparing?. I read reviews to keep up with technology and brands and also compare who is doing what.As "educated" readers most of us know that a rating of 2 year old product cannot compare with a recent product. If someone is interested in a product and rating I am sure they are most likely to read the whole review. The rating helps me to understand better what is written about the product.
I really miss the rating system, though I agree it had its flaws. Though I didn't realize the average score was a 91, after years of reading the magazine it was obvious that anything with a sub-90 score had received a less-than-stellar review. Though I, and no-doubt many other readers knew that, the inexperienced readers probably didn't. I' d like to see a new rating system, one that is as objective as possible, with a clear scale published along with it each month. Also, the new section in the back of the magazine with the list of recommended products is great, but I think it could be better if it were accompanied by an overall score for each product. Thanks for letting me put in my 2 cents.
Will- I believe the substance and analysis is in the reviews, not the numbers.Kurt- we are publishing as much objective data as possible in the measurements, and even expanding that kind of material. As an example, we're working steadily toward standardizing some of the video processing performance data along the line of what we did in the June AVR video processing feature, and we're moving to include those tables in all of our AVR and surround processor reviews. And even those processing tests require human interpretation when looking at the test patterns to determine what's Excellent, Borderline or Fail.Unless something measures demonstrably out of whack, the objective measurements only tell part of a component's story. The heart of all ratings systems I know of are the reviewer's subjective analysis and I don't know a way around that.
Man, I miss the ratings. I am a loyal reader and suscriber. The ratings were the main reason I preferred your mag over the rest. It does seem to me that in a political way you caved to advertisers. That may not be the case, but in my mind thats the way i see it.... I will continue to suscribe but the anticipation i used to have for your mag is gone ! Thanks for doing a OK job. Hows that for taking the high road. It Sucks you guys lost the balls to tell it straight like you used to ! That was not so hard, was it ? TY
Frankly, until Shane took over the reigns of Home Theater, the magazine had become unreadable IMO. It's reviews were superficial and lacked focus. The past 4 issues have been very well done and I expect they will get better as time goes along. Keep up the good work Shane!
I agree with you 100%, Shane. I found the ratings empty and only took up space that could be better used by a subjective reviewer's description. IMO, to compare a product using a number told me nothing IF my tastes weren't comparable to the reviewer's. There are no 'perfect' products out there (not suggesting that a ratings system said there were) so I must accept compromise and realize that one piece of equipment will do one thing better than another, yet, that other piece excels at something else. It's the reviewer's written assessment that can only tell me that. Personally, I'd like to see MORE space allowed for a written review. Keep up the good work!
The ratings were the main reason I subscribed many years ago. Because of their removal, I have decided not to renew my subscription. I agree that the ratings have sufferred over the years from "inflation" for fear of upsetting advertisers. What you really needed to do was institute an objective, scientific ratings system like Consumer Reports where like products are grouped together and rated within that category. And the ratings need to be kept updated as more superior products enter the market. Something rated a 10 today, might slip in six months as the next model is released. The web is an excellent place to keep this kind of dynamic system updated. Don't eliminate a great thing like the ratings system when it gets broken due to ratings inflation, but refine it to make it even better. It's the one outstanding feature that makes your product unique from the competition.
I still miss the ratings! Obviously many people are going to disagree on this subject, but it did define your magazine. Sound & Vision has the same thing your are attempting to do with their Sound & Visions Best pages. Why would you duplicate their magazine? I already subscripe to it, I dont need another one. I also dont want to read 25% more text to know whether or not a item is worthy of my cash. This is where I think the misunderstanding is, not everyone wants to read pages and pages to know about a product. I'm lazy, I read my mag to know whats coming out in the market and dream about what next equipment I'm going to buy. Once I see a good score, I will probably read the whole article. But if Im not shopping for a LCD TV, I'm probably not going to read the whole article. I'm a casual reader I guess you could call me. Right now, when it comes to which mag I will keep between S&V & HT, S&V has my vote. I hope your planned improvements make me want to keep both.
I encourage HT to bring back a ratings system back. I think the magazine may be "over analyzing" the issue. The goal of a ratings system should be simple: to allow the reader to quickly understand the reviewer's opinion of the product. The ratings system acts as a complement to, and not a replacement for, the substantive review. Sure, there are issues and subjectivity involved, but is that not true of the longer (more substantive) review as well? I would suggest a simplified system with a 1 to 10 "overall" rating (or 1 to 100). There is no need to share with the readers different "sub scores" (e.g., price, build, quality, etc.). These are, of necessity, part of the "overall" score. Again, I believe the readers completely understand that a ratings system is subjective and far from perfect (just like the reviews themselves). Yet, just like the reviews, they provide the reader with useful information, but in a simplified, "quicker" format than the
Shane,The magazine looks outstanding, possibly, no, definitely, the best I've ever seen it. The newest issue is just superb, and accomplishes exactly what I've wanted to see: MORE REVIEWS. While i initially liked the numerical ratings, i realize now that they're really not needed--just read the damn review! You'll learn more. i think the numbers are for the ones that simply dont want to delve deeply into the reviee. Keep up the great work, and ignore that cranky ex-editor guy. His crappy attitude is exactly why he's an EX- editor.
I just got my October issue and I like the new format (not too sure about your website yet it seems -- er cluttered) When the ratings first disappeared I was a bit taken aback, but I had already started to notice (only being a subscriber and reading HT for about a year) that all the ratings were too close together and seemingly above 90 consistently, which spoiled their usefulness for me. What I like in most automobile reviews is that they list comparable products. So rather than rating products, give three (max) close competitors. This would be really useful to me as a reader and consumer to look at the technical specs and reviews for these products and determine what might work best for my application. I like a lot of you writers, but I think their needs and lifestyle differ from mine. Ultimately I need to make the decision as to what will work best for me. All I ask is that HT help me by continuing to write good, in-depth reviews and comparators would be invaluable in this process. Thanks!
I think a new rating sistem is strongly required.Look at audio.de and stereo.de for example They have 2 ratings : 1 for quality (audio.de reviewed speakers from 46 points to 108 points) and a value rating (1 to 5). I really like that a lot! The quality ratting should never not deppend on price and should not have a maxim value. Thus we will be able to compare a 10 year old product with a new one. Let me give an example : let's say A/V recievers we reviewed 3 years ago got quality points from 40 to 110. Now however high definition audio matter a lot so we give every new reciever up to maximum 10 points for this functionality and so now the maximum is 120.
I really had given up and thought I had decided not to write again because it seems poitless, but here I am again. I DO like the new look, great job! But 6 of 7 of the products reviewed are TOP PICKS! Might as well give them all 90s scores! I like one of the posts above suggesting reviews with at least a few other components comparing them against eachother. Also, to my amazement, Atlantic Technology speakers, floor model, were not in the top products section but they show up being the Rave Review! How are they not in the Top Products section? Why dont you do a reader survey? See how many readers want ratings and how they would prefer them. Makes sense to me.
There should be a rating system that focuses on objective points. Value judgement numbers are useless and are generally purely subjective. An executive editor may score an expensive pre-pro very highly on value while an average reader cannot afford 1/4 the price for this component-useless. When talking about inflation, why limit the top scores at all? If last year's model scores a 90 on video processing, this year's model could score 107. Or you could implement individual criteria rankings, perhaps visually expressing them as lines with the newer, better products in front of the older. This would allow people to focus on what is important to them. These are just a few ideas off the top of my head; if the editors thought about it for a while, I feel confident that they could achieve something that curbs misleading information while retaining the useful aspects of a rating system. I also agree that top picks and recommended components are not an ideal substitution.
Shane, What's that they say about the customer? The customer is always... misinformed? The customer... just needs a little time to get used to it? Mmmm... Maybe, the customer... just needs a little more convincing? No... none of those sound quite right.Haven't your customers spoken? I can appreciate your desire to keep your artistic integrity, but maybe you should just bring the ratings back.
I could live without the ratings..but in return I'd like "Home Theater" to spend some time reviewing something other than the same 3 things every issues seams to focus on. Receivers, DVD/Blu-Ray players and flat panel TV's. How about things like projector screens, HT seating, HDMI switchers/cables, Sound absorption panels, star ceilings, concessions, universal remotes ETC. The sound and video are of course important to the home theater experience but it's not the only thing that makes it special. You seem to be missing a very large part of the experience that most people will remember long after they forget that my sub can go down to 25hz.
Shane,I don't miss the old rating system; but if you feel compelled to restore something, then make it "stars," similar to what you have with Blu-ray ratings that HT reviews. I love the new look of the magazine -- really compliments the great writing, pics, and content. All excellent! Bravo!
I always felt the rating system was wacked.I mean how does a decent inexpensive receiver get the same score as a full blown,top of the line pre-pro and amp?How is it that a sound bar can get an equal rating as a $10k speaker system?The rating system was more for the manufacturers than the readers.It always left me confused but I bet the marketers saw a 91 rating on a good,(not great)component and rubbed their hands in glee.I believe a rating system is necessary... ratings that will inform the readers and leave no doubt that x receiver is a fantastic bargain but in no way is the equal of a Sunfire Grand,a Pioneer 5000 series or a Yamaha Z series receiver.Shane, give us an accurate rating system!The changes you've brought about so far have been on the money.
At first, I was disappointed in your decision to drop your numerical ratings system. It seemed a good way to help quantify a very subjective arena - and sometimes verbage is not the best way to weed out the overall performance quality of a given product. But your solution in ranking the overall favored pick in each catagory satisfies my need for a "ranking system". Almost. It would be VERY interesting to me if, say, a $3,000 tv was able to outperform a $8,000 tv. I know your reviews can't be all things to all people (in every issue!), but I don't believe I would be in the minority to want to hear about about THAT!
I don't know if this is the right place to post this comment or not. But i wanted to share my views with everyone and everywhere.All of us are somehow or the other suffering from the global recession. We have to think about recoverying this loss.I tried everything and found that real estate is the best option that can beat the global economic melt down.Good luckSingle Property Websites
| Displays Electronics Speakers | |
Sources Other Gear Software | |
Top Picks of the Year Top Picks | |
Custom Install How To Buy How To Use | |
Tech 101
|
|
Latest News Features Blogs | |
Resources Subscriptions |
WHERE TECHNOLOGY BECOMES ENTERTAINMENT
© 2024 Sound&VisionAVTech Media Americas Inc., USA
All rights reserved
