One of the most popular - and in truth, most valid - ways of comparing two products is to, well, directly compare two products. A battle royale, two-enter-one-leaves style of head-to-head competition where it's clear which product is the victor.

Done correctly, direct A/B comparisons are by far the most accurate ways of determining product superiority.

The problem is, they're often not done correctly. Sometimes, they can't be done correctly. In those cases, the results couldn't be further from accurate.

Televisions

Determining the best TV is one of the hottest Internet topics. It's understandable, to a degree. Most TVs cost a lot of money, and most people want to be sure they're getting the best for their money. Then of course there's the rampant fanboyism that is neither scientific, nor useful.

The problem is, it's nearly impossible for the average person to accurately determine what TV is better. Not because of any failing on their part, or from lack of research effort. (Manufacturer specs are often outright lies, so they're no help.)

The real problem is the lack of any decent testing space at retail. In a controlled environment, with correctly set up televisions, I'd bet good money that anyone reading this would pick the same TV I would. If not exactly the same, at least a TV that performs very similarly to my pick.

But you'll never find a controlled environment. You'll never see a correctly set up television. The Costcos and Best Buys of the world are about as bad a place to compare televisions as an Amish farmhouse.

The lighting in every store is radically different than what you have in your home. The plasma that looks washed out in a store will likely look better in your home than the LCD that looked amazing right next to it on the shelf. Worse yet, it's unlikely the TVs at a store will be set even remotely correctly. They're set to look great at a store, which is the exact mode you shouldn't be using at home. I've even seen TVs in stores deliberately set brighter than others, because that was the TV they were pushing that week. Dishonest? Yes. Does it work? Absolutely.

Cables

A few months ago I wrote an article about why all HDMI cables are the same. In it, I outlined the physics of how HDMI cables worked, cited sources, and showed why your picture will look the same whether you use a $2 HDMI cable or a $400 HDMI cable.

I'll admit to some sensationalism in the title (build quality is a big difference between cables, but it doesn't affect picture or sound quality), but what was amazing was how many people said something along the lines of "well I have two cables, I compared them, I saw a difference, you're wrong."

Now, it's not possible for them to have seen a difference. I proved that with, you know, facts. So what's going on? The simplest (and most condescending) explanation is that they're not seeing a difference at all. They want to see a difference to justify the $100+ by which they've just been ripped off.

Personally, I'm not a big fan of that answer. So let's go a step further: how would one test cables at home? The most obvious solution, perhaps, is to plug in both HDMI cables to two different inputs on the TV and go back and forth. A direct A/B comparison, correct? Incorrect. Are all the settings the same between the two inputs? If there's a visual difference, then probably not.

What about switching out the cables? Watch one, replace, watch the other. How long a time between light-on-eyeballs is there? No one's memory is that good, especially with the added bias of knowing, "well here's the expensive cable."

Speakers

Now this one is even more difficult. So difficult, in fact, it's nearly impossible. Your ear is incredibly sensitive to changes in volume, and almost always louder is better. So matching volume is critical, but how do you do that? Most SPL meters aren't as sensitive as your ears. Matching the volume on the receiver/preamp won't work, as speakers all have different sensitivities, so "-20" on the receiver via speaker A might be louder than the same level through speaker B.

Then there's placement, as the speakers need to be in the same spot so the room has the same effect on them. Everyone's audio memory is pretty fickle, so if you wait around while someone moves the speakers into place, it's likely you've forgotten what the speakers sounded like to begin with.

No Conclusion, mostly

Is this a thinly veiled laudation of magazine reviews and trained reviewers? Sort of (well, at least the good ones). I've always said that you should trust your own eyes and ears when it comes to audio and video gear, and let them be the final judge. But let your knowledge of these potential errors in direct comparison be your guide. With any test, isolating the variables that can affect the outcome, but aren't themselves the factors under test, is more important than the test itself. This is true even with A/V gear.

With TVs, keep in mind that a well-lit store isn't the best place to compare TVs and that the TV's settings will have a profound effect on the picture quality. If you can't access them, it's nearly impossible to correctly judge the display.

With cables (or any piece of gear, really), know that the simplest A/B test may have significant variables that can dramatically affect the outcome. When the results go against physics - well, I don't know what to tell you. Take a close look at your test setup. There may be a difference somewhere, and it's not in the cable.

With speakers, trust your ears. Whatever sounds good to you is the best speaker.

As any good scientist will tell you: If you're getting results radically different from everyone else, it's probably time to re-check your testing methodology. Sure, it's possible you've discovered penicillin, but it's more likely you've just got a jar of mold.