Feedback on "quick review" process

[amirm]

Hello everyone. My workload of reviews continues to be very, very heavy. I provide the same set of tests regardless of desirability of the unit. Often I skip a test or two only to be asked where they are.

Question is, can we agree on a "quick review" format where stuff gets tested quickly allowing higher throughput and owner getting his gear back sooner? Or do we continue as is?

As an example, personally the moment I look at the dashboard on a DAC, I learn 90% of what is in the rest of the test, if not 100%. The dashboard takes five minutes to setup and capture. The rest of the tests multiply the effort proportionally making the job much more resource intensive.

In the case of speakers, the spinorama alone could be the main measurement for say, a DIY speaker that is not of great interest. Since I include spinorama export in the review, others have been posting directivity and such anyway. Graphs like CSD/waterfall, heatmap directivity, etc. are all time consuming to capture.

Every review right now blows an entire day. Measurements usually take a couple of hours and photographing and annotating them takes another two hours. Then there is the write-up in the post.

Anyway, the default path is to continue as is but thought I bring up the topic and see what members think.

 

[Icboschert]

+1 for speed reviews on DACs if the dashboard looks good and clean.
On speakers, I like to see the directivity graphs but could forego them.

 

[Absolute]

I'm all for quick reviews of less interesting stuff like diy speakers, outdated cheap dacs/amps and speakers. Current stuff of high interest should of course remain full reviews, imo.

Perhaps it's a possibility for owners really interested in full reviews to pay/donate for the extended measurements where the object is of little interest to the majority?

 

[Vasr]

Ideally, if someone else can set up a table like the nice bit of programming in the equipment review comparator, there would be a list of equipment on-hand with Amir and yet to be tested, then people here could vote on at any time and Amir would pick the next highest voted at any time. May need some way to avoid "starvation" for equipment that aren't as popular and stay at the bottom - perhaps a quick review for them.

 

[Jimbob54]

If its a choice between some condensed reviews or less reviews, then the former.

For the kit you currently measure, Im far more interested in DACs, Dongles and HP amps than speakers and AVRs so cant comment on those.

I suspect no one needs an extensive review of a $50 dongle DAC. SINAD and power output should be fine which are both dashboard.

Class leading DACs - maybe more detail. Expensive DACs that SHOULD perform better, maybe more detail.

HP amps- same deal.

However, I get a sense from comments on some reviews that there is expectation you will test EVERY attribute. Polarity on a recent hero DAC being one recent example. I dont believe that is where you want to go, nor do I think you should, but there will be a substantial portion of the readhership that line in the minutiae and think you're some kind of proxy for a QA function.

 

[pozz]

Proposed quick formats:

Speakers

1. Spinorama
2. Distortion Spectrum (86dB vs. 96dB)
3. Heatmap directivity

Electronics

1. Dashboard
2. SNR
3. Distortion vs. frequency
4. IMD vs. level
5. Distortion vs. power (if applicable)
6. Peak/burst power (if applicable, for power amps)
7. Channel balance vs. volume (if applicable)

No comparison charts. No subjective section or listening tests.

 

[maverickronin]

Cutting back to dashboard, OI, power, and FR, and SINAD at 50mv for DAC and headamps would be still be plenty useful.

The IMD and other multi-tone tests are more on the technically interesting side rather than something helpful for making a purchasing decision.

 

[Fluffy]

I would say drop DACs altogether, but I guess most people won't feel the same. So quick review for DACs sound good. I feel speakers are the most important thing right now, so make a shorter format review only if it's an unimportant one or the spinorama reveals very poor performance so there is no point measuring other aspects.

If this is coming up because of the headphone reviews idea, I think you should drop the headphone reviews. There are more than enough headphone reviewers out there, and it looks like diving into it will just create even more confusion, with the whole compensation deal.

 

[amirm]

If this is coming up because of the headphone reviews idea, I think you should drop the headphone reviews. There are more than enough headphone reviewers out there, and it looks like diving into it will just create even more confusion, with the whole compensation deal.

Headphones are not in the picture yet. The issue has been constant for a year or more as review format has grown substantially for just about every product category.

 

[daftcombo]

I like full reviews.

Perhaps a format of quick reviews with the tests but no annotations nor text if there's nothing extraordinary to say, as @WolfX-700 sometimes does?

(Listening tests are only needed for speakers now though IMO.)

 

[NTK]

Proposed quick formats:

Speakers

1. Spinorama
2. Distortion Spectrum (86dB vs. 96dB)
3. Heatmap directivity

Electronics

1. Dashboard
2. SNR
3. Distortion vs. frequency
4. IMD vs. level
5. Distortion vs. power (if applicable)
6. Peak/burst power (if applicable, for power amps)
7. Channel balance vs. volume (if applicable)

No comparison charts. No subjective section or listening tests.

For speakers, wouldn't we want the impedance plot too?

 

[Jimbob54]

Proposed quick formats:

Speakers

1. Spinorama
2. Distortion Spectrum (86dB vs. 96dB)
3. Heatmap directivity

Electronics

1. Dashboard
2. SNR
3. Distortion vs. frequency
4. IMD vs. level
5. Distortion vs. power (if applicable)
6. Peak/burst power (if applicable, for power amps)
7. Channel balance vs. volume (if applicable)

No comparison charts. No subjective section or listening tests.

No disrespect to Wolf, but without the subjective and listening tests , his reviews feel a bit lacking. In fact, they aren't reviews. They are measurements.

 

[Matias]

Honestly, I prefer to have the current bandwidth of reviews and have them all complete. ASR has almost daily reviews, this is unprecedented in the review industry. Why speed it up any further? Too many reviews and they soon will be hard to keep up reading too!

Owners should be informed of the queue and an estimate on how long it will take to get their gear back, and let them make the choice to send or not.

I still think ASR should have a 2nd person on staff testing stuff. At least doing the measurements, charts and photos, and let Amir do the evaluation and writing. Just hire and train a local kid to do the basics, including unpacking and packing and the heavy lifting of amps and speakers.

 

[pozz]

For speakers, wouldn't we want the impedance plot too?

Thought about it, but I think the bare minimum doesn't require it.

There's also SINAD vs. level for electronics and multiple types of SNR (apart from SNR/DNR at max output) in general and output impedance for headphone amps which take time.

 

[Inner Space]

Question is, can we agree on a "quick review" format where stuff gets tested quickly allowing higher throughput and owner getting his gear back sooner? Or do we continue as is?

As an example, personally the moment I look at the dashboard on a DAC, I learn 90% of what is in the rest of the test, if not 100%. The dashboard takes five minutes to setup and capture. The rest of the tests multiply the effort proportionally making the job much more resource intensive.

I think you should start every test intending to make it complete and comprehensive, but you should then abandon it as soon as you hit a major dealbreaker. It's of no real interest to us to dig down further if the device under test has already disqualified itself early on. If you never hit a dealbreaker, then we end up with a full suite of measurements and a de facto recommendation. That should help your efficiency, while missing nothing of significance.

 

[Biblob]

Proposed quick formats:

Speakers

1. Spinorama
2. Distortion Spectrum (86dB vs. 96dB)
3. Heatmap directivity

Electronics

1. Dashboard
2. SNR
3. Distortion vs. frequency
4. IMD vs. level
5. Distortion vs. power (if applicable)
6. Peak/burst power (if applicable, for power amps)
7. Channel balance vs. volume (if applicable)

No comparison charts. No subjective section or listening tests.

Yeah, this is something I can stand behind.

I'd like to add that I have a personal interest in DIY speakers, because they can offer so much value if done right. And that's why skipping these would be a loss, I think.

But I agree running many test on a product which is badly designed, is a waste of time.

 

[flipflop]

IMO:

• Scrap the early reflections graph, all directivity graphs, and CSD from speaker reviews.
• In electronics reviews, withhold jitter, like you often do with FR, unless there's a problem with it. Dynamic range can go, too.
• Skip listening impressions for both types of reviews.
• For expensive products, terminate testing early if the dashboard/spinorama shows terrible performance.

Also, please stop buying new gear until you have reviewed what you've already got.

 

[amirm]

Keep the feedback coming. Much appreciate the comments.

 

[waynel]

A few comments:

1) Always measure from an analog source and a digital source when possible to cover use cases
2) Sweep AVRs pre outs to 4V
3) My preference is that you spread yourself to thin by adding headphones
4) When you find a best in class product spend a little more time with it

 

[MZKM]

I already compute heat maps and the Early Reflections for speakers. My heat maps aren’t that nice looking (it’s all done in Sheets with conditional formatting, and only 10 degrees whereas Klippel has more angles), but they do the job. If you want, I can post these.

Horizontal heat map of the SVS Ultra for example:

It looks a bit different as others use 1 color per dB, I also do this except I allow basic red to cover the span of:
<= average SPL + 1dB
=average SPL
>= average SPL - 1dB

That way it is more forgiving (but again, every other color is in 1dB increments). It took me a while to find an online color gradient generator, where you tell it the HEX colors you want and the # of colors you want in-between.


Early Reflections of the SVS Ultra for example:

Waterfall plots show resonances which show up in the Spinorama and impedance/phase plots.

Impedance/phase should not be dropped.
I would prefer distortion be kept.
I would prefer the max SPL in the bass be kept.

______________
For DACs, cheap stuff can get:
SINAD, THD vs level, IMD vs level.

Expensive and/or very popular products should be given more data.

As is sorta typical, aspects like jitter don’t need to be shown unless out of the ordinary.

 

[Robin L]

If you can tell that a product is substandard during the first test, I don't see the purpose in going further. Ferreting out the cause is the manufacturer's concern.