Periodic Audio IEMs and the Technical capability of headphones - Part 2

Background 1: Low-Distortion IEMs

About one year ago, a couple of us from HypetheSonics visited Periodic Audio to meet with the owner and discuss their element-themed in-ear monitors. Periodic Audio have been involved in all manner of audio product development (you can see their existing offerings at periodicaudio.com), but we wanted specifically to talk to Periodic Audio about some unique features of their IEMs that dovetailed with our research into a headphone’s ‘technical capability’.

Most manufacturers of in-ear monitors tend to simply throw products (and frequency responses) at the wall, hoping something sticks. Periodic Audio, on the other hand, have a strong conviction about their sound signature and are (to the best of our knowledge) entirely unique in that all of their IEMs are designed – at least in theory – to have the exact same frequency response, with the differences in sound then a function of material properties such as diaphragm composition. Periodic Audio’s original Beryllium IEM also happened to measure with the lowest distortion we’d ever seen in an IEM at our standard 94 dB test level, and since the development of their original Be earbuds, Periodic Audio spent a substantial amount of time developing a co-polyester casing for their newer IEMs to damp acoustic resonances. There does appear to be a benefit to this as their new Carbon V3, with its lab-grown diamond diaphragm, is now our best-measuring IEM in terms of THD at our standard 80 dB test level. The appeal for us was that identically-tuned IEMs could form a perfect set of guinea pigs for our R&D into non-tonal sonic differences. It would be an understatement to say that the resulting rabbit hole turned out to be a little deeper than we anticipated, but it led to some interesting insight and developments that helped improve both our headphone and our DAP measurement procedures.

Background 2: A New Measurement Procedure

The measurement work described here stems from a long and fruitful history of collaboration between HypetheSonics and SoundExpert.org, and represents a natural extension of the work we’ve been doing for many years in measuring errors in portable digital audio players (DAPs) and digital to analog converters (DACs). The crux of this work is based on SoundExpert’s contribution to the audio world of the ‘Df’ or difference metric, which can be described simply as the normalized RMS waveform error (or difference) of an output time signal, compared to the original input signal. The test vector(s) used to compute these differences can, in theory, be any type of audio signal. Our usual test vector for DACs includes standard signals such as isolated sine waves (commonly used to report total harmonic distortion or THD), dual sine waves (commonly used to test for intermodulation distortion, or IMD), square waves, white noise, program simulation noise (whose spectra is designed to mimic actual music) and a mix of actual music tracks. This last and longest component of our test vector forms the most important metric as SoundExpert’s online listening tests have shown this to have a high correlation with listener preference. Although this test covers the full range of human hearing, it places a higher weight on the spectral content of actual music. The Df error from actual music tracks turns out to be one of the most difficult types of error to minimize in any electrical or electro-mechanical reproduction system. Because music tracks encompass all manner of signal type, Df error levels are typically orders of magnitude higher than those from traditionally-reported total harmonic distortion (THD) from a 1 kHz sine tone. We have long argued (and this opinion now seems to be rather widespread, despite every OEMs’ refusal to budge on the issue) that the standard metric supplied by most manufacturers – THD based on a single sine wave – is irrelevant and misleading. Almost all modern DACs are easily capable of accurately reproducing isolated sine tones, yet most are incapable of reaching the same vanishingly-small level of error with actual music.

The current state of the audio world is such that the massive imbalance in demand for DACs vs ADCs, has created a situation where no measurement equipment exists right now which is able to accurately quantify all errors in the best-available modern DACs to a level that does not include artifacts from the ADC or measurement rig itself. Fortunately, however, the relatively large errors incurred with real music test vectors makes it possible to discriminate between even the best currently-available DAPs, DACs and amps with relatively modest analog-to-digital converter equipment. Our current DAC-measurement equipment consists of a Texas Instrument PCM4222 ADC with a miniDSP USB interface, and is able to report errors lower than 80 dB below the fundamental for these complex waveforms with the best currently-available DACs. Headphone errors are of course expected to be far larger than this, but it is nevertheless still worth addressing some potential criticisms of this measurement equipment and process.

The reported Df error measures are not absolute. When measuring the best currently-available DACs, part of the reported error originates from the ADC and measurement process itself. This is the reason why our DAP/DAC databases are limited to one unique/unchanging set of measurement equipment. Absolute numbers (especially for the more accurate devices) will likely change with different measurement equipment, however overall error ranking should not (this effect can be seen in our older legacy measurement database produced with RME recording equipment). Additionally, not all errors are equally audible. A fair critique is that a larger absolute error might, for various reasons, be less audible than a smaller error of a different type. There has been some effort to address this, for example via the ‘PK’ metric in the DeltaWave null comparator tool. This tool isn’t open source and doesn’t appear to be documented, but it is free to download and use and we have been actively investigating it. There is a comment made on the DeltaWave website that is worth noting. It states: “Unlike the DF Metric proposed by Serge Smirnoff, PK Metric is not confused by small errors in timing or noise”. The Df algorithm applies time warping and shifting within small (50 or 400 ms) window samples to circumvent issues such as clock drift and allow matching signals to line-up before differences are computed, but any resulting differences in magnitude or phase are indeed considered errors, so there is a point to concede here. Certain types of signal processing, such as a pure phase rotation, could indeed result in a large Df error despite the resulting effect appearing rather benign to the listener – at least under some circumstances. A good example of this seems in the current context seems to be the Anker-Soundcore Liberty 3 Pro true-wireless Bluetooth IEM. The Liberty 3 Pro measures with huge Df errors and an outrageous impulse response which would suggest the manufacturer is intentionally applying some form of digital signal processing (DSP) – we suspect EQ and/or an attempt to widen the sound stage, even in default mode. The Liberty 3 Pro are actually good-sounding headphones, so clearly not all errors are equally bad. Nonetheless, from a purist mathematical viewpoint, one could argue that there is still an error in the waveform with respect to the original reference track, and therefore, for our first attempts at these measurements, we are using a simple mathematical RMS-based metric to assess the total waveform error. We believe this is important for the following reason: users should be free to apply their own DSP – or not – and shouldn’t have to deal with errors or transformations upstream in the audio chain that aren’t documented and/or can’t be corrected for. Phase errors might be less critical in the current (wired) two-channel IEM world where minimal or zero effort is made in regard to spatial placement of the sound (with the soundstage tending to collapse toward the center of your head), but the final frontier of two-channel audio is to reproduce the perfect three-dimensional spatial sound field with individually tailored head-related transfer functions, and in this situation upstream phase errors would be problematic as they could have significant effects on spatial perception and could be difficult or impossible to untangle. The Df error is therefore something that we ideally want to drive to zero – or at least to levels below the threshold of human hearing. Many modern DACs are fast approaching that point, and have arguably already surpassed it at modest output levels (see for example our measurements of the Shanling M0 Pro). Headphones, on the other hand, are likely to require a paradigm shift in technology to reduce error levels substantially below where they are today. One such possible path forward is suggested later.

A Df Metric Test for Headphones

In trying to develop a waveform-error test for headphones, a number of problems had to be overcome.

Firstly, the length of our original DAC test vectors was enormous – nearly 2.5 hours in total. DAPs and DACs can simply be left anywhere to record, but headphones are much more challenging. Even with the double anechoic chamber we use for testing headphones, external noise and vibrations are difficult to completely eliminate throughout entire stretches of a very long recording. Certain IEMs can also have a tendency to shift or even wriggle free from the ear simulator over time, especially when constantly stimulated with a moderately-loud signal. Very long test vectors also severely limit how many measurements you can make in any given period of time. Ideally, a shorter test vector was needed for headphones in order to eliminate external noise and vibration and placement issues during the recording period.

Secondly, headphones aren’t designed to produce a flat frequency response. In order to sound ‘normal’, or ‘realistic’, or ‘pleasing’, an in-ear monitor is typically tuned to roughly mimic the response that the external anatomy would have created, were the sound to have reached the ear in the manner nature intended. There seems to be two main reasons why evolution left us all with pinna – our external ear anatomy amplifies sound (though not uniformly across all frequencies), and also reflects sound both vertically and horizontally into the ear canal. Amplitude differences between left and right ear can assist with horizontal placement, but vertical placement and distancing relies on timing differences from those pinna reflections. Our brains are used to processing the resulting phase and magnitude responses, but once an ear canal is blocked, that physical mechanism is by-passed and must be simulated by an IEM in order to trick the brain into thinking it’s hearing ‘normal’ sound – that is, sound that has reached the brain in the traditional way we’ve heard all our lives. The most important effect the IEM needs to replicate here is the tonal shift, which immediately causes the signal to depart from that of the source or reference track.

Thirdly, the use of a coupler or in-ear simulator adds its own modulation to the sound received at the virtual eardrum in the same way a real ear would – i.e., the amplitude and phase of the original wave is further altered by the specific transfer impedance response of the ear or ear simulator.

The first part of our work involved developing a new, compressed set of test tracks. The result of this initial work was a new, shorter, combined test vector, less than 15 minutes in total, that incorporated a mix of music (rock, jazz, folk, EDM, classical) along with various standard test signals. That same combined test vector is now being used for both our headphone and DAC tests. The observant HypetheSonics’ reader might have noted that our updated test vector now includes some high-resolution (96 kHz/24-bit) test signals. There is one important comment to make about these: our tests only include errors in the audible frequency range (20 Hz – 20 kHz). The fact that some digital audio devices give improved reproduction with hi-res signals while others don’t is mainly caused by differences in these devices’ anti-aliasing filters when switching sample rates; in some devices, the change in antialiasing filter directly impacts frequencies below 20 kHz. Despite the much-shorter duration, our new test vector is able to almost perfectly reproduce the time-averaged statistical results of the original (2.5 hour) test vector on digital audio players. Our hypothesis is therefore that these output statistics should correlate well with all forms of modern music.

To EQ or not to EQ, that is the Question

While DACs and amps are, or arguably should be, designed to reproduce the original signal with zero coloration or deviation from a perfectly-flat frequency response, the same is not true for headphones. For the reasons discussed above, some change in headphone tonality (with respect to the flat reference signal) is normal and expected. This creates a dilemma for a headphone Df error test, since the output of any modern headphone (even one whose response is closely tuned to some idealized preference target) would never null against the flat reference waveform.

The perfect solution to this problem isn’t something we’ve yet reached a firm consensus on, even internally. Some of us at HypetheSonics feel that headphone waveform error should simply be measured as is, with magnitude degradation being attributed as tonal error and phase degradation being attributed as any remaining (non-tonal) error. This would, however, appear to cause an obvious bias with an inherent increase in error as the headphone’s tonality departs from a flat response, which must happen (to some degree) for all headphones. Other thoughts at HypetheSonics are that we ought to first attempt to correct for the frequency response of the headphone, especially since the concept of ‘technical performance’ indicates this metric ought to be independent of frequency response. Although a headphone’s frequency response (amplitude vs frequency) is the curve most people obsess over, there are a number of reasons to believe its importance may be waning, and why, therefore, we would want to try to remove that aspect from this test.

Firstly, the plasticity of the human brain means that we can learn to accept a modest shift in tonality as sounding completely normal after a certain period of time. Secondly, most headphone manufacturers now understand roughly what type of tuning appeals to a mainstream audience, and in the last decade the spread in headphone tuning appears to be narrowing, with many headphones closing in on something vaguely resembling the Harman target. In other words, differences in tonality are becoming much more modest, and therefore our brains can more easily accept such changes. Much of the variance around current headphone tuning seems, in our opinion, to be more a form of correction for equal loudness to satisfy users with different listening-volume preferences. (In fact, some manufacturers, such as Bose and Apple, now adjust their Bluetooth headphones’ FR as a function of listening volume to account for this.) As differences in headphone tuning (for a given SPL) become smaller, characteristics such as fit, comfort, build quality, design aesthetics, price, marketing and any other (non-tonal-related) reproduction errors will inevitably become more important discriminating factors. This is especially the case if we consider one other factor – good-quality parametric equalization is now widely available in many Android devices (via UAPP or Wavelet), and even on iOS devices via Bluetooth dongles, such as the Qudelix-5k. At some point, high-quality parametric equalization may even become a standard feature of true-wireless earbuds. Automatic EQ tools on websites such as HypetheSonics now make it very easy to generate the output coefficients needed to equalize one headphone to nearly perfectly match any other headphone’s response, any existing target, or even to a completely new target (for example, HypetheSonics’ MC or ‘Modifiable Copy’ feature lets you interactively drag and draw to modify any existing FR target for equalization). For all these reasons, it seems that headphone tuning may not continue being the massive discriminating factor it was just a few years ago.

Whether or not we are correct about the diminishing significance of amplitude versus frequency curves, we already have metrics available to score any departure from a given target frequency response, and so we don’t need a new test to reproduce that same effect. Target curves themselves are also not really universal – even for a single individual, who might tomorrow decide to listen to some new music with a very different spectral content. The only curve that isn’t arbitrary is the flat response curve of the reference track, and this can be corrected for by perfectly inverting the frequency response curve of the headphone measurement during the same recording session (without moving the headphone in the coupler). The goal is then to look for the error around the headphone’s mean response. However, ultimately it seems that these two positions (to EQ or not to EQ) might not be as different as they first appeared – at least in terms of the error metrics that emerge from current headphones.

To help answer some of these question about the use of equalization in these headphone error tests, we studied the effect of the total waveform error (Wf), along with the isolated magnitude (Mg) and phase (Ph) errors using a set of representative headphones, which included Periodic Audio’s dynamic driver IEMs, balanced armature, hybrid and electrostatic in-ear monitors. We consider this set to be representative of all existing in-ear monitors since planar magnetic and dynamic drivers tend to have very similar characteristics (with comparably low 2nd-order dominated harmonic distortion) and xMEM drivers (and estat drivers) tend to have characteristics very similar to that of balanced armature (with comparably large, 3rd-order dominated harmonic distortion). We looked at various correlations with the raw waveform measured directly from the headphone and also with a minimum phase equalization applied to that waveform in order to revert the output to a perfectly flat frequency response. We also included theoretical tests by simply equalizing the reference track to mimic a specific headphone in order to compare errors that arise purely from an idealized tonal shift with those of the headphone itself. The ear simulator used for the headphone recordings was the GRAS RA0045, but beyond the quality of the 40AO microphone, the coupler is not considered critical since determination of the frequency response is not the goal here and any modification to the frequency response would also be accounted for with any applied EQ.

The recorded output is processed in 400 ms time slices and errors accumulated into a histogram, showing normalized occurrence on the y axis and the error in decibels (below full scale) on the x axis. Some example outputs are show below: