pouët.net

Some "quick" technical notes behind this demo

Lets start off with the encoder

The encoder analyses the complexity of variable chunks per 20/40ms and this determines the chunk length to analyse. This is useful for preserving transients as well.

The encoding process extracts the following data from the chunk

frequency
This is 10bits. however an additional bit to determine if the frequency will traverse backwards. This can get better matches for asymettrical waveforms

phase
a range of 0-255 which is the start offset into the waveform

volume
a range of 0-31. This implies 5bit resolution, however on inspecting the maximum amplitudes, many would not reach the highest peaks. For this reason, the encoding is at 7bit volume resolution per layer (0-127) however only the first quarter of this is used which can then fit into 5bits

waveform
a range of 0-7. any one of 8 waveforms (noise, sine, triangle, pulse, sawtooth, resonance1, resonance2, trisaw)

It does all the above for x amount of layers. In this demo I opted for 8 layers (channels)

When comparing this with fourier transform based encoding, it requires only a quarter of the layers to achieve the same quality.


Now onto the decoder

the lower 3 bits of the 11bit frequency point to the finetuned pitch tables. the upper 8 bits are the main step values. this is incremented all together to form 8 step values which are written to the main decodeloop per layer (to the waveform low bytes)

the phase is merely written to the index per layer

the waveform value is calculated by pointing to the 512 byte waveform (this is 256 bytes repeated twice) to allow the indexing crossover. exception is the noise waveform.

the volume data points to the relevant scaled volume table (I did previously experiment in signed volume tables, but this introduced unwanted noise due to carry flag clear/set due to crossover

the decoder has 8 phase accumulators per layer. The main optimisation is that the phase is only updated once per 8 bytes. This is due to the step values already calculated per 8 bytes. At the end of the 8 bytes, the last step value is written to the indexing which allows it to loop seamlessly at the required frequency


Now info about the spectrogram
Yes it is realtime, but nothing too complex about that as we already have access to the frequency and amplitudes per layer.
There are 2 main stages, the fader (which fades each level of the frequencies) and the plotter (which plots the main updated strips)


the sample rate is at 5200hz (this is rather low) but as i am calculating pitch tables on the fly for 8 layers (and the main decode has phase accumulators updating in realtime) this used quite a lot of cpu time (though saves a lot on precalculated data).

I do have variants of the decoder that use larger tables to precalculate both volume and waveform together and run at higher sample rates, but use over 30k of tables (compared to 5k). Precalculating pitchs to the waveforms is not an option either as i would need over 1000 of these per waveform

Average compression per second is between 300-700 bytes per second, however since releasing this demo, I have implemented "phase quantisation" where i can shave off roughly 100-200 bytes from the total without much loss in quality

The encoder and decoder is being updated continiously with improvements (In particular i am working more on improving the quality at sub 300 bytes a second)

Some people were curious about how it would sound with some voices. I have prepared some examples. Bear in mind that encoding takes a very very long time, hence I used the quantised phase options to encode faster as a preview (This gives roughly a 25% degradation in quality) but at the plus point of using on average under 450 bytes a second for both these examples. examples of three voiced (with instrumentation combined) here. https://youtu.be/LgIcf25rTiw