Measuring entropy

[GotthardtZ]

The current "MeasureEntropy" implementation works like this: for all "input" bytes, read them, look up that byte in a dictionary, increase occurrence. Then sum these occurrences (totalFrequency) and calculate the total entropy along the dictionary entries.

Probably it can be made faster: create an uint "counter" array of 256, use a for loop to read all bytes from input, increase counter in array. You don't need to sum the occurrences (totalFrequency) - it equals to the length of the input.
Then calculate entropy along each "counter" array element where the counter is nonzero.

However...
There is one problem with measuring the entropy per input byte: it's an order-0 model. No compression software is an order-0 model, so making any decisions based on the order-0 entropy is not "solid" or "realistic" enough.

[mitiko]

The entropy measure is pretty meaningless if we don't do the entropy coding afterwards. That's why I'm not considering adding an order-1 entropy calculator - most of the code would be the same as if I just wrote a simple ok-ish arithmetic coder.

[GotthardtZ]

I'm not sure I get what you mean.

This is my understanding: an entropy measure function gives a very similar result to an arithmetic encoder. An arithmetic encoder actually does the encoding while the measure function is just simulation the encoding (with -log2(p)*x). So they are the same when it's all about measuring entropy. If you actually want to do the compression (not just measure entropy) then you do need an arithmetic encoder. Is this what you meant?

However since you have only a transform at this moment and no modelling, you yourself alone cannot really judge if the transform is good or not.
Measuring the "goodness" of the transform is by actually trying it in a real compression scenario. Looking at the results posted in encode.su it seems like the current transformation hurts compression.
It's probably the best not to use the (current) entropy measure at all for comparing/measuring the goodness of different ideas. I suggest to judge the "goodness" of any future improvements to see if it really helps (external) compressors. If after an improvement compression with an external compressor gets better, go with it. But if it's not helping compression or even hurting compression that's most probably not good.