Blog

I asked this at math.stackexchange.com, but got no answers. Let $ (X,B,\mu)$ be a probability space. Let $ T,S:X→X$ be two measurable measure preserving maps that commute (i.e $ TS=ST$ ). Let $ A$ be a (countable measurable) partition of $ X$ . Show that $ h(ST,A)≤h(S,A)+h(T,A)$ . If $ S=T$ , it’s rather easy.Read more

I am interested in using Shannon’s entropy in combinatorics. It is often presented with a motivation of how much information can be passed, but assume I am not interested in that, I want to understand why it is useful as a tool to bound objects (with examples now). Several examples that demonstrate that very nontrivialRead more

I am interested in calculating various entropy measures of time series, and was surprised to find that none of them are included with Mathematica. Sample Entropy (SampEn) is a fairly common technique for this purpose, so I’ve written a module to perform the calculation, but I’m surprised at how slow it is (especially compared toRead more

i’m currently making an application using python to hide a message in an audio file (WAV only for now). The hiding itself is done pretty quickly and without any problems, but i want to calculate the best place for it by calculating entropy. The purpose is that the higher entropy of the fragment, the lessRead more

The $ \phi$ entropy is defined as $ \text{Ent}_{\phi}[X]= \mathbb{E}[\phi (X)]-\phi(\mathbb{E}[ X])$ where $ X$ is a random variabel and $ \phi$ is a convex function ($ \text{Ent}_{\phi}[X] \geq 0$ ). By choosing $ \phi(x)=x^2$ we get $ \text{Ent}_{\phi}[X]=\text{Var}(X)$ . if $ \phi(x)=x\log x$ and $ X=\frac{dv}{d\mu} $ (radom nikodym derivative) we get $ \text{Ent}_{\phi}[X]=Read more

this is a very common question but I think I’m posting it in a different way. Suppose that we have 256 different possible messages with uniform distribution to be represented in a binary code. Then it would be necessary at least 8 bits in each word to represent all the 256 = 28 possibilities. Nevertheless,Read more