Entropy of the multinomial distribution

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Entropy of the multinomial distribution

anon
It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Entropy of the multinomial distribution

Andreas Noack
I'm not aware of any plans for supporting this but, as usual, contributions are very welcome.

On Mon, Apr 18, 2016 at 1:07 AM, Anon <[hidden email]> wrote:
It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Entropy of the multinomial distribution

anon
I asked this question on stats.SE and ended up answering it myself just now, here is the link: http://stats.stackexchange.com/questions/207893/entropy-of-the-multinomial-distribution/208075#208075.

The derivation is fairly straight forward and it scales linearly in both n and k, if this is acceptable then I can code it up and submit a pull request if you'd like, but it's so simple it might be easier for you to just quickly write it up since you're intimately familiar with the package while it would take me a bit of time to get up to speed on the particulars of Distributions.jl.

On Monday, April 18, 2016 at 2:05:23 PM UTC-7, Andreas Noack wrote:
I'm not aware of any plans for supporting this but, as usual, contributions are very welcome.

On Mon, Apr 18, 2016 at 1:07 AM, Anon <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="I_spL_NTHgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">esp...@...> wrote:
It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="I_spL_NTHgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">julia-stats...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Entropy of the multinomial distribution

anon
In reply to this post by Andreas Noack
Also, here is the code I used to make sure the derivation was correct:

using Iterators

multEval(x::Vector{Int}, n::Int, p::Vector{Float64}) = (factorial(n) / prod(gamma(x+1))) * prod(p.^x)

# multinomial entropy
function multEnt(n::Int, p::Vector{Float64})
@assert n >= 0
multientropy = 0
for par in partitions(n)
if length(par) <= length(p)
point = vcat(par, zeros(Int, max(0, length(p)-length(par))))
for perm in Set(permutations(point))
multientropy -= multEval(perm, n, p) * log(multEval(perm, n, p))
end
end
end
return multientropy
end

# efficient multinomial entropy
function func(n::Int, p::Vector{Float64})
@assert n >= 0
multientropy = (-log(factorial(n)) - n*sum(p.*log(p))
+ sum([sum([binomial(n, x) * p[i]^x * (1 - p[i])^(n - x) * log(factorial(x)) for x in 0:n]) for i in 1:length(p)]))
return multientropy
end

On Monday, April 18, 2016 at 2:05:23 PM UTC-7, Andreas Noack wrote:
I'm not aware of any plans for supporting this but, as usual, contributions are very welcome.

On Mon, Apr 18, 2016 at 1:07 AM, Anon <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="I_spL_NTHgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">esp...@...> wrote:
It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="I_spL_NTHgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">julia-stats...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Entropy of the multinomial distribution

anon
In reply to this post by Andreas Noack
Ok I just went ahead and wrote the entropy function myself for multinomial.jl and submitted the pull request.

On Monday, April 18, 2016 at 2:05:23 PM UTC-7, Andreas Noack wrote:
I'm not aware of any plans for supporting this but, as usual, contributions are very welcome.

On Mon, Apr 18, 2016 at 1:07 AM, Anon <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="I_spL_NTHgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">esp...@...> wrote:
It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="I_spL_NTHgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">julia-stats...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.