>hB SrSSKJrJr SSKJr SSKrSSKJ r SSK J r SCSjr SrSDS jrS rS rS rSESjrSESjrSFSjrSESjrSESjrSESjrSGSjrSHSjr\S:XGa\"S5 \"S5 /SQr/SQr\Hr\"\"\55 M \Hr\"\"\55 M \"\"\55 \"\"\55 /SQr \ RB"S5 \ RD"S5 \ RF"S5 \RH"SSS 5r%\ RL"\%\"\%55 \ RB"S5 \ RD"S!5 \ RF"S5 \RH"SSS"5r%\ RL"\%\"\\"\%S\%- 555 \RN"/S#Q/S$Q/S%Q/5r(\(RSS5r*\(RSS5r+\"\*5r,\"\+5r-\"\(5r.\"\*\+\(5r/\"\+\*\(5r0\"S \Rb5\Rd"\*\+5-r3\"S \Rb5\Rd"\+\*5-r4\"\*\+\(5r5\"S&5 \"\,\-\.\/\0\3\4\55 \"S'5 \RN"/S(Q5r\"\5r6\ \"S)5 \"S*5 \"S+5 \"S,5 \"S-5 \"S.\*-5 \"S/5 S\*S - r7\"S0\7-5 \"\7S\7- /5r8\"S1\8-5 \"S2\8\7\Rr"S 5--\/4-5 \"S35 \"S4\7S5S6\,S- \Rr"S75- S535 \"S85 \"S95 \RN"/S:Q/S;Q/S\Rv"S\"\:S5\"\:S 5/5S- \Rr"S75- -5 \"S?5 \RN"/S@Q/SAQ/SBQ/5r>!#44T9: ~q BFF1I%%^^A& 88D>Aw"aqAw%$*!w! $*!  Orc\[R"U5[R"U5- $)z There is a one-to-one transformation of the entropy value from a log base b to a log base a : H_{b}(X)=log_{b}(a)[H_{a}(X)] Returns ------- log_{b}(a) )r r)rbs r logbasechanger;s 66!9RVVAY rc<[[RS5U-$)z Converts from nats to bits r;r ers r natstobitsr@s q !A %%rc<[S[R5U-$)z Converts from bits to nats r=r>rs r bitstonatsrBs BDD !A %%rr=cn[R"U5n[R"US:*5(a[R"US:5(d [S5e[R"[R "U[R "U5-55*nUS:wa[SU5U-$U$)a This is Shannon's entropy Parameters ---------- logbase, int or np.e The base of the log px : 1d or 2d array_like Can be a discrete probability distribution, a 2d joint distribution, or a sequence of probabilities. Returns ----- For log base 2 (bits) given a discrete distribution H(p) = sum(px * log2(1/px) = -sum(pk*log2(px)) = E[log2(1/p(X))] For log base 2 (bits) given a joint distribution H(px,py) = -sum_{k,j}*w_{kj}log2(w_{kj}) Notes ----- shannonentropy(0) is defined as 0 r r&px does not define proper distributionr=)r r r ValueErrorr nan_to_numlog2r;)pxlogbaseentropys rshannonentropyrKs2 BB 66"'??"&&q//ABBvvbmmBrwwr{N344G!|Qw''11rc>[R"U5n[R"US:*5(a[R"US:5(d [S5eUS:wa$[ SU5*[R "U5-$[R "U5*$)z Shannon's information Parameters ---------- px : float or array_like `px` is a discrete probability distribution Returns ------- For logbase = 2 np.log2(px) r rrDr=)r r rrEr;rG)rHrIs r shannoninforMsr BB 66"'??"&&q//ABB!|q))BGGBK77}rc z[U5(a[U5(d [S5eUb[U5(d [S5eUc[R"X5n[R"U[R "[R "X- 55-5nUS:XaU$[SU5U-$)a\ Return the conditional entropy of X given Y. Parameters ---------- px : array_like py : array_like pxpy : array_like, optional If pxpy is None, the distributions are assumed to be independent and conendtropy(px,py) = shannonentropy(px) logbase : int or np.e Returns ------- sum_{kj}log(q_{j}/w_{kj} where q_{j} = Y[j] and w_kj = X[k,j] 1px or py is not a proper probability distribution&pxpy is not a proper joint distribtionr=)r!rEr outerrrFrGr;)rHpypxpyrIcondents r condentropyrUs(   M"$5$5LMM  d 3 3ABB |xxffTBMM"''"'*:;;   M"$5$5LMM  d 3 3ABB |xx BD 2bd4 56rr c[U5(d [S5e[U5nUS:Xa"[U5nUS:wa[ SU5U-$U$S[ U5R 5;dU[R:Xa+[R"[R"U55*$X-n[R"UR55nUS:Xa SSU- - U-$SSU- - [ SU5-U-$)a Renyi's generalized entropy Parameters ---------- px : array_like Discrete probability distribution of random variable X. Note that px is assumed to be a proper probability distribution. logbase : int or np.e, optional Default is 2 (bits) alpha : float or inf The order of the entropy. The default is 1, which in the limit is just Shannon's entropy. 2 is Renyi (Collision) entropy. If the string "inf" or numpy.inf is specified the min-entropy is returned. measure : str, optional The type of entropy measure desired. 'R' returns Renyi entropy measure. 'T' returns the Tsallis entropy measure. Returns ------- 1/(1-alpha)*log(sum(px**alpha)) In the limit as alpha -> 1, Shannon's entropy is returned. In the limit as alpha -> inf, min-entropy is returned. z+px is not a proper probability distributionr r=inf) r!rEfloatrKr;strlowerr r_rrr)rHalpharImeasuregenents r renyientropyrfks:   FGG %LE z# a< G,v5 5 #e*""$ $rvvbz""" B VVBFFH F!|!E'{V##!E'{]1g66??rcg)aO Generalized cross-entropy measures. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) measure : str, optional The measure is the type of generalized cross-entropy desired. 'T' is the cross-entropy version of the Tsallis measure. 'CR' is Cressie-Read measure. N)rHrRrSrcrIrds rgencrossentropyrisr__main__zQFrom Golan (2008) "Information and Entropy Econometrics -- A Review and Synthesisz Table 3.1)皙?rkrkrkrk)S㥛?g;On?g'1Z?gK?gMbp?) gh㈵>g-C6?gMbP?g{Gz?g?g333333?rkg?g333333?gffffff?g?g??o Information ProbabilityiEntropye)rrUUUUUU?)qq?rtrt)gqq?rtUUUUUU?z Table 3.3zdiscretize functions)2g3333335@g@F@g?@g3@gLD@gYC@g333333&@g/@gfffff?@g9@g3333334@gffffff,@g8@g5@g&@g2@L0@g3333336@g333333@g;@rvǧA@-@g1@g333333<@gffffff0@g0@gG@g#@g2@g @@g:@g0@g333333@gffffff5@g4@gL=@rwg @g6@g)@gfffff:@g9@gfffff6@gffffff&@g333334@g333333:@g"@g%@g333333/@z0Example in section 3.6 of Golan, using table 3.3z'Bounding errors using Fano's inequalityz"H(P_{e}) + P_{e}log(K-1) >= H(X|Y)zor, a weaker inequalityzP_{e} >= [H(X|Y) - 1]/log(K)z P(x) = %sz?X = 3 has the highest probability, so this is the estimate Xhatz1The probability of error Pe is 1 - p(X=3) = %0.4gzH(Pe) = %0.4g and K=3z-H(Pe) + Pe*log(K-1) = %0.4g >= H(X|Y) = %0.4gzor using the weaker inequalityzPe = z0.4gz >= [H(X) - 1]/log(K) = z>Consider now, table 3.5, where there is additional informationz.The conditional probabilities of P(X|Y=y) are )ryg?)rsrsrs)rursrmz2The probability of error given this information iszPe = [H(X|Y) -1]/log(K) = %0.4gz+such that more information lowers the error)gV-?gV-?gw/?)g(\?g+?g%C?)gzG?rlgPn?)N)r#N)r=)Nr=)r r=R)r r=T)=__doc__statsmodels.compat.pythonrrscipyrnumpyr matplotlibrplt scipy.specialrr r!r8r;r@rBrKrMrUrYr[r]rfri__name__printr Yr7psubplotylabelxlabellinspacexplotarraywrrHrRH_XH_YH_XY H_XgivenY H_YgivenXr?rJD_YXD_XYI_XYdiscXpeH_perGw2mean markovchainrhrrrs D1$3 !H#P & & F,3@<&P(6\.@f, z   + A"A  k!n  k!n .  . ;AKKJJ}JJ} AaAHHQ AKKJJyJJ} AaAHHQ^T!AaC[12 *-.?@AA qB qB  C  C ! DBr!$IBr!$I 244 r2!6 6D 244 r2!6 6D bA D + #c4ItT4@ !hh:;A qME  <= 34 ./ #$ () +  KL RUB = BC 2ad) $D !D () 9 "RWWQZ-  + ,- *+ E"T2C!GRWWQZ3G2M NO JK :; :./?@ AB "I >? +>"Q%;PQ_`bcd`eQf8g0hij0jlnlslstulv/v wx 78((,-=>NOPKir