>hISrSSKJrJr SSKrSSKrSSKJ s J r SSKJ r J r SSjrSSjrSrSr"S S 5r"S S \5rS rSrSr"SS5rg)aTools for working with groups This provides several functions to work with groups and a Group class that keeps track of the different representations and has methods to work more easily with groups. Author: Josef Perktold, Author: Nathaniel Smith, recipe for sparse_dummies on scipy user mailing list Created on Tue Nov 29 15:44:53 2011 : sparse_dummies Created on Wed Nov 30 14:28:24 2011 : combine_indices changes: add Group class Notes ~~~~~ This reverses the class I used before, where the class was for the data and the group was auxiliary. Here, it is only the group, no data is kept. sparse_dummies needs checking for corner cases, e.g. what if a category level has zero elements? This can happen with subset selection even if the original groups where defined as arange. Not all methods and options have been tried out yet after refactoring need more efficient loop if groups are sorted -> see GroupSorted.group_iter )lrangelzipN)Index MultiIndexc [U[5(a[R"U5nO[R"U5nUR nUR S:HnU(anURSnURR(d[R"USS9nURSUR 4/URS-5nOUn[R"USSS9upn U(a!URU5RSW5nU(aCUV s/sH2n XRS /[US 5-5-[U 5-PM4 n n XX4$XU4$s sn f) zIuse np.unique to get integer group indices for product, intersection C)orderT) return_indexreturn_inverse%sr) isinstancetuplenp column_stackasarraydtypendimshapeflags c_contiguousarrayviewuniquereshapejoinlen) groupsprefixsep return_labelsdtis2dncolsgroups_uniuni_idxuni_inviilabels oC:\Users\julio\OneDrive\Documentos\Trabajo\Ideas Frescas\venv\Lib\site-packages\statsmodels/tools/grouputils.pycombine_indicesr/%s:&%  (F# B KK1 D  Q||((XXfC0F++FLL12V\\!_DEIIgD59;C' hhrl""2u- R4&SV"455rB  ++$$  s!9E$c 2[R"U5n[R"U5R5nURS:Xa USS2S4nO"URS:aU(a [ S5eU(a[R "U5SUR S-:a[R"U5Sn[R"[UR S5Vs/sHn[R"XSS2U4S9PM sn5$[R"U5n[R"[U5/[UR SS5-5n[!U5HupgXU:HR#S5XV'M U$s snf)zsimple bincount version, again group : ndarray, integer assumed to be consecutive integers no dtype checking because I want to raise in that case uses loop over columns of x for comparison, simple python loop r Nrznot implemented yetr)weights)rrsqueezer ValueErrormaxrpd factorizerrangebincountrzerosr list enumeratesum)xgroup use_bincountcoluniquesresultr,cats r. group_sumsrDPsD 1 A JJu  % % 'Evv{ agJ ! .// 66%=1qwwqz> )LL'*Exx!, ,C EQV95,   ))E"3w<.4 +<<= )GBC<,,Q/FJ*  s+%Fc[R"US5(a![R"URU5$URU-$)z]sum by groups given group dummy variable group_dummy can be either ndarray or sparse matrix N) data_util_is_using_ndarray_typerdotT)r= group_dummys r.group_sums_dummyrKws; '' T::vvacc;''ss[  cSSKJn [R"[ U5S-5n[R "[ U5[R S9nURX0U45nU$)acreate a sparse indicator from a group array with integer labels Parameters ---------- groups : ndarray, int, 1d (nobs,) an array of group indicators for each observation. Group levels are assumed to be defined as consecutive integers, i.e. range(n_groups) where n_groups is the number of group levels. A group level with no observations for it will still produce a column of zeros. Returns ------- indi : ndarray, int8, 2d (nobs, n_groups) an indicator array with one row per observation, that has 1 in the column of the group level for that observation Examples -------- >>> g = np.array([0, 0, 2, 1, 1, 2, 0]) >>> indi = dummy_sparse(g) >>> indi <7x3 sparse matrix of type '' with 7 stored elements in Compressed Sparse Row format> >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [0, 1, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) current behavior with missing groups >>> g = np.array([0, 0, 2, 0, 2, 0]) >>> indi = dummy_sparse(g) >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [1, 0, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) r)sparser )r)scipyrNraranger onesint8 csr_matrix)r!rNindptrdataindis r. dummy_sparserWsP\ YYs6{1} %F 773v;bgg .D   dF3 4D KrLcT\rSrSrS SjrSrSrSS\4SjrSr S S jr S S jr S r g)GroupcX l[U5up4nX4UsUlUlUl[ UR5UlSUlURUlUR(aURS-Ulgg)N.=) namer/ group_intr*r)r n_groups separatorr")selfr>r^r)r*r+s r.__init__Group.__init__sm / 6g25w. dhDHH  ii ;;+++DK rLcB[R"UR5$N)rr8r_rbs r.counts Group.countss{{4>>**rLc 2URnURnURnURS:aAUVs/sH2nXR S/[ US5-5-[ U5-PM4 nnU$UVs/sH oASU--PM nnU$s snfs snf)Nr rr)r"r)rarrr r)rbr"r)r#r,r-s r.labels Group.labelsshhnn 88a<"$"XXtfSQ[&899U2YF" $ 366#BdRi'#E6 $7s 9B:BNFcURnUb[[U55nXQ XEnURnU(d"USS2S4USSS24:HR U5$[ UR 5$)zT drop_idx is only available if sparse=False drop_idx is supposed to index into uni N)r)rr r>astyperWr_)rbdrop_idxrNrr)idxr>s r.dummy Group.dummysn hh  S"C (C !T'Nc$'l2::5A A/ /rLcr[XR5(a URnURX45$rf)r __class__r>)rbothers r. interactionGroup.interactions+ e^^ , ,KKE~~tm,,rLc*[XRUS9$N)r?)rDr_)rbr=r?s r.rDGroup.group_sumss!^^,GGrLc[[U55n[X- URUS9nXUR- nXT4$ry)floatr rDr_)rbr=r?nobsmeans_g x_demeaneds r. group_demeanGroup.group_demeans@SV}QXt~~*6800 ""rL)r_r`r^r"rar)r*r T) __name__ __module__ __qualname____firstlineno__rcrhrkintrqrvrDr__static_attributes__rLr.rYrYs/,$+ "%s0&- H#rLrYc8^\rSrSrSU4SjjrSrSrSrU=r$) GroupSortedic>[URU] XS9 [R"[R "U55SS-R 5n[S/U-U[U5/-5Ul g)Nr^rr ) superrtrcrnonzerodifftolistrr groupidx)rbr>r^rprts r.rcGroupSorted.__init__sa dnnd,U,>zz"''%.)!,Q.668aS3Ys5zl(:; rLc#P# URHup[X5v M g7frf)rslice)rblowupps r. group_iterGroupSorted.group_iter s HC/ !&s$&cf[R"UR5SS2S4U- nX:*nX#$)areturn the index array for lagged values Warning: if k is larger then the number of observations for an individual, then no values for that individual are returned. TODO: for the unbalanced case, I should get the same truncation for the array with lag=0. From the return of lag_idx we would not know which individual is missing. TODO: do I want the full equivalent of lagmat in tsa? maxlag or lag or lags. not tested yet Nr )rrr)rblaglag_idxmask_oks r. lag_indicesGroupSorted.lag_indicess6**T]]+AqD1C7>rL)rr) rrrrrcrrr __classcell__)rts@r.rrs< "  rLrcUSn[U[[[R[ R [ R45(agg)z Checks if the first item of an array-like object is also array-like If so, we have a MultiIndex and returns True. Else returns False. rTF)rr:rrndarrayr5Series DataFrame)r=items r._is_hierarchicalr$s7 Q4D$ubjj"))R\\JKKrLc0[R"U/SU06$)Nnames)r from_tuplesindexrs r._make_hierarchical_indexr1s  ! !E7 8% 88rLc[UR5n[[[U555n[U5Vs/sHnSU-S-R U5PM sn$s snf)Nz group{0:0})r rstrr7format)rn_namespadis r._make_generic_namesr5sS%++G c#g, C5:7^ D^[_S ( ( +^ DD Ds!A$c\rSrSrSSjr\S5r\S5r\S5r\S5r SSjr SS jr SS jr SS jr SS jrSS jrSSjrSSjrSrSSjrSSjrSrg)Groupingi;NcB[U[[45(a2Ub([US5(aUR USS9 OX!lXlO[U5(a[X5UlO[XS9UlUc[[UR 5n[UR S5(aUR R USS9 OX R l[UR 5Ul [UR R 5Ul SUl g)a index : index-like Can be pandas MultiIndex or Index or array-like. If array-like and is a MultipleIndex (more than one grouping variable), groups are expected to be in each row. E.g., [('red', 1), ('red', 2), ('green', 1), ('green', 2)] names : list or str, optional The names to use for the groups. Should be a str if only one grouping variable is used. Notes ----- If index is already a pandas Index then there is no copy. N set_namesTinplacer)rrrhasattrrrrrrrr r}nlevelsslicesrbrrs r.rcGrouping.__init__<s eeZ0 1 1 5+..OOE4O8"'KJ&&5eC "55 }+DJJ74::{33JJ(((=',JJ$ O 4::++,  rLc[URS5(aURR$URR$)Nlevshape)rrrrrgs r. index_shapeGrouping.index_shapebs3 4::z * *::&& &::## #rLc[URS5(aURR$[R"UR5R$)Nlevels)rrrr5 Categoricalrgs r.rGrouping.levelsis< 4::x ( (::$$ $>>$**-44 4rLc[URSS5nUc`[URS5(aURRnU$[R "UR5R SnU$)Ncodesrk)getattrrrrkr5rr)rbrs r.rkGrouping.labelspsf GT2 =tzz8,, )) tzz288> rLc.URR$rfrrgs r. group_namesGrouping.group_names{szzrLc8Uc URn[X5ng)z Resets the index in-place. N)rrrs r.reindexGrouping.reindexs =$$E%rLcURRU5R5n[R"U5nUR 5 [ UR[5(a3UVs/sHnURRX1S9SPM! snUl gUVs/sHo0RRU5PM snUl gs snfs snf)z Sets the slices attribute to be a list of indices of the sorted groups for the first index level. I.e., self.slices[0] is the index where each observation is in the first (sorted) group. levelrN) rget_level_valuesrrrsortrr get_loc_levelrget_loc)rbrr!r=s r. get_slicesGrouping.get_slicess,,U3::<&! djj* - -$*,$*q ::33A3CAF$*,DK;AA&Q::--a0&ADK,Bs 3&C&$CcT[R"URU5Ulg)zQ Sets the attribute counts to equal the bincount of the (integer-valued) labels. N)rr8rkrhrbrs r.count_categoriesGrouping.count_categoriess kk$++e"45 rLcU(d URnU(al[R"[[ U55US9nUR 5nURR UR5(d [S5eU(a2[ U5[ UR55:wa [S5egg)z Sanity checksrzData is not be sortedzDuplicate index entriesN) rr5rrr requals Exceptionr)rb is_sortedrrtest test_sorteds r. check_indexGrouping.check_indexsJJE <<s5z 2%@D))+K::$$[%6%677 788 5zS00 9::1 rLcUc URn[R"US5(a{URS:Xa&[R "XSS9nUR 5nO#[R"XS9nUR SS9n[R"U5UR4$[R"US5(a0UnURU5nUR 5nX3R4$Sn[U5e) zApplies a (potentially hierarchical) sort operation on a numpy array or pandas series/dataframe based on the grouping index or a user-supplied index. Returns an object of the same type as the original data as well as the matching (sorted) Pandas index. Nr T)rcopyrFrz7data must be a Numpy array or a Pandas Series/DataFrame) rrFrGrr5r sort_indexrrr_is_using_pandasrr3)rbrUroutmsgs r.r Grouping.sorts =JJE  + +D$ 7 7yyA~ii=nn&ll45nnUn388C=#))+ +  ' 'd 3 3C++e$C.."C > !KCS/ !rLc URSUR:wa [S5eURUS9R"U40UD6nSUR;a[ R "U5$[ R"U5$)z]Apply function to each column, by group Assumes that the dataframe already has a proper indexrz/dataframe does not have the same shape as indexrr )rr}rgroupbyapplyrravelr)rb dataframefunctionrkwargsrs r.transform_dataframeGrouping.transform_dataframeso ??1  *MN Ne,228FvF  >88C= 88C= rLc URSUR:wa [S5e[R"XR S9nUR "XR4SU0UD6$)z0Apply function to each column, by group r+array does not have the same shape as indexrr)rr}rr5rrr)rbrrrrrs r.transform_arrayGrouping.transform_arraysZ ;;q>TYY &IJ JLLjj9 '' 252*02 2rLc [R"U5nURSUR:wa [ S5eUR US9 /nUR HInURS:Xa XSS24nOURS:XaXnURU"WU40UD65 MK [R"U5nURSURS5$)zApply function to each group. Similar to transform_array but does not coerce array to a DataFrame and back and only works on a 1D or 2D numpy array. function is called function(group, group_idx, **kwargs). rrrrNr r) rrrr}rrrrappendrr)rbrrrr processedssubsets r.transform_slicesGrouping.transform_slicess  5! ;;q>TYY &IJ J e$ AzzQ!tq   Xfa:6: ;  HHY'   Y__R%899rLc8URSS9 UR$)Nr rrW_dummiesrgs r. dummies_timeGrouping.dummies_times "}}rLc8URUS9 UR$)Nrrrs r.dummies_groupsGrouping.dummies_groupss &}}rLc@[URU5nX lg)acreate a sparse indicator from a group array with integer labels Parameters ---------- groups : ndarray, int, 1d (nobs,) An array of group indicators for each observation. Group levels are assumed to be defined as consecutive integers, i.e. range(n_groups) where n_groups is the number of group levels. A group level with no observations for it will still produce a column of zeros. Returns ------- indi : ndarray, int8, 2d (nobs, n_groups) an indicator array with one row per observation, that has 1 in the column of the group level for that observation Examples -------- >>> g = np.array([0, 0, 2, 1, 1, 2, 0]) >>> indi = dummy_sparse(g) >>> indi <7x3 sparse matrix of type '' with 7 stored elements in Compressed Sparse Row format> >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [0, 1, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) current behavior with missing groups >>> g = np.array([0, 0, 2, 0, 2, 0]) >>> indi = dummy_sparse(g) >>> indi.todense() matrix([[1, 0, 0], [1, 0, 0], [0, 0, 1], [1, 0, 0], [0, 0, 1], [1, 0, 0]], dtype=int8) N)rWrkr)rbrrVs r.rWGrouping.dummy_sparses^DKK./ rL)rrhrrr}rrf)NN)r)TTN)rrrrrcpropertyrrrkrrrrrrrrrrr rWrrrLr.rr;s$L$$ 55   &B 6 ;"4 !2:*0rLr)r r\Fr)__doc__statsmodels.compat.pythonrrnumpyrpandasr5statsmodels.tools.datatoolsrUrFrrr/rDrKrWrYrrrrrrrLr.rsk83**$'%V$N!4nE#E#P % B 9E nnrL