这篇groupby写的不好。太复杂了。其实实际上经常用的就那么几个。举个例子,把常用的往那一放就很容易理解和拿来用了。日后再写一篇。
groupby功能:分组
groupby + agg(聚集函数们): 分组后,对各组应用一些函数,如’sum’,‘mean’,‘max’,‘min’…
groupby默认纵方向上分组,axis=0
DataFrame
import pandas as pd
import numpy as np
df = pd.DataFrame({'key1':['a', 'a', 'b', 'b', 'a'],
'key2':['one', 'two', 'one', 'two', 'one'],
'data1':np.random.randn(5),
'data2':np.random.randn(5)})
print(df)
data1 data2 key1 key2
0 -0.410122 0.247895 a one
1 -0.627470 -0.989268 a two
2 0.179488 -0.054570 b one
3 -0.299878 -1.640494 b two
4 -0.297191 0.954447 a one
分组,并对分组进行迭代
list(df.groupby(['key1']))#list后得到:[(group1),(group2),......]
[('a', data1 data2 key1 key2
0 -0.410122 0.247895 a one
1 -0.627470 -0.989268 a two
4 -0.297191 0.954447 a one), ('b', data1 data2 key1 key2
2 0.179488 -0.054570 b one
3 -0.299878 -1.640494 b two)]
list后得到:[(group1),(group2),…]
每个数据片(group)格式: (name,group)元组
1. 按key1(一个列)分组,其实是按key1的值
groupby对象支持迭代,产生一组二元元组:(分组名,数据块),(分组名,数据块)…
for name,group in df.groupby(['key1']):
print(name)
print(group)
a
data1 data2 key1 key2
0 -0.410122 0.247895 a one
1 -0.627470 -0.989268 a two
4 -0.297191 0.954447 a one
b
data1 data2 key1 key2
2 0.179488 -0.054570 b one
3 -0.299878 -1.640494 b two
2. 按[key1, key2](多个列)分组
对于多重键,产生的一组二元元组:((k1,k2),数据块),((k1,k2),数据块)…
第一个元素是由键值组成的元组
for name,group in df.groupby(['key1','key2']):
print(name) #name=(k1,k2)
print(group)
('a', 'one')
data1 data2 key1 key2
0 -0.410122 0.247895 a one
4 -0.297191 0.954447 a one
('a', 'two')
data1 data2 key1 key2
1 -0.62747 -0.989268 a two
('b', 'one')
data1 data2 key1 key2
2 0.179488 -0.05457 b one
('b', 'two')
data1 data2 key1 key2
3 -0.299878 -1.640494 b two
3. 按函数分组
4. 按字典分组
5. 按索引级别分组
6.将函数跟数组、列表、字典、Series混合使用也不是问题,因为任何东西最终都会被转换为数组
将这些数据片段做成字典
dict(list(df.groupby(['key1'])))#dict(list())
{'a': data1 data2 key1 key2
0 -0.410122 0.247895 a one
1 -0.627470 -0.989268 a two
4 -0.297191 0.954447 a one, 'b': data1 data2 key1 key2
2 0.179488 -0.054570 b one
3 -0.299878 -1.640494 b two}
分组后进行一些统计、计算等
1. 分组后,返回一个含有分组大小的Series
按key1分组
df.groupby(['key1']).size()
key1
a 3
b 2
dtype: int64
dict(['a1','x2','e3'])
{'a': '1', 'e': '3', 'x': '2'}
按[key1,key2]分组
df.groupby(['key1','key2']).size()
key1 key2
a one 2
two 1
b one 1
two 1
dtype: int64
2. 对data1按key1进行分组,并计算data1列的平均值
df['data1'].groupby(df['key1']).mean()
#groupby没有进行任何的计算。它只是进行了一个分组
key1
a -0.444928
b -0.060195
Name: data1, dtype: float64
df.groupby(['key1'])['data1'].mean()#理解:对df按key1分组,并计算分组后df['data1']的均值
#等价于:df.groupby(['key1']).data1.mean()
key1
a -0.444928
b -0.060195
Name: data1, dtype: float64
说明:
groupby没有进行任何的计算。它只是进行了一个分组。
数据(Series)根据分组键进行了聚合,产生了一个新的Series,其索引为key1列中的唯一值。
这种索引操作所返回的对象是一个已分组的DataFrame(如果传入的是列表或数组)或已分组的Series
df.groupby(['key1'])['data1'].size()
key1
a 3
b 2
Name: data1, dtype: int64
3.对data1按[key1,key2]进行分组,并计算data1的平均值
df['data1'].groupby([df['key1'],df['key2']]).mean()
key1 key2
a one -0.353657
two -0.627470
b one 0.179488
two -0.299878
Name: data1, dtype: float64
df.groupby(['key1','key2'])['data1'].mean()
#等价于:df.groupby(['key1','key2']).data1'.mean()
key1 key2
a one -0.353657
two -0.627470
b one 0.179488
two -0.299878
Name: data1, dtype: float64
通过两个键对数据进行了分组,得到的Series具有一个层次化索引(由唯一的键对组成):
df.groupby(['key1','key2'])['data1'].mean().unstack()
key2 |
one |
two |
key1 |
|
|
a |
-0.353657 |
-0.627470 |
b |
0.179488 |
-0.299878 |
在上面这些示例中,分组键均为Series。实际上,分组键可以是任何长度适当的数组。非常灵活。
横方向上
按列的数据类型(df.dtypes)来分
df共两种数据类型:float64和object,所以会分为两组(dtype(‘float64’),数据片),(dtype(‘O’), 数据片)
list(df.groupby(df.dtypes, axis=1))
[(dtype('float64'), data1 data2
0 -0.410122 0.247895
1 -0.627470 -0.989268
2 0.179488 -0.054570
3 -0.299878 -1.640494
4 -0.297191 0.954447), (dtype('O'), key1 key2
0 a one
1 a two
2 b one
3 b two
4 a one)]
agg的应用
groupby+agg 可以对groupby的结果同时应用多个函数
SeriesGroupBy的方法agg()参数:
aggregate(self, func_or_funcs, * args, ** kwargs)
func: function, string, dictionary, or list of string/functions
返回:aggregated的Series
s= pd.Series([10,20,30,40])
s
0 10
1 20
2 30
3 40
dtype: int64
for n,g in s.groupby([1,1,2,2]):
print(n)
print(g)
1
0 10
1 20
dtype: int64
2
2 30
3 40
dtype: int64
s.groupby([1,1,2,2]).min()
1 10
2 30
dtype: int64
#等价于这个:
s.groupby([1,1,2,2]).agg('min')
1 10
2 30
dtype: int64
s.groupby([1,1,2,2]).agg(['min','max'])#加[],func仅接受一个参数
常常这样用:
df
|
data1 |
data2 |
key1 |
key2 |
0 |
-0.410122 |
0.247895 |
a |
one |
1 |
-0.627470 |
-0.989268 |
a |
two |
2 |
0.179488 |
-0.054570 |
b |
one |
3 |
-0.299878 |
-1.640494 |
b |
two |
4 |
-0.297191 |
0.954447 |
a |
one |
比较下面,可以看出agg的用处:
df.groupby(['key1'])['data1'].min()
key1
a -0.627470
b -0.299878
Name: data1, dtype: float64
df.groupby(['key1'])['data1'].agg({'min'})
|
min |
key1 |
|
a |
-0.627470 |
b |
-0.299878 |
#推荐用这个√
df.groupby(['key1']).agg({'data1':'min'})#对data1列,取各组的最小值,名字还是data1
|
data1 |
key1 |
|
a |
-0.627470 |
b |
-0.299878 |
#按key1分组后,aggregate各组data1的最小值和最大值:
df.groupby(['key1'])['data1'].agg({'min','max'})
|
max |
min |
key1 |
|
|
a |
-0.297191 |
-0.627470 |
b |
0.179488 |
-0.299878 |
#推荐用这个√
df.groupby(['key1']).agg({'data1':['min','max']})
|
data1 |
|
min |
max |
key1 |
|
|
a |
-0.627470 |
-0.297191 |
b |
-0.299878 |
0.179488 |
可以对groupby的结果更正列名(不推荐用这个,哪怕在后面单独更改列名)
# 对data1,把min更名为a,max更名为b
df.groupby(['key1'])['data1'].agg({'a':'min','b':'max'})#这里的'min' 'max'为两个函数名
d:\python27\lib\site-packages\ipykernel_launcher.py:2: FutureWarning: using a dict on a Series for aggregation
is deprecated and will be removed in a future version
|
a |
b |
key1 |
|
|
a |
-0.627470 |
-0.297191 |
b |
-0.299878 |
0.179488 |