1.1. Pandas分析步骤
- 载入数据
- 将 CDN_IP请求数 进行 COUNT。类似如下SQL:
SELECT cdn_ip, count(*) FROM log GROUP BY cdn_ip ORDER BY count(*) LIMIT 0, 100;
1.2. 代码
cat pd_ng_log_stat.py #!/usr/bin/env python #-*- coding: utf-8 -*- from ng_line_parser import NgLineParser import pandas as pd import socket import struct class PDNgLogStat(object): def __init__(self): self.ng_line_parser = NgLineParser() def _log_line_iter(self, pathes): """解析文件中的每一行并生成一个迭代器""" for path in pathes: with open(path, 'r') as f: for index, line in enumerate(f): self.ng_line_parser.parse(line) yield self.ng_line_parser.to_dict() def load_data(self, path): """通过给的文件路径加载数据生成 DataFrame""" self.df = pd.DataFrame(self._log_line_iter(path)) def uv_cdn_ip(self, top = 100): """统计cdn ip量""" group_by_cols = ['cdn_ip'] # 需要分组的列,只计算和显示该列 # 直接统计次数 url_req_grp = self.df[group_by_cols].groupby( self.df['cdn_ip']) return url_req_grp.agg(['count'])['cdn_ip'].nlargest(top, 'count') def main(): file_pathes = ['www.ttmark.com.access.log'] pd_ng_log_stat = PDNgLogStat() pd_ng_log_stat.load_data(file_pathes) # 统计 CDN IP 访问量 print pd_ng_log_stat.uv_cdn_ip() if __name__ == '__main__': main()
运行统计和输出结果
python pd_ng_log_stat.py count cdn_ip 140.205.127.2 31943 101.200.101.203 26306 101.200.101.214 24667 ...... 140.205.253.155 4065 140.205.253.174 4048 140.205.253.131 3972 [100 rows x 1 columns]
昵称: HH文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-cdn-ip-req-count-pandas/
QQ: 275258836文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-cdn-ip-req-count-pandas/
ttlsa群交流沟通(QQ群②: 6690706 QQ群③: 168085569 QQ群④: 415230207(新) 微信公众号: ttlsacom)文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-cdn-ip-req-count-pandas/
感觉本文内容不错,读后有收获?文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-cdn-ip-req-count-pandas/
逛逛衣服店,鼓励作者写出更好文章。文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-cdn-ip-req-count-pandas/ 文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-cdn-ip-req-count-pandas/

我的微信
微信公众号
扫一扫关注运维生存时间公众号,获取最新技术文章~
评论