每日PV-MRJob-Python数据分析(2)

HH python每日PV-MRJob-Python数据分析(2)已关闭评论11,2631字数 1411阅读4分42秒阅读模式

1.1. 前言

这边我们使用Python的M/R框架MRJob来分析

1.2. M/R步骤

M/R分析数据步骤一般有 3 步:文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

Mapper => Shuffle => Reduce文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

Mapper: 一般是解析文件中的每一行数据并生成自定义的 key 和 value 输出文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

Shuffle: 是一个 patition、sort、combine 组成的。一般这个过程是自动进行的。当然我们也可以定义或重写方法来实现自己的 Shuffle。文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

Reduce: 接收 Mapper 和 Shuffle 输出的 key和value迭代器 计算出我们需要的结构。Reduce 可以有多个文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

 文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

1.3. 如我们计算每日PV经过这三个步骤

Mapper: 将以行数据解析成 key=YYYY-MM-dd value=1的形式文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

Shuffle: 通过Shuffle后的结果会生成以 key 的值排序的 value迭代器文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

结果如: 2016-09-24 [1, 1, 1 ... 1, 1]文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

Reduce: 在这边我们计算出 2016-09-24 是访问量文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

输出如: 2016-09-24 sum([1, 1, 1 ... 1, 1])文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

1.4. 代码

cat mr_pv_day.py
# -*- coding: utf-8 -*-
 
from mrjob.job import MRJob
from ng_line_parser import NgLineParser
 
class MRPVDay(MRJob):
 
    ng_line_parser = NgLineParser()
 
    def mapper(self, _, line):
        self.ng_line_parser.parse(line)
        dy, tm = str(self.ng_line_parser.access_time).split()
        yield dy, 1 # 每一天的
        yield 'total', 1 # 所有的
 
    def reducer(self, key, values):
        yield key, sum(values)
 
 
def main():
    MRPVDay.run()
 
if __name__ == '__main__':
    main()

运行统计和输出结果文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

python mr_pv_day.py < www.ttmark.com.access.log
 
No configs found; falling back on auto-configuration
Creating temp directory /tmp/mr_pv_day.root.20160924.122246.076809
Running step 1 of 1...
reading from STDIN
Streaming final output from /tmp/mr_pv_day.root.20160924.122246.076809/output...
"2016-06-13"    4149
"2016-06-14"    10234
"2016-06-15"    9825
......
"2016-09-16"    11076
"2016-09-17"    10231
"2016-09-18"    6739
"total" 988839
Removing temp directory /tmp/mr_pv_day.root.20160924.122246.076809...

昵称: HH文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

QQ: 275258836文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

ttlsa群交流沟通(QQ群②: 6690706 QQ群③: 168085569 QQ群④: 415230207(新) 微信公众号: ttlsacom)文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

感觉本文内容不错,读后有收获?文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

逛逛衣服店,鼓励作者写出更好文章。文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/ 文章源自运维生存时间-https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/

weinxin
我的微信
微信公众号
扫一扫关注运维生存时间公众号,获取最新技术文章~
HH
  • 本文由 发表于 27/10/2016 00:47:17
  • 转载请务必保留本文链接:https://www.ttlsa.com/python/python-big-data-analysis-every-day-pv-mrjob/