- A+
所属分类：python
1.1. 前言
我们计算外链点击数经过这四个步骤:
Mapper: 将以行数据解析成 key=外链点击数 value=1的形式
Shuffle: 通过Shuffle后的结果会生成以 key 的值排序的 value迭代器
结果如: 外链点击数 [1, 1, 1 ... 1, 1]
Reduce 1: 在这边我们计算出 外链点击数 的访问量
输出如: None [sum([1, 1, 1 ... 1, 1]), key]
Reduce 2: 对sum([1, 1, 1 ... 1, 1]) 进行排序并输出 TOP 100
输入如: 331136 "www.ttmark.com"
1.2. 代码
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
more mr_url_ref.py # -*- coding: utf-8 -*- from mrjob.job import MRJob from mrjob.step import MRStep from ng_line_parser import NgLineParser import heapq class MRUrlRef(MRJob): ng_line_parser = NgLineParser() def mapper(self, _, line): self.ng_line_parser.parse(line) yield self.ng_line_parser.reference_url, 1 # 外链域名 def reducer_sum(self, key, values): """统计 VU""" yield None, [sum(values), key] def reducer_top100(self, _, values): """访问数降序""" for cnt, path in heapq.nlargest(100, values): yield cnt, path def steps(self): return [ MRStep(mapper=self.mapper, reducer=self.reducer_sum), MRStep(reducer=self.reducer_top100) ] def main(): MRUrlRef.run() if __name__ == '__main__': main() |
运行统计和输出结果
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
python mr_url_ref.py < www.ttmark.com.access.log No configs found; falling back on auto-configuration Creating temp directory /tmp/mr_url_ref.root.20160924.141646.495410 Running step 1 of 2... reading from STDIN Running step 2 of 2... Streaming final output from /tmp/mr_url_ref.root.20160924.141646.495410/output... 574546 "-" 331136 "www.ttmark.com" 32990 "m.baidu.com" ...... 5 "www.seo3.org" 5 "www.paixie.net" 5 "www.google.ro" Removing temp directory /tmp/mr_url_ref.root.20160924.141646.495410... |
昵称: HH
QQ: 275258836
ttlsa群交流沟通(QQ群②: 6690706 QQ群③: 168085569 QQ群④: 415230207（新） 微信公众号: ttlsacom)
感觉本文内容不错，读后有收获？
微信公众号
扫一扫关注运维生存时间公众号，获取最新技术文章~
