upyun 存储文件迁移到七牛

要将upyun上存储的文件迁移到七牛存储上，不管出于何种目的，多多少少会遇到这种需求的。

方法如下：文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

1. 列出upyun空间上所有文件
2. 通过七牛提供的qfetch工具上传到七牛

列出upyun文件

vi upyun_list.py
#! /usr/bin/env python
# -*- coding: utf-8 -*-

from base64 import b64encode
import requests
import urllib
import Queue
import sys

bucket = sys.argv[1]
username = 'ttlsa'
password = 'www.ttlsa.com'

queue = Queue.LifoQueue()
queue_list = Queue.LifoQueue()
count = 0

def do_http_request(method, key, upyun_iter):
    uri = '/' + bucket + (lambda x: x[0] == '/' and x or '/' + x)(key)
    if isinstance(uri, unicode):
        uri = uri.encode('utf-8')
    uri = urllib.quote(uri)
    headers = {}
    headers['Authorization'] = "Basic " + b64encode(username + ':' + password)
    headers['User-Agent'] = "uptechs"
    headers['X-List-Limit'] = 300
    if method is not 'DELETE':
        if upyun_iter is not None or upyun_iter is not 'g2gCZAAEbmV4dGQAA2VvZg':
            headers['x-list-iter'] = upyun_iter

    url = "http://v0.api.upyun.com" + uri
    requests.adapters.DEFAULT_RETRIES = 5
    session = requests.session()
    try:
        response = session.request(method, url, headers=headers, timeout=30)
        status = response.status_code
        if status == 200 and method != 'DELETE':
            content = response.content
            try:
                iter_header = response.headers['x-upyun-list-iter']
            except Exception as e:
                iter_header = 'g2gCZAAEbmV4dGQAA2VvZg'
            return content + "`" + str(iter_header)
        elif status == 200 and method == 'DELETE':
            return True
        else:
            print 'status: ' + str(status) + '--->' + url
            print 'message: ' + str(response.headers['X-Error-Code'])
            print 'message: ' + str(response.content)
            return None
    except Exception as e:
        pass

def getlist(key, upyun_iter):
    content = do_http_request('GET', key, upyun_iter)
    if not content:
        return None
    content = content.split("`")
    items = content[0].split('\n')
    content = [dict(zip(['name', 'type', 'size', 'time'],
                        x.split('\t'))) for x in items] + content[1].split()
    return content

def print_file_with_iter(path):
    upyun_iter = None
    while True:
        while upyun_iter != 'g2gCZAAEbmV4dGQAA2VvZg':
            res = getlist(path, upyun_iter)
            if res:
                upyun_iter = res[-1]
                for i in res[:-1]:
                    try:
                        if not i['name']:
                            continue
                        new_path = path + i['name'] if path == '/' else path + '/' + i['name']
                        if i['type'] == 'F':
                            queue.put(new_path)
                            queue_list.put(new_path)
                        elif i['type'] == 'N':
                            print sys.argv[3] + new_path
                            with open(bucket + '_file.txt', 'a') as f:
                                f.write(sys.argv[3] + new_path + '\n')
                    except Exception as e:
                        print e
        else:
            if not queue.empty():
                path = queue.get()
                upyun_iter = None
                queue.task_done()
            else:
                break

if __name__ == '__main__':
    path = sys.argv[2]
    print_file_with_iter(path)

执行upyun_list.py文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

# python upyun_list.py  ttlsa-files  /  http://imgs.ttlsa.com文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

输出的内容格式如下文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

http://imgs.ttlsa.com/201601/142135564177453.png文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

http://imgs.ttlsa.com/201601/142137469838888.png文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

http://imgs.ttlsa.com/201601/142137554414262.png文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

同时，在本地目录保存在ttlsa-files_file.txt文件中。文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/

上传到七牛

# qfetch_linux_amd64  -ak=$qiniu_ak -sk=$qiniu_sk -bucket="upyun-ttlsa-files" -file="ttlsa-files_file.txt"  -job="upyun_ttlsa-files" -worker=2

qfetch具体选项要参见七牛提供的文档。http://developer.qiniu.com/code/v6/tool/qfetch.html文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/ 文章源自运维生存时间-https://www.ttlsa.com/python/upyun-store-files-migrate-to-qiniu/