自动化运维管理fabric

自动化日常管理任务和部署">如何使用fabric 自动化日常管理任务和部署

自动化，批量化是作为管理员，或者运维人员必须面临的问题。自动化和批量化也有很多方式，可以用单一工具也可以自己写shell脚本，甚至可以开发出来一套完备的任务管理系统。其实我们大多时候可以在一台主机上面通过ssh来控制所有机器，来完成我们的任务工作。是否有这样的工具来支持我们呢？

文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

fabric 常用接口

fabric是对ssh的一个集成工具，对我们而言只需要使用相应的接口，来高效的完成工作，我们常用到的功能基本是 ： 本地或者远端执行命令， 分发文件，收集文件，还有一些权限相关的操作。 这些fabric都给我们提供了对应的接口。
如下所示：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

run (fabric.operations.run)
sudo (fabric.operations.sudo)
local (fabric.operations.local)
get (fabric.operations.get)
put (fabric.operations.put)
prompt (fabric.operations.prompt)
reboot (fabric.operations.reboot)

fabric 还提供了上下文管理器

接口部分提供了命令运行的方式，不过都无法保持上下文关系，为了解决这个问题，fabric的context manager 就派上了用场：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

cd (fabric.context_managers.cd)
lcd (fabric.context_managers.lcd)
path (fabric.context_managers.path)
settings (fabric.context_managers.settings)
prefix (fabric.context_managers.prefix)

fabric 安装

easy_install fabric

fabric 编程模型介绍

由于fabric是基于python的，所以写fabric脚本就是写python脚本，你可以像写python脚本一样，可以依赖其他模块或者其他工具来完成工作。Fabric 脚本，通过fab工具运行fabric python脚本。fab工具默认执行fabfile.py ,也可以通过-f 参数指定 脚本文件名。fabric优势多多，简单，方便，日志输出清晰，命令
中可以使用AWK 命令 下面我们看一个 hello world 程序。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

from fabric.api import *
def helloworld(who='world'):
    print  "Hello {0}!".format(who) 
def helloworld1(you='world',me='ruiaylin'):
    print  "Hello {0}! i am {1} ! ".format(you,me)

执行命令(其中参数的传递直接跟在任务后跟变量名和参数)：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

➜  fabric  fab -f helloword.py  helloworld
Hello world! 
Done.
➜  fabric  fab -f helloword.py  helloworld1:you='ruichao',me='ruiaylin'
Hello ruichao! i am ruiaylin ! 
Done.

fabric主要接口方法

我们已经看了一个简单例子下面我们来看一下fabric的主要接口。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

run (fabric.operations.run)

Fabric 中使用最多的就是 run 方法了。run是用来在一台或者多台远程主机上面执行shell 命令。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

• 方法的返回值是可以通过变量来进行捕获
• 可以通过变量的.failed 和 .succeeded 来检查命令是否执行成功
• 还有一个很赞的就是 run 方法中执行命令的时候，可以支持awk 很给力

使用方法：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

# creat a directory
run(" mkdir /tmp/testdir/ -p ")
# check process
result = run("ps -ef |grep mysqld|grep -v safe |grep -v grep  | wc -l "
#Check if command
result.failed

sudo (fabric.operations.sudo)

使用 sudo 命令执行对顶的命令。使用方法与run 类似。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

local (fabric.operations.local)

local 命令是执行本机的命令或者脚本.使用方法和run 还有sudo类似，但是有一个区别
就是： 捕获结果的时候，是通过指定 capture=False 或者capture=True来确定。来看
实例：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

# example like this ： 
def helloworld(who='world'):
    print  "Hello {0}!".format(who) 
    yy = local(" pwd ", capture=False)
    print 'start :  yy = ' , yy , ' : ::  ',yy.succeeded
    zz = local(" pwd ", capture=True)
    print 'start :  zz = ' , zz , ' : ::  ',zz.succeeded
#result ：
➜  fabric  fab -f helloword.py  helloworld  -H 10.211.55.3 -u root
[10.211.55.3] Executing task 'helloworld'
Hello world!
[localhost] local:  pwd 
/Users/ruiaylin/Documents/workpython/fabric
start :  yy =    : ::   True
[localhost] local:  pwd 
start :  zz =  /Users/ruiaylin/Documents/workpython/fabric  : ::   True

get (fabric.operations.get)

get 方法是从远程主机 copy file 到本地,功能跟scp一样。可以从远程主机下载
备份，或者日志文件等等。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

• 通过参数 remote_path 指定远程文件的路径
• 通过参数 local_path 指定远程文件的路径

使用方法如下：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

# Download some logs
get(remote_path="/tmp/xxx.log", local_path="/tmp/xxx.log")  
# Download a database back-up
get("/backup/db.gz", "./db.gz")

put (fabric.operations.put)

某些需要上传和分发文件的时候，put命令就派上了用场，使用方式类似 get。也同样可以
通过.failed .succeeded进行命令是否执行成功的判断。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

• local_path - 本地路径
• remote_path - 远程路径
• mode - 文件属性

如下例子：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

upload = put("requirements.txt", "requirements.txt", mode=0664)

并行执行

目前官方来看 1.X 版本的fabric 并行执行的时候不是thread safe的。如果需要并行执行task。需要在方法上面使用注解 @parallel 为了防止管控机器上面过多的并发任务可以通过 @parallel(pool_size=5)来设置. 并行的执行输出都会输出到一个终端上面，比较混乱。最好是写到日志，以task为维度。跟下面的代码类似。文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

MySQL 安装实例

安装步骤如下文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

• 获取主机ip
• check主机可达性
• 检查linux平台详情
• 是否有运行的mysql实例
• 如果有获取对应的端口
• 检查是否和要安装的端口冲突
• 处理mysql用户以及属组
• 处理安装相关目录和权限
• copy 安装包到目标机
• 解压处理，将主要软件工具软连接到path路径中
• 生成对应标准配置文件并分发到目标机对应目录
• 初始化数据库
• 启动数据库
• 基本步骤安装完毕

基本脚本如下：文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

script 1 sub task :文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

from fabric.api import *
from fabric.colors import green,red,blue,cyan,yellow
import os , sys
import socket
import datetime
import logging
import logging.handlers
#get logger for logging 
def initLoggerWithRotate():
    logname=''.join(env.host_string.split('.'))+'.log'
    logFileName="logs/%s"%logname
    logger = logging.getLogger("fabric")
    formater = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s","%Y-%m-%d %H:%M:%S")
    file_handler = logging.handlers.RotatingFileHandler(logFileName, maxBytes=104857600, backupCount=5)
    file_handler.setFormatter(formater)
    stream_handler = logging.StreamHandler(sys.stderr)
    logger.addHandler(file_handler)
    logger.addHandler(stream_handler)
    logger.setLevel(logging.INFO)
    return logger
#mkdir
def runmkdir(dir):
    run(''' mkdir -p %s '''%dir)
#stp 1 check host
def checkhost(logger):
     host = env.host_string 
     s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     flag_c = 0
     try:
         s.connect((host, 22))
         flag_c = 1
         logger.info( green( ' --> host %s can be reachable ' %host ) )
     except socket.error as e: 
         logger.warning( yellow( ' --> Error on connect %s' %e ) )
     s.close()
     return flag_c
#stp 2 check alive instance on target host 
def checkmysqlinstance(logger):
    try:
        wc = run(''' ps -ef |grep mysqld|grep  -v safe | grep -v grep | wc -l  ''') 
        if int(wc) > 0  : 
            logger.warning(yellow( ' --> %sinstance exist on the target host  '%wc )) 
            portraw = run('''  ps -ef |grep mysqld|grep -v safe |grep -v grep  |awk ' {for(i=1;i<=NF;i++){if($i ~/--port/ ){print $i}}}' |awk -F '=' '{print $2}'
            ''')
            ports = [x.strip() for x in portraw.split() ]
            logger.warning( yellow( ' --> existing instance port : [ %s ] '%( ','.join( ports ))))
            if port in ports:
                logger.error( red( ' --> Install port %s exist , install failed '%port))
                logger.error( red( ' <<<exit>>>>>  task on host %s stop & exit() '%thost))
                sys.exit()
    except Exception, e:
        logger.warning(yellow( ' --> checkmysqlinstance() exception : %s '%e )) 
        raise e 
#stp 3 initdir for installation
def createUser(logger,user='mysql',group='dba'):
    try:
        if int(run('grep "^mysql" /etc/passwd|wc -l')) == 0 :
            run('groupadd dba ')
            run('useradd -c "mysql software owner" -g dba -G dba mysql')
            run('mkdir -p /home/mysql ; chown -R mysql.dba /home/mysql ')
            logger.info(cyan( ' --> create user [ mysql ] in group [ dba ]  success ' )) 
        else : 
            logger.info(yellow ( ' --> user [ mysql ] in group [ dba ] exist & skip  ' )) 
    except Exception, e:
        logger.warning(yellow( ' --> createUser() exception : %s '%e )) 
        raise e
#stp 4 initail directory for mysql        
def initdir(logger,port=3306):  
    try :
        logger.info( green( ' --> begin to create dirs for installation '))
        datadir='/data/'
        logdir ='/log/'
        mandir = 'mysql%s'%port
        subddir ='/data/mysql%s/{data,log,run,tmp}'%(port)
        subldir ='/log/mysql%s/{binlog,iblog}'%(port) 
        #data
        ck1 = run(' df -vh  | grep  /data | wc -l ')
        if ck1  == 0 : 
            logger.error(green(' --> no /data/ partition exist' ) )
            #sys.exit()
        if int( run(' ls /  | grep  /data | wc -l ')) == 0 or int( run(' ls /data/ | grep -w %s | wc -l '%mandir) ) == 0 : 
            runmkdir(subddir) 
            logger.info(green(' --> /data/*** create Ok ' ) )
        else : 
            logger.info(green(' --> /data/mysql%s exsit '%port ))
            logger.info(green(' --> pls,handle it and restart this task '))
            sys.exit()
        #log 
        ck2 = run(' df -vh | grep /log/  | wc -l  ')
        if int( run(' df -vh | grep /log/  | wc -l  ') ) == 0  and int( run(' ls / | grep -w log  | wc -l  ') ) == 0: 
            logger.warning( yellow(' --> no /log/ partition exist') ) 
            logger.warning( yellow(' --> create link for /log/ --> /data/log/') ) 
            runmkdir('/data/log')
            run('ln -s /data/log  /log ')
            runmkdir(subldir) 
            logger.info(green(' --> /log/*** create Ok ' ) )
        else : 
            if  int(run(' ls /log/ | grep -w %s | wc -l '%mandir)) == 0: 
                runmkdir(subldir) 
                logger.info(green(' --> /log/*** create Ok ' ) )
            else : 
                logger.info(yellow(' --> /log/mysql%s exsit '%port ))
                logger.error(red(' --> pls,handle it and restart this task ' ))
                sys.exit() 
        #change 
        runmkdir('/data/tmp')
        logger.info(green(' --> change dirs owner&privs start'))
        run('chown -R mysql:dba /data/*')
        run('chown -R mysql:dba /log') 
        logger.info(green(' --> change dirs owner&privs done'))
    except Exception, e:
        logger.warning(yellow( ' --> initdir() exception : %s '%e )) 
        raise e 
#stp 5 put mysql install package
def copymysql(logger,version='5.7'): 
    try:
        dits = {
        'ubuntu':'mysql-server_5.6.21-1ubuntu12.04_amd64.deb-bundle.tar',
        'centos':'mysql-server.tar.gz'
        }
        issue = run ('cat /etc/issue') 
        ss = issue.lower()
        logger.info( green( ' %s '%ss))
        if int ( run( ' ls /usr/local/ | grep mysql | wc -l ') ) > 0 : 
            logger.info( yellow( ' --> mysql software installed , skip   ' )) 
            return
        plats = dits.keys()
        for x in plats: 
            if ss.find(x) != -1: 
                logger.info( green( ' --> the target host platform is %s'% x ) )
                put( local_path="configs/%s"%dits[x],remote_path="/tmp/%s"%dits[x] )
                logger.info( green( ' --> tar the ball to prop dir '))
                run( 'tar zxvf /tmp/%s -C /usr/local/ '%dits[x] )
                run( 'ln -s /usr/local/%s  /usr/local/mysql  '%dits[x][:-7] )
                break 
    except Exception, e:
        logger.warning(yellow( ' --> copymysql() exception : %s '%e )) 
        raise e 
#gen my.cnf file 
def getnewServerId(logger,port):  
    host = env.host_string
    print 'getnewServerId : ',host
    pics = host.split('.')
    a=int(pics[0])
    b=int(pics[1])
    c=int(pics[2])
    d=int(pics[3])
    suf = int(port) % 256
    server_id =  b * 256 * 256 * 256 + c * 256 * 256 + d * 256 + suf
    logger.info( cyan( ' --> gen server_id done , %s %s is %s '%( host , port , server_id) ) )
    return server_id
def genmycnf(logger,port=3306,itype='h'):
    host = env.host_string
    bps={
    "a":"48|32|3100|3000",
    "b":"62|40|4600|4500",
    'c':'94|64|7600|7500',
    'd':'94|32|3100|3000',
    'e':'125|75|10100|10000',
    'f':'188|120|15100|15000',
    'g':'188|60|7600|7500',
    'h':'1|256M|800|750'
    } 
    try:
        myfile=''.join(host.split('.'))+'.cnf'
        cpmycnf="""cp configs/my.cnf  tmp/%s """%myfile 
        local( 'rm -f  tmp/%s'%myfile  )
        local("cp configs/my.cnf tmp/%s "%myfile )  
        sid=getnewServerId(logger,port)
        keys=bps.keys()
        bpxs=bps[itype]
        mem,bpsize,maxc,maxuc=bpxs.split('|')
        if bpsize[-1] != "M":
            bpsize = bpsize +'g'
        chrgcmd="""  sed -i -e "s/3306/%s/g" -e "s/server_id=10000/server_id=%s/g" -e "s/=32g/=%s/g" -e "s/max_connections=3100/max_connections=%s/g" -e "s/max_user_connections=3000/max_user_connections=%s/g" tmp/%s """
        local( chrgcmd%(port,sid,bpsize,maxc,maxuc,myfile) ) 
        logger.info( green( ' --> gen my.cnf success  ') )
        logger.info( green( ' --> copy my.cnf to dist host ') )
        put( local_path="tmp/%s"%myfile, remote_path="/data/mysql%s/my.cnf"%(port) )
    except Exception, e:
        logger.warning(yellow( ' --> genmycnf() exception : %s '%traceback.format_exc()  ) ) 
        raise e

script 2 whole task :文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/

import inst_utils
from inst_utils import *
def install_mysql(port):
    logger = initLoggerWithRotate()
    thost = env.host_string
    try:
        logger.info(green( 'stp 1 get the host %s '%thost ))
        #check host reachable  
        rs1 = checkhost(logger )
        if int(rs1)== 0 :
            logger.info(red( 'stp 2 check the host is reachable failed ' ))
        logger.info(green( 'stp 2 check the host is reachable OK ' ))
        plat_type = run(''' uname -o ''')
        if plat_type !=  'GNU/Linux' :
            logger.warning(yellow('stp 3 target platform is not GNU/Linux & exit() '))  
            sys.exit()
        logger.info(green('stp 3 target platform is GNU/Linux')) 
        #check target host exsist mysql instance 
        logger.info(green( 'stp 4 checkmysqlinstance  ' ))
        checkmysqlinstance(logger)
        #create MySQL user 
        logger.info( green( 'stp 5 createUser ' ))
        createUser(logger) 
        put(local_path="configs/bash_profile", remote_path="/home/mysql/.bash_profile")  
        #checking dir
        logger.info( green( 'stp 6 initdir ' ))
        initdir(logger,port) 
        #copy file 
        logger.info( green( 'stp 7 copymysql ' ))
        copymysql(logger)
        logger.info( green( 'stp 8  genmycnf  ') ) 
        genmycnf(logger,port,'h')
    except Exception, e:
        print  'main : exception : ' ,  e

文章源自运维生存时间-https://www.ttlsa.com/python/automation-operation-and-maintenance-tool-fabric/