• UID12
  • 登录2016-06-14
  • 粉丝110
  • 关注50
  • 发帖1415
  • 主页
  • 金币8518枚
社区居民
最爱沙发
忠实会员
喜欢达人
原创写手
极分享 发布于2016-06-02 15:11
0/569

Minos 一个分布式的发布和监控系统

楼层直达

Minos 是小米公司开发的一个分布式的发布和监控系统。最初是小米开发的用来在 Hadoop 和 ZooKeeper 集群上发布和管理的工具。Minos 可轻松扩展来支持其他的系统,目前已经支持包括 HDFS、YARN 和 Impala 。

 

源码:https://github.com/XiaoMi/minos

 

What is Minos

Minos is a distributed deployment and monitoring system. It was initially developed and used at Xiaomi to deploy and manage the Hadoop, HBase and ZooKeeper clusters used in the company. Minos can be easily extended to support other systems, among which HDFS, YARN and Impala have been supported in the current release.

Components

The Minos system contains the following four components:

Client

This is the command line client tool used to deploy and manage processes of various systems. You can use this client to perform various deployment tasks, e.g. installing, (re)starting, stopping a service. Currently, this client supports ZooKeeper, HDFS, HBase, YARN and Impala. It can be extended to support other systems. You can refer to the following Using Client to learn how to use it.

Owl

This is the dashboard system to display the status of all processes, where users can take a overview of the whole clusters managed by Minos. It collects data from servers through JMX interface. And it organizes pages in cluster, job and task corresponding to the definition in cluster configuration. It also provides some utils like health alerter, HDFS quota updater and quota reportor. You can refer to Installing Owl to learn how to install and use it.

Supervisor

This is the process management and monitoring system. Supervisor is an open source project, a client/server system that allows its users to monitor and control a number of processes on a UNIX-like operating system.

Based on the version of supervisor-3.0b1, we extended Supervisor to support Minos. We implemented an RPC interface under the deployment directory, so that our deploy client can invoke the services supplied by supervisord.

When deploying a Hadoop cluster for the first time, you need to set up supervisord on every production machine. This only needs to be done once. You can refer to Installing Supervisor to learn how to install and use it.

Tank

This is a simple package management Django app server for our deployment tool. When setting up a cluster for the first time, you should set up a tank server first. This also needs to be done only once. You can refer to Installing Tank to learn how to install and use it.

Setting Up Minos on Centos/Ubuntu

Prerequisites

Install Python

Make sure install Python 2.7 or later from http://www.python.org.

Install JDK

Make sure that the Oracle Java Development Kit 6 is installed (not OpenJDK) fromhttp://www.oracle.com/technetwork/java/javase/downloads/index.html, and that JAVA_HOME is set in your environment.

Building Minos

Clone the Minos repository

To Using Minos, just check out the code on your production machine:

git clone https://github.com/XiaoMi/minos.git

Build the virtual environment

All the Components of Minos run with its own virtual environment. So, before using Minos, building the virtual environment firstly.

cd minos
./build.sh build

Note: If you only use the Client component on your current machine, this operation is enough, then you can refer toUsing Client to learn how to deploy and manage a cluster. If you want to use the current machine as a Tank server, you can refer to Installing Tank to learn how to do that. Similarly, if you want to use the current machine as a Owl server or a Supervisor server, you can refer to Installing Owl and Installing Supervisor respectively.

Installing Tank

Start Tank

cd minos
./build.sh start tank --tank_ip ${your_local_ip} --tank_port ${port_tank_will_listen}

Note: If you do not specify the tank_ip and tank_port, it will start tank server using 0.0.0.0 on 8000 port.

Stop Tank

./build.sh stop tank

Installing Supervisor

Prerequisites

Make sure you have intstalled Tank on one of the production machines.

Start Supervisor

cd minos
./build.sh start supervisor --tank_ip ${tank_server_ip} --tank_port ${tank_server_port}

When starting supervisor for the first time, the tank_ip and tank_port must be specified.

After starting supervisor on the destination machine, you can access the web interface of the supervisord. For example, if supervisord listens on port 9001, and the serving machine's IP address is 192.168.1.11, you can access the following URL to view the processes managed by supervisord:

http://192.168.1.11:9001/

Stop Supervisor

./build.sh stop supervisor

Monitor Processes

We use Superlance to monitor processes. Superlance is a package of plug-in utilities for monitoring and controlling processes that run under supervisor.

We integrate superlance-0.7 to our supervisor system, and use the crashmail tool to monitor all processes. When a process exits unexpectedly, crashmail will send an alert email to a mailing list that is configurable.

We configure crashmail as an auto-started process. It will start working automatically when the supervisor is started. Following is a config example, taken from minos/build/template/supervisord.conf.tmpl, that shows how to configure crashmail:

[eventlistener:crashmailbatch-monitor]
command=python superlance/crashmailbatch.py \
        --toEmail="alert@example.com" \
        --fromEmail="robot@example.com" \
        --password="123456" \
        --smtpHost="mail.example.com" \
        --tickEvent=TICK_5 \
        --interval=0.5
events=PROCESS_STATE,TICK_5
buffer_size=100
stdout_logfile=crashmailbatch.stdout
stderr_logfile=crashmailbatch.stderr
autostart=true

Note: The related configuration information such as the server port or username is set inminos/build/template/supervisord.conf.tmpl, if you don't want to use the default value, change it.

Using Client

Prerequisites

Make sure you have intstalled Tank and Supervisor on your production machines.

A Simple Tutorial

Here we would like to show you how to use the client in a simple tutorial. In this tutorial we will use Minos to deploy an HDFS service, which itself requires the deployment of a ZooKeeper service.

The following are some conventions we will use in this tutorial:

  • Cluster type: we define three types of clusters: tst for testing, prc for offline processing, and srv for online serving.
  • ZooKeeper cluster name: we define the ZooKeeper cluster name using the IDC short name and the cluster type. For example, dptst is used to name a testing cluster at IDC dp.
  • Other service cluster names: we define other service cluster names using the corresponding ZooKeeper cluster name and the name of the business for which the service is intended to serve. For example, the dptst-example is the name of a testing cluster used to do example tests.
  • Configuration file names: all the services will have a corresponding configuration file, which will be named as${service}-${cluster}.cfg. For example, the dptst ZooKeeper service's configuration file is named as zookeeper-dptst.cfg, and the dptst example HDFS service's configuration file is named as hdfs-dptst-example.cfg.

Configuring deploy.cfg

There is a configuration file named deploy.cfg under the root directory of minos. You should first edit this file to set up the deployment environment. Make sure that all service packages are prepared and configured in deploy.cfg.

Configuring ZooKeeper

As mentioned in the cluster naming conventions, we will set up a testing ZooKeeper cluster at the dp IDC, and the corresponding configuration file for the cluster will be named as zookeeper-dptst.cfg.

You can edit zookeeper-dptst.cfg under the config/conf/zookeeper directory to configure the cluster. The zookeeper-dptst.cfg is well commented and self explained, so we will not explain more here.

Setting up a ZooKeeper Cluster

To set up a ZooKeeper cluster, just do the following two steps:

  • Install a ZooKeeper package to the tank server:

    cd minos/client
    ./deploy install zookeeper dptst
    
  • Bootstrap the cluster, this is only needed once when the cluster is setup for the first time:

    ./deploy bootstrap zookeeper dptst
    

Here are some handy ways to manage the cluster:

  • Show the status of the ZooKeeper service:

    ./deploy show zookeeper dptst
    
  • Start/Stop/Restart the ZooKeeper cluster:

    ./deploy stop zookeeper dptst
    ./deploy start zookeeper dptst
    ./deploy restart zookeeper dptst
    
  • Clean up the ZooKeeper cluster:

    ./deploy cleanup zookeeper dptst
    
  • Rolling update the ZooKeeper cluster:

    ./deploy rolling_update zookeeper dptst
    

Configuring HDFS

Now it is time to configure the HDFS system. Here we set up a testing HDFS cluster named dptst-example, whose configuration file will be named as hdfs-dptst-example.cfg, as explained in the naming conventions.

You can edit hdfs-dptst-example.cfg under the config/conf/hdfs directory to configure the cluster. The hdfs-dptst-example.cfg is well commented and self explained, so we will not explain more here.

Setting Up HDFS Cluster

Setting up and managing an HDFS cluster is similar to setting up and managing a ZooKeeper cluster. The only difference is the cluster name, dptst-example, which implies that the corresponding ZooKeeper cluster is dptst:

./deploy install hdfs dptst-example
./deploy bootstrap hdfs dptst-example
./deploy show hdfs dptst-example
./deploy stop hdfs dptst-example
./deploy start hdfs dptst-example
./deploy restart hdfs dptst-example
./deploy rolling_update hdfs dptst-example --job=datanode
./deploy cleanup hdfs dptst-example

Shell

The client tool also supports a very handy command named shell. You can use this command to manage the files on HDFS, tables on HBase, jobs on YARN, etc. Here are some examples about how to use the shell command to perform several different HDFS operations:

./deploy shell hdfs dptst-example dfs -ls /
./deploy shell hdfs dptst-example dfs -mkdir /test
./deploy shell hdfs dptst-example dfs -rm -R /test

You can run ./deploy --help to see the detailed help messages.

Installing Owl

Owl must be installed on the machine that you also use the Client component, they both use the same set of cluster configuration files.

Prerequisites

Install Gnuplot

Gnuplot is required for opentsdb, you can install it with the following command.

Centos: sudo yum install gnuplot
Ubuntu: sudo apt-get install gnuplot

Install Mysql

Ubuntu:
sudo apt-get install mysql-server
sudo apt-get install mysql-client

Centos:
yum install mysql-server mysql mysql-devel

Configuration

Configure the clusters you want to monitor with owl in minos/config/owl/collector.cfg. Following is an example that shows how to modify the configuration.

[collector]
# service name(space seperated)
service = hdfs hbase

[hdfs]
# cluster name(space seperated)
clusters=dptst-example
# job name(space seperated)
jobs=journalnode namenode datanode
# url for collecotr, usually JMX url
metric_url=/jmx?qry=Hadoop:*

Note: Some other configurations such as and opentsdb port is set in minos/build/minos_config.py. You can change the default port for avoiding port conflicts.

Start Owl

cd minos
./build.sh start owl --owl_ip ${your_local_ip} --owl_port ${port_owl_monitor_will_listen}

After starting Owl, you can access the web interface of the Owl. For example, if Owl listens on port 8088, and the machine's IP address is 192.168.1.11, you can access the following URL to view the Owl web interface:

http://192.168.1.11:8088/

Stop Owl

./build.sh stop owl

FAQ

  1. When installing Mysql-python, you may get an error of _mysql.c:44:23: error: my_config.h: No such file or directory (centos) or EnvironmentError: mysql_config not found (ubuntu). As mysql_config is part of mysql-devel, installing mysql-devel allows the installation of Mysql-python. So you may need to install it.

    ubuntu: sudo apt-get install libmysqlclient-dev
    centos: sudo yum install mysql-devel
    
  2. When installing twisted, you may get an error of CompressionError: bz2 module is not available and compile appears:

    Python build finished, but the necessary bits to build these modules were not found:
    _sqlite3           _tkinter           bsddb185
    bz2                dbm                dl
    

    Then, you may need to install bz2 and sqlite3 such as

    sudo apt-get install libbz2-dev
    sudo apt-get install libsqlite3-dev
    
  3. When setting up the stand-alone hbase on Ubuntu, you may fail to start it because of the /etc/hosts file. You can refer to http://hbase.apache.org/book/quickstart.html#ftn.d2907e114 to fix the problem.

  4. When using the Minos client to install a service package, if you get an error of socket.error: [Errno 101] Network is unreachable, please check your tank server configuration in deploy.cfg file, you might miss it.

Note: See Minos Wiki for more advanced features.


0人打赏
您需要登录后才可以回帖
发表回复
极贡献
技术问答
专题荟萃
程序人生
视觉设计
Android开发
iOS开发
编程语言
前端开发
后端开发
服务器架构
软件测试
运维方案
创业路上



最热文章墙

  • 59848/343   【精品推荐】200多种Android动画效果的强悍框架,太全了,不看这个,再有动画的问题,不理你了^@^

  • 33080/188   情人节福利,程序员表白的正确姿势:改几行代码就变成自己的表白了

  • 31759/141   省时省力的Android组件群来了,非常棒的原型参考

  • 28796/223   【精品推荐】Android版产品级的音乐播放器源码,功能太强大了,最好的产品原型有木有?

  • 26157/2   超全!整理常用的iOS第三方资源

  • 25485/0   Python爬虫:常用浏览器的useragent

  • 24647/70   原创表白APP,以程序员的姿势备战新年后的7夕,持续完善中!

  • 22807/138   2016抢红包软件及源码

  • 19930/29   麻省理工的一帮疯子,真的实现了随意操控万物!(绝对黑科技)

  • 19780/25   Android工程师面试题大全

  • 19177/27   2016程序员跳槽全攻略

  • 18963/9   GitHub上排名前50的iOS项目:总有一款你用得着

  • 18494/20   码魂:程序员的牛B漫画

  • 16856/3   吐槽那些程序员的搞笑牛逼注释

  • 16760/149   Android版类似UC浏览器:非常赞,产品级的源码

  • 15685/1   iOS 动画总结

  • 15206/41   一个绚丽的loading动效分析与实现!

  • 15149/10   2016年最全的Android面试考题+答案 精编版

  • 14767/73   【持续更新中】Android福利贴(二):资料源码大放送

  • 14708/83   Android小而全的博客源码:非常适合全面掌握开发技巧

  • 14644/10   女程序员的梦,众网友的神回复

  • 14489/44   惊艳的App引导页:背景图片切换加各个页面动画效果

  • 14454/11   年会上现场review代码是怎么样的体验!

  • 14145/23   个人收集的Android 各类功能源代码

  • 14057/5   新一代Android渠道打包工具:1000个渠道包只需要5秒

  • 13252/19   珍藏多年的素材,灵感搜寻网站

  • 13199/53   基于瀑布流的美女图片浏览App,有注释的源代码

  • 12856/17   用JavaScript 来开发iOS和Android 原生应用:React Native开源框架中文版来啦

  • 12746/15   基于Android支付宝支付设计和开发方案

  • 12684/74   仿京东商城客户端Android最新版,不错的原型和学习资料

  • 12488/20   Android福利第三波【Android电子书】

  • 12306/17   什么是真正的黑客:收获12200+Stars,人气远超微软开源VS

  • 12116/18   65条最常用正则表达式,你要的都在这里了

  • 12085/94   Android带弹幕的视频播放器源码,来自大名鼎鼎的Bilibili弹幕网站

  • 11980/70   【精品推荐】类似360安全卫士安Android源码:非常赞的产品原型

  • 11709/7   用程序员的姿势抢过年的火车票

  • 11660/7   一张图搞定iOS学习路线,非常全面

  • 11601/11   有木有这样一张酷图帮你集齐所有git命令超实用

  • 11560/0   iOS中文版资源库,非常全

  • 11135/10   成为Java顶尖程序员 ,看这11本书就够了

  • 10869/18   一张图搞定Android学习路线,非常全面

  • 10802/10   微信支付终于成功了(安卓,iOS),在此分享

  • 10451/44   在线音乐播放器完整版(商用级的源码):非常赞,可听免费高品质专辑

  • 10417/3   基于Node.js的强大爬虫,能直接发布抓取的文章哦

  • 10331/29   【持续更新中】Android福利贴(一):资料源码

  • 10317/0   GitHub iOS 库和框架Top100 

  • 9900/4   46 个非常有用的 PHP 代码片段

  • 9664/3   即时通信第三方库

  • 9541/61   【技巧一】搭配Android Studio,如何实现App远程真机debug?

  • 9492/9   烧了5亿美金,这家神秘的公司即将颠覆人类未来!

  • 9443/8   流媒体视频直播方案

  • 9294/9   B站建开源工作组:APP想支持炫酷弹幕的看过来

  • 9173/18   八个最优秀的Android Studio插件

  • 9143/2   【精品推荐】高质量PHP代码的50个实用技巧:非常值得收藏

  • 9054/9   中国黑客的隐秘江湖:攻守对立,顶尖高手月入千万美元

  • 8616/6   开箱即用!Android四款系统架构工具

  • 8513/3   一张图看清Linux 内核运行原理

  • 8410/10   十大技巧快速提升Android应用开发性能

  • 8383/3   10款GitHub上最火爆的国产开源项目——可以媲美西半球

  • 8068/1   Android性能优化视频,文档以及工具

  • 返回顶部