今天，天气不错….  出去逛了下，在咖啡店看了会美女….  休闲时，看到携程的一朋友在github关注了一个python redis cluster集群模块….  立马警醒，不能在看妹子了….  人家大周末也学习，我也要学习….   

标注下，关于redis cluster配置安装的文章是在 http://xiaorui.cc/?p=1443   http://xiaorui.cc

公司的业务大量的依靠redis做支撑，每个业务线的redis使用量都不小，像我们buzz业务就有好几个128G内存的redis服务器，在程序里面做的一致性hash。也尝试过用keepalived的ha方式来保证redis的高可用，但是keepalived也只能保证两个master的高可行，再就是用了keepalived之后，只能是一个读，一个做写，在某种程度上来说，在我们这种写入远比读业务要多的场景下，他是在是浪费….    单点问题一直是我们关注的，我自己也尝试过豌豆荚的codis，也听过codis作者的演讲…..    但现在redis3.0正式版也已经出来了，问了下周围的圈子里面的人，也是网易有道云在用，貌似在rc的版本，他们就在使用了….     当然不能人云亦云，我自己也要测试下redis codis….

     同事说，公司的针对redis cluster的机器已经分配下来了，今天就先在单机简单的测试下，后期可能在部署上采用docker redis的方式…..    现在同事们已经很享受用docker来快速的部署业务了。    我会在今天下午的时候，把redis cluster的dockerfile分享出来。   

我的测试机器是ubuntu14:10,本来是打算用centos的，结果起不来….  首先下载并安装redis3

wget http://download.redis.io/releases/redis-3.0.1.tar.gz
tar zxvf redis-3.0.1.tar.gz
cd redis-3.0.1/
make
make install

在单机测试的话，我们用redis-server分别启动多个文件就可以了，我这里采用的是6个端口，在redis cluster 里是三主、三从…. 当时也可以按照自己的情况设立主从….  需要注意的是redis.conf要开发cluster的选项。  我简单写了一个批量修改redis.conf的脚本.

for i in `seq -w 0 5`;do
    cp redis.conf 700i.conf
    sed -i "s/6379/700i/g" 700i.conf
    sed -i 's/daemonize no/daemonize yes/g' 700i.conf
    sed -i 's/# cluster-enabled yes/cluster-enabled yes/g' 700i.conf
    sed -i 's/# cluster-node-timeout 15000/cluster-node-timeout 15000/g' 700i.conf
    sed -i "s/# cluster-config-file node.*/cluster-config-file nodes-700i.conf/g" 700i.conf
    redis-server 700$i.conf
done

如果你想自己手动修改的话，注意这么几个点.   端口一定不要重复，然后就是cluster-enabled yes  

port xxxx
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000

如果只是更改了端口，会造成一个无法创建集群的问题。 

Installing RDoc documentation for redis-3.2.1...
root@ubuntu:~/redis-3.0.1# ./src/redis-trib.rb  create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
Connecting to node 127.0.0.1:7000: OK
[ERR] Node 127.0.0.1:7000 is not configured as a cluster node.
root@ubuntu:~/redis-3.0.1#

启动后，我们就可以在linux 进城管理里面看到redis的进程.   注意后面的端口是刚才咱们指定的。 

root@ubuntu:~/redis-3.0.1/src# ps aux|grep redis
root      4974  0.1  0.9  40416  9388 ?        Ssl  12:11   0:03 redis-server *:7000 [cluster]
root      4983  0.1  0.9  40416  9452 ?        Ssl  12:11   0:04 redis-server *:7001 [cluster]
root      4992  0.1  0.9  40416  9440 ?        Ssl  12:11   0:03 redis-server *:7002 [cluster]
root      5001  0.1  0.9  40416  9420 ?        Ssl  12:11   0:04 redis-server *:7003 [cluster]
root      5010  0.1  0.7  40416  7392 ?        Ssl  12:11   0:04 redis-server *:7004 [cluster]
root      5019  0.1  0.9  40416  9424 ?        Ssl  12:11   0:04 redis-server *:7005 [cluster]

redis cluster的配置是用的ruby脚本写得，那么就需要你最少安装了ruby (apt-get install ruby )和gem。更主要的是你还要用安装ruby所需要的redis模块。 

root@ubuntu:~/redis-3.0.1# ./src/redis-trib.rb  create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- redis (LoadError)
    from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from ./src/redis-trib.rb:25:in `<main>'
root@ubuntu:~/redis-3.0.1#

安装下ruby的redis驱动。 

root@ubuntu:~/redis-3.0.1# gem install redis
Fetching: redis-3.2.1.gem (100%)
Successfully installed redis-3.2.1
1 gem installed
Installing ri documentation for redis-3.2.1...
Installing RDoc documentation for redis-3.2.1...
root@ubuntu:~/redis-3.0.1#

然后我们用redis-trib.rb来创建redis cluster集群，下面是成功的提示。 

root@ubuntu:~/redis-3.0.1# cd src
root@ubuntu:~/redis-3.0.1/src# ./redis-trib.rb  create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7001: OK
Connecting to node 127.0.0.1:7002: OK
Connecting to node 127.0.0.1:7003: OK
Connecting to node 127.0.0.1:7004: OK
Connecting to node 127.0.0.1:7005: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: bde32811cbc7a55886413c52b8444d3353b9da86 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: d8815ad0a69436196908ae458f94704ff7325a53 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: ee05942ee38a56421a07eea01bc6072fe5e23bfd 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
S: 25f13acb0707a29cd2952dd0c6bdffa13721b001 127.0.0.1:7003
   replicates bde32811cbc7a55886413c52b8444d3353b9da86
S: d6dd1200fe377dc3aee2fd7d11a57504a6bd7529 127.0.0.1:7004
   replicates d8815ad0a69436196908ae458f94704ff7325a53
S: 2282e7f24bacf1b6b316382eb8f6448bc41bd89d 127.0.0.1:7005
   replicates ee05942ee38a56421a07eea01bc6072fe5e23bfd
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: bde32811cbc7a55886413c52b8444d3353b9da86 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: d8815ad0a69436196908ae458f94704ff7325a53 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: ee05942ee38a56421a07eea01bc6072fe5e23bfd 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
M: 25f13acb0707a29cd2952dd0c6bdffa13721b001 127.0.0.1:7003
   slots: (0 slots) master
   replicates bde32811cbc7a55886413c52b8444d3353b9da86
M: d6dd1200fe377dc3aee2fd7d11a57504a6bd7529 127.0.0.1:7004
   slots: (0 slots) master
   replicates d8815ad0a69436196908ae458f94704ff7325a53
M: 2282e7f24bacf1b6b316382eb8f6448bc41bd89d 127.0.0.1:7005
   slots: (0 slots) master
   replicates ee05942ee38a56421a07eea01bc6072fe5e23bfd
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@ubuntu:~/redis-3.0.1/src#

这就说明，集群已经OK了， redis-cli客户端默认是不支持集群模式的….    如果像以前那么用redis-cli的话，会出现下面的情况…..   各种moved… error

 
root@ubuntu:~/redis-3.0.1/src# redis-cli -p 7000
127.0.0.1:7000>
127.0.0.1:7000>
127.0.0.1:7000> set a 11
(error) MOVED 15495 127.0.0.1:7002
127.0.0.1:7000> set b 22
OK
127.0.0.1:7000> set c 33
(error) MOVED 7365 127.0.0.1:7001
127.0.0.1:7000> set d 33
(error) MOVED 11298 127.0.0.1:7002
127.0.0.1:7000> set e 33
(error) MOVED 15363 127.0.0.1:7002
127.0.0.1:7000> set f 33
OK
127.0.0.1:7000> set g 33
(error) MOVED 7233 127.0.0.1:7001
127.0.0.1:7000> get g
(error) MOVED 7233 127.0.0.1:7001

只有加了-c参数，才能开启redis cluster 模式….   所以一定要好好看文档，就是因为这没有加-c 参数，导致了花费些时间来我排除问题 。

root@ubuntu:~/redis-3.0.1/src# redis-cli -h
redis-cli 3.0.1

Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]
  -a <password>      Password to use when connecting to the server.
  -r <repeat>        Execute specified command N times.
  -i <interval>      When -r is used, waits <interval> seconds per command.
                     It is possible to specify sub-second times like -i 0.1.
  -n <db>            Database number.
  -x                 Read last argument from STDIN.
  -d <delimiter>     Multi-bulk delimiter in for raw formatting (default: \n).
  -c                 Enable cluster mode (follow -ASK and -MOVED redirections).

我们最后来测试下….

root@ubuntu:~/redis-3.0.1/src# redis-cli -c -p 7000
127.0.0.1:7002> set blog xiaorui.cc
-> Redirected to slot [7653] located at 127.0.0.1:7001
OK
127.0.0.1:7001> set news xiaorui.cc
-> Redirected to slot [5161] located at 127.0.0.1:7000
OK
127.0.0.1:7000> set nima xiaorui.cc
-> Redirected to slot [16259] located at 127.0.0.1:7002
OK
127.0.0.1:7002> set ganni xiaorui.cc
-> Redirected to slot [1198] located at 127.0.0.1:7000
OK
127.0.0.1:7000> set love xiaorui.cc
-> Redirected to slot [16198] located at 127.0.0.1:7002
OK
127.0.0.1:7002> set satlstack xiaorui.cc
-> Redirected to slot [2455] located at 127.0.0.1:7000
OK

看来用以前的python redis模块也是不能直接用集群模式了，需要替换成支持redis cluster的python模块….