mongodb 安装使用-副本集+分片架构部署 -2

文章介绍的不清楚，安装有几处不合理，新的参考文章：Mongodb3.2.12 副本集 + 分片完整安装

由于最近的项目中使用到了mongodb,前两天安装完后一直在忙别的事情，mongodb 安装，今天配置一下副本集+分片。

组件介绍

四个组件：mongos、config server、shard、replica set。 mongos，数据库集群请求的入口，所有的请求都通过mongos进行协调，不需要在应用程序添加一个路由选择器，mongos自己就是一个请求分发中心，它负责把对应的数据请求请求转发到对应的shard服务器上。在生产环境通常有多mongos作为请求的入口，防止其中一个挂掉所有的mongodb请求都没有办法操作。 config server，顾名思义为配置服务器，存储所有数据库元信息（路由、分片）的配置。 mongos本身没有物理存储分片服务器和数据路由信息，只是缓存在内存里，配置服务器则实际存储这些数据。mongos第一次启动或者关掉重启就会从 config server 加载配置信息，以后如果配置服务器信息变化会通知到所有的 mongos 更新自己的状态，这样 mongos 就能继续准确路由。在生产环境通常有多个 config server 配置服务器，因为它存储了分片路由的元数据，这个可不能丢失！就算挂掉其中一台，只要还有存货， mongodb集群就不会挂掉。 shard，这就是传说中的分片了。举例说明：一台机器的一个数据表 Collection1 存储了 1T 数据，压力太大了！在分给4个机器后，每个机器都是256G，则分摊了集中在一台机器的压力。也许有人问一台机器硬盘加大一点不就可以了，为什么要分给四台机器呢？不要光想到存储空间，实际运行的数据库还有硬盘的读写、网络的IO、CPU和内存的瓶颈。在mongodb集群只要设置好了分片规则，通过 mongos操作数据库就能自动把对应的数据操作请求转发到对应的分片机器上。在生产环境中分片的片键可要好好设置，这个影响到了怎么把数据均匀分到多个分片机器上，不要出现其中一台机器分了1T，其他机器没有分到的情况，这样还不如不分片！ replica set，上两节已经详细讲过了这个东东，怎么这里又来凑热闹！其实上图4个分片如果没有 replica set 是个不完整架构，假设其中的一个分片挂掉那四分之一的数据就丢失了，所以在高可用性的分片架构还需要对于每一个分片构建 replica set 副本集保证分片的可靠性。生产环境通常是 2个副本 + 1个仲裁。

准备开始

现在有3台机器，首先看一下集群的磁盘部署

[root@new-cdh18 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_newcdh3-lv_root
1011G 6.2G 953G 1% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sdc1 477M 50M 402M 12% /boot
/dev/mapper/vg_newcdh3-lv_home
2.2T 73M 2.1T 1% /home
cm_processes 3.9G 824K 3.9G 1% /opt/cm-5.4.0/run/cloudera-scm-agent/process

发现大部分磁盘空间挂载在/home,于是决定把数据库放在/home下。首先改变mongod 用户的home，把它挂载到 /home 下。

[root@new-cdh18 ~]# service mongod stop
Stopping mongod: [ OK ]
[root@new-cdh18 ~]# usermod -d /home/mongod/ -u 478 mongod
[root@new-cdh18 ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
…
…
…
mongod:x:478:477:mongod:/home/mongod/:/bin/false

mongo 不建议使用root 用户进行操作，以后准备使用mongod用户进行操作，修改sh 和密码，修改成可使用的用户（每台机器都修改）

[root@new-cdh20 ~]# chsh -s /bin/bash mongod
Changing shell for mongod.
Shell changed.
[root@new-cdh20 ~]# passwd mongod
[root@new-cdh20 ~]# su mongod
bash-4.1$ cat > /home/mongod/.bashrc <<EOF
> # .bashrc

> # Source global definitions
> if [ -f /etc/bashrc ]; then
> . /etc/bashrc
> fi

> # User specific aliases and functions
> EOF
bash-4.1$ cat > /home/mongod/.bash_profile <<EOF
> # .bash_profile

> # Get the aliases and functions
> if [ -f ~/.bashrc ]; then
> . ~/.bashrc
> fi

> # User specific environment and startup programs

> PATH= $PATH:$ HOME/bin

> export PATH
> EOF
bash-4.1$ source /etc/profile
bash-4.1$ source .bash_profile
[mongod@new-cdh18 mongod]$ ll
total 24
drwxr-xr-x 3 mongod mongod 4096 May 9 10:23 config
drwxr-xr-x 8 mongod mongod 4096 May 9 10:59 mongodb
drwxr-xr-x 3 mongod mongod 4096 May 9 10:17 mongos
drwxr-xr-x 4 mongod mongod 4096 May 9 10:33 shard1
drwxr-xr-x 4 mongod mongod 4096 May 9 10:36 shard2
drwxr-xr-x 4 mongod mongod 4096 May 9 10:37 shard3

系统配置

软件环境： OS：Oracle Linux 6.7 MongoDB:mongodb3.2.6 配置hosts：追加到hosts

…
192.168.36.18 new-cdh18 mongo1
192.168.36.19 new-cdh19 mongo2
192.168.36.20 new-cdh20 mongo3

开始部署

1、创建文件夹和配置文件。

mongo1:

#创建数据节点
[mongod@new-cdh18 ~]# mkdir -p /home/mongod/mongodb/shard11
[mongod@new-cdh18 ~]# cat > /home/mongod/mongodb/shard11/shard11.conf <<EOF
> shardsvr=true
> replSet=shard1
> port=28017
> dbpath=/home/mongod/mongodb/shard11
> oplogSize=2048
> logpath=/home/mongod/mongodb/shard11/shard11.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF
[mongod@new-cdh18 ~]# mkdir -p /home/mongod/mongodb/shard21
[mongod@new-cdh18 ~]# cat > /home/mongod/mongodb/shard21/shard21.conf <<EOF
> shardsvr=true
> replSet=shard2
> port=28018
> dbpath=/home/mongod/mongodb/shard21
> oplogSize=2048
> logpath=/home/mongod/mongodb/shard21/shard21.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF

#创建仲裁节点
[mongod@new-cdh18 ~]# mkdir -p /home/mongod/mongodb/arbiter1
[mongod@new-cdh18 ~]# cat > /home/mongod/mongodb/arbiter1/arbiter1.conf <<EOF
> shardsvr=true
> replSet=shard1
> port=28031
> dbpath=/home/mongod/mongodb/arbiter1
> oplogSize=100
> logpath=/home/mongod/mongodb/arbiter1/arbiter1.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF
[mongod@new-cdh18 ~]# mkdir -p /home/mongod/mongodb/arbiter2
[mongod@new-cdh18 ~]# cat > /home/mongod/mongodb/arbiter2/arbiter2.conf <<EOF
> shardsvr=true
> replSet=shard2
> port=28032
> dbpath=/home/mongod/mongodb/arbiter2
> oplogSize=100
> logpath=/home/mongod/mongodb/arbiter2/arbiter2.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF

#创建配置节点
[mongod@new-cdh18 ~]# mkdir -p /home/mongod/mongodb/config/
[mongod@new-cdh18 ~]# cat > /home/mongod/mongodb/config/config1.conf <<EOF
> configsvr=true
> dbpath=/home/mongod/mongodb/config/
> port=20000
> logpath=/home/mongod/mongodb/config/config1.log
> logappend=true
> fork=true
> httpinterface=true
> #nojournal=true
> EOF

#创建路由节点
[mongod@new-cdh18 ~]# mkdir -p /home/mongod/mongodb/mongos
[mongod@new-cdh18 ~]# cat > /home/mongod/mongodb/mongos/mongos1.conf <<EOF
> configdb=mongo1:20000,mongo2:20000,mongo3:20000
> port=28885
> chunkSize=100
> logpath=/home/mongod/mongodb/mongos/mongos1.log
> logappend=true
> fork=true
> EOF
[mongod@new-cdh18 ~]#

mongo2:

#创建数据节点
[mongod@new-cdh19 ~]# mkdir -p /home/mongod/mongodb/shard12
[mongod@new-cdh19 ~]# cat > /home/mongod/mongodb/shard12/shard12.conf <<EOF
> shardsvr=true
> replSet=shard1
> port=28017
> dbpath=/home/mongod/mongodb/shard12
> oplogSize=2048
> logpath=/home/mongod/mongodb/shard12/shard12.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF
[mongod@new-cdh19 ~]# mkdir -p /home/mongod/mongodb/shard22
[mongod@new-cdh19 ~]# cat > /home/mongod/mongodb/shard22/shard22.conf <<EOF
> shardsvr=true
> replSet=shard2
> port=28018
> dbpath=/home/mongod/mongodb/shard22
> oplogSize=2048
> logpath=/home/mongod/mongodb/shard22/shard22.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF

#创建仲裁节点
[mongod@new-cdh19 ~]# mkdir -p /home/mongod/mongodb/arbiter1
[mongod@new-cdh19 ~]# cat > /home/mongod/mongodb/arbiter1/arbiter1.conf<<EOF
> shardsvr=true
> replSet=shard1
> port=28031
> dbpath=/home/mongod/mongodb/arbiter1
> oplogSize=100
> logpath=/home/mongod/mongodb/arbiter1/arbiter1.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF
[mongod@new-cdh19 ~]# mkdir -p /home/mongod/mongodb/arbiter2
[momgod@new-cdh19 ~]# cat > /home/mongod/mongodb/arbiter2/arbiter2.conf <<EOF
> shardsvr=true
> replSet=shard2
> port=28032
> dbpath=/home/mongod/mongodb/arbiter2
> oplogSize=100
> logpath=/home/mongod/mongodb/arbiter2/arbiter2.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF

#创建配置节点
[mongod@new-cdh19 ~]# mkdir -p /home/mongod/mongodb/config/
[mongod@new-cdh19 ~]# cat > /home/mongod/mongodb/config/config2.conf <<EOF
> configsvr=true
> dbpath=/home/mongod/mongodb/config/
> port=20000
> logpath=/home/mongod/mongodb/config/config2.log
> logappend=true
> fork=true
> #nojournal = true
> EOF

#创建路由节点
[mongod@new-cdh19 ~]# mkdir -p /home/mongod/mongodb/mongos
[mongod@new-cdh19 ~]# cat >/home/mongod/mongodb/mongos/mongos2.conf <<EOF
> configdb=mongo1:20000,mongo2:20000,mongo3:20000
> port=28885
> chunkSize=100
> logpath=/home/mongod/mongodb/mongos/mongos2.log
> logappend=true
> fork=true
> EOF
[mongod@new-cdh19 ~]#

mongo3:

#创建数据节点
[mongod@new-cdh20 ~]# mkdir -p /home/mongod/mongodb/shard13
[mongod@new-cdh20 ~]# cat > /home/mongod/mongodb/shard13/shard13.conf <<EOF
> shardsvr=true
> replSet=shard1
> port=28017
> dbpath=/home/mongod/mongodb/shard13
> oplogSize=2048
> logpath=/home/mongod/mongodb/shard13/shard13.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF
[mongod@new-cdh20 ~]# mkdir -p /home/mongod/mongodb/shard23
[mongod@new-cdh20 ~]# cat > /home/mongod/mongodb/shard23/shard23.conf <<EOF
> shardsvr=true
> replSet=shard2
> port=28018
> dbpath=/home/mongod/mongodb/shard23
> oplogSize=2048
> logpath=/home/mongod/mongodb/shard23/shard23.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF

#创建仲裁节点
[mongod@new-cdh20 ~]# mkdir -p /home/mongod/mongodb/arbiter1
[mongod@new-cdh20 ~]# cat > /home/mongod/mongodb/arbiter1/arbiter1.conf<<EOF
> shardsvr=true
> replSet=shard1
> port=28031
> dbpath=/home/mongod/mongodb/arbiter1
> oplogSize=100
> logpath=/home/mongod/mongodb/arbiter1/arbiter1.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF
[mongod@new-cdh20 ~]# mkdir -p /home/mongod/mongodb/arbiter2
[mongod@new-cdh20 ~]# cat > /home/mongod/mongodb/arbiter2/arbiter2.conf <<EOF
> shardsvr=true
> replSet=shard2
> port=28032
> dbpath=/home/mongod/mongodb/arbiter2
> oplogSize=100
> logpath=/home/mongod/mongodb/arbiter2/arbiter2.log
> logappend=true
> fork=true
> rest=true
> httpinterface=true
> nojournal = true
> EOF

#创建配置节点
[mongod@new-cdh20 ~]# mkdir -p /home/mongod/mongodb/config/
[mongod@new-cdh20 ~]# cat > /home/mongod/mongodb/config/config3.conf <<EOF
> configsvr=true
> dbpath=/home/mongod/mongodb/config/
> port=20000
> logpath=/home/mongod/mongodb/config/config3.log
> logappend=true
> fork=true
> #nojournal = true
> EOF

#创建路由节点
[mongod@new-cdh20 ~]# mkdir -p /home/mongod/mongodb/mongos
[mongod@new-cdh20 ~]# cat >/home/mongod/mongodb/mongos/mongos3.conf <<EOF
> configdb=mongo1:20000,mongo2:20000,mongo3:20000
> port=28885
> chunkSize=100
> logpath=/home/mongod/mongodb/mongos/mongos3.log
> logappend=true
> fork=true
> EOF
[root@new-cdh20 ~]#

启动服务。

在mongo1,mongo2,mongo3启动shard1、shard2、arbiter1、arbiter2服务 mongo1:

[mongod@new-cdh18 ~]# mongod --config /home/mongod/mongodb/shard11/shard11.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27178
child process started successfully, parent exiting
[mongod@new-cdh18 ~]# mongod --config /home/mongod/mongodb/shard21/shard21.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27200
child process started successfully, parent exiting
[mongod@new-cdh18 ~]# mongod --config /home/mongod/mongodb/arbiter1/arbiter1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27221
child process started successfully, parent exiting
[mongod@new-cdh18 ~]# mongod --config /home/mongod/mongodb/arbiter2/arbiter2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27242
child process started successfully, parent exiting
[mongod@new-cdh18 ~]# mongod --config /home/mongod/mongodb/config/config1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27279
child process started successfully, parent exiting
[mongod@new-cdh18 ~]#

mongo2:

[mongod@new-cdh19 ~]# mongod --config /home/mongod/mongodb/shard12/shard12.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27227
child process started successfully, parent exiting
[mongod@new-cdh19 ~]# mongod --config /home/mongod/mongodb/shard22/shard22.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27264
child process started successfully, parent exiting
[mongod@new-cdh19 ~]# mongod --config /home/mongod/mongodb/arbiter1/arbiter1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27336
child process started successfully, parent exiting
[mongod@new-cdh19 ~]# mongod --config /home/mongod/mongodb/arbiter2/arbiter2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27357
child process started successfully, parent exiting
[mongod@new-cdh19 ~]# mongod --config /home/mongod/mongodb/config/config2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27378
child process started successfully, parent exiting
[mongod@new-cdh19 ~]#

mongo3:

[mongod@new-cdh20 ~]# mongod --config /home/mongod/mongodb/shard13/shard13.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27249
child process started successfully, parent exiting
[mongod@new-cdh20 ~]# mongod --config /home/mongod/mongodb/shard23/shard23.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27271
child process started successfully, parent exiting
[mongod@new-cdh20 ~]# mongod --config /home/mongod/mongodb/arbiter1/arbiter1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27292
child process started successfully, parent exiting
[mongod@new-cdh20 ~]# mongod --config /home/mongod/mongodb/arbiter2/arbiter2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27313
child process started successfully, parent exiting
[mongod@new-cdh20 ~]# mongod --config /home/mongod/mongodb/config/config3.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27334
child process started successfully, parent exiting
[mongod@new-cdh20 ~]#

然后在mongo1、mongo2、mongo3上分别启动mongos服务 mongo1:

[mongod@new-cdh18 ~]# mongos --config /home/mongod/mongodb/mongos/mongos1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27616
child process started successfully, parent exiting

mongo2:

[mongod@new-cdh19 ~]# mongos --config /home/mongod/mongodb/mongos/mongos2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27521
child process started successfully, parent exiting

mongo3:

[mongod@new-cdh20 ~]# mongos --config /home/mongod/mongodb/mongos/mongos3.conf
about to fork child process, waiting until server is ready for connections.
forked process: 27450
child process started successfully, parent exiting

配置分片副本集

任意登陆一个机器，比如登陆mongo1，连接mongodb

创建第一个分片副本集

[mongod@new-cdh18 mongod]$ mongo mongo1:28017/admin
MongoDB shell version: 3.2.6
connecting to: mongo1:28017/admin
> config = {_id: “shard1”, members:[
… {_id: 0, host:“mongo1:28017”},
… {_id: 1, host:“mongo2:28017”},
… {_id: 2, host:“mongo3:28017”,slaveDelay : 7200, priority:0},
… {_id: 3, host:“mongo1:28031”, arbiterOnly : true},
… {_id: 4, host:“mongo2:28031”, arbiterOnly : true},
… {_id: 5, host:“mongo3:28031”, arbiterOnly : true}
… ]
… };
{
“_id” : “shard1”,
“members” : [
{
“_id” : 0,
“host” : “mongo1:28017”
},
{
“_id” : 1,
“host” : “mongo2:28017”
},
{
“_id” : 2,
“host” : “mongo3:28017”,
“slaveDelay” : 7200,
“priority” : 0
},
{
“_id” : 3,
“host” : “mongo1:28031”,
“arbiterOnly” : true
},
{
“_id” : 4,
“host” : “mongo2:28031”,
“arbiterOnly” : true
},
{
“_id” : 5,
“host” : “mongo3:28031”,
“arbiterOnly” : true
}
]
}
> rs.initiate(config);
{ “ok” : 1 }
shard1:OTHER>
shard1:SECONDARY>
shard1:PRIMARY>
shard1:PRIMARY>

创建第二个分片副本集

[mongod@new-cdh18 mongod]$ mongo mongo1:28018/admin
MongoDB shell version: 3.2.6
connecting to: mongo1:28018/admin
> config = {_id: “shard2”, members:[
… {_id: 0, host:“mongo1:28018”},
… {_id: 1, host:“mongo2:28018”},
… {_id: 2, host:“mongo3:28018”,slaveDelay : 7200, priority:0},
… {_id: 3, host:“mongo1:28032”, arbiterOnly : true},
… {_id: 4, host:“mongo2:28032”, arbiterOnly : true},
… {_id: 5, host:“mongo3:28032”, arbiterOnly : true}
… ]
… };
{
“_id” : “shard2”,
“members” : [
{
“_id” : 0,
“host” : “mongo1:28018”
},
{
“_id” : 1,
“host” : “mongo2:28018”
},
{
“_id” : 2,
“host” : “mongo3:28018”,
“slaveDelay” : 7200,
“priority” : 0
},
{
“_id” : 3,
“host” : “mongo1:28032”,
“arbiterOnly” : true
},
{
“_id” : 4,
“host” : “mongo2:28032”,
“arbiterOnly” : true
},
{
“_id” : 5,
“host” : “mongo3:28032”,
“arbiterOnly” : true
}
]
}
> rs.initiate(config);
{ “ok” : 1 }
shard2:OTHER>
shard2:SECONDARY>
shard2:SECONDARY>

串联路由服务器与分配副本集

[mongod@new-cdh18 mongod]$ mongo mongo2:28885/admin
MongoDB shell version: 3.2.6
connecting to: mongo2:28885/admin
mongos> db.runCommand({“addshard” :“shard1/mongo1:28017,mongo2:28017”})
{ “shardAdded” : “shard1”, “ok” : 1 }
mongos> db.runCommand({“addshard” :“shard2/mongo2:28018,mongo1:28018”})
{ “shardAdded” : “shard2”, “ok” : 1 }
mongos>

因为mongo3:28017和mongo3:28018是延迟节点，所以不在副本集当中

查看分片服务器的配置

mongos> db.runCommand( { listshards : 1 } );
{
“shards” : [
{
“_id” : “shard1”,
“host” : “shard1/mongo1:28017,mongo2:28017”
},
{
“_id” : “shard2”,
“host” : “shard2/mongo1:28018,mongo2:28018”
}
],
“ok” : 1
}
mongos>

测试分片副本集

指定分片库、分片集合和片键

mongos> db.runCommand( { “enablesharding”: “testdb”})
{ “ok” : 1 }
mongos> db.runCommand( { shardcollection : “testdb.step”,key: {_id: 1} } )
{ “collectionsharded” : “testdb.step”, “ok” : 1 }
mongos>

end

文章介绍的不清楚，安装有几处不合理，新的参考文章：Mongodb3.2.12 副本集 + 分片 完整安装

组件介绍

准备开始

mongo 不建议使用root 用户进行操作，以后准备使用mongod用户进行操作，修改sh 和密码，修改成 可使用的用户（每台机器都修改）

系统配置

开始部署

1、创建文件夹和配置文件。

启动服务。

配置分片副本集

创建第一个分片副本集

创建第二个分片副本集

串联路由服务器与分配副本集

查看分片服务器的配置

测试分片副本集

文章介绍的不清楚，安装有几处不合理，新的参考文章：Mongodb3.2.12 副本集 + 分片完整安装

mongo 不建议使用root 用户进行操作，以后准备使用mongod用户进行操作，修改sh 和密码，修改成可使用的用户（每台机器都修改）