mongodb支持自动分片功能,这无疑给DBA和开发人员带来很大便利,我们无需要手动按一定规则进行拆分DB,当数据量增大后,我们只需要扩充设备即可。 mongodb shard配置文档如下: auo shard 试验设备列表 主机列表 主机名 ip 用途 监听端口 web访问url localhost 192.168.100.245 config server 27019 http://192.168.100.245:28019/ TEST40 192.168.100.212 shard 1 27018 http://192.168.100.212:28018/ chn-xg-5-5c1 192.168.100.210 shard 2 27018 http://192.168.100.210:28018/ band-all 192.168.100.229 mongos process/mongos Router 27017 http://192.168.100.229:27017/ 分析发现config server、shard、mongos的监听端口和web端口均是第二位的差异。监听端口第二位是7,web端口是8,其它全部相同。 mongod --fork --logpath /var/log/mongodb.log --logappend mogod默认启动为前台进程,当连接session断开,mongod程序自动关闭 一、配置shard 各个模块 1、启动shard1 [root@TEST40 ~]# mongod --shardsvr --fork --logpath /var/log/mongodb.log --logappend [1] 28521 [root@TEST40 ~]# Tue Aug 17 03:01:57 MongoDB starting : pid=28521 port=27018 dbpath=/data/db/ 32-bit ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data ** see http://blog./post/137788967/32-bit-limitations Tue Aug 17 03:01:57 db version v1.6.0, pdfile version 4.5 Tue Aug 17 03:01:57 git version: 2c7f164b653f0d703947572ede064aed41cc2185 Tue Aug 17 03:01:57 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37 Tue Aug 17 03:01:57 [initandlisten] waiting for connections on port 27018 Tue Aug 17 03:01:57 [websvr] web admin interface listening on port 28018 2、启动shard2,shard2的dbpath与默认路径不同,因此需要制定dbpath [root@chn-xg-5-5c1 cache2]# mongod --shardsvr --dbpath /data/cache2/db --fork --logpath /var/log/mongodb.log --logappend [1] 31264 [root@chn-xg-5-5c1 cache2]# Tue Aug 17 03:11:48 MongoDB starting : pid=31264 port=27018 dbpath=/data/cache2/db 32-bit ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data ** see http://blog./post/137788967/32-bit-limitations Tue Aug 17 03:11:48 db version v1.6.0, pdfile version 4.5 Tue Aug 17 03:11:48 git version: 2c7f164b653f0d703947572ede064aed41cc2185 Tue Aug 17 03:11:48 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37 Tue Aug 17 03:11:48 [initandlisten] waiting for connections on port 27018 Tue Aug 17 03:11:48 [websvr] web admin interface listening on port 28018 [root@chn-xg-5-5c1 cache2]# 3、配置config server,config server需要dbpath和repairpath dbpath默认为/data/configdb,repairpath默认为/data/db/,这与shard是不同的 [root@localhost data]# mongod --configsvr --fork --logpath /var/log/mongodb.log --logappend [1] 2144 [root@localhost data]# Tue Aug 17 03:13:54 MongoDB starting : pid=2144 port=27019 dbpath=/data/configdb 32-bit ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data ** see http://blog./post/137788967/32-bit-limitations Tue Aug 17 03:13:54 db version v1.6.0, pdfile version 4.5 Tue Aug 17 03:13:54 git version: 2c7f164b653f0d703947572ede064aed41cc2185 Tue Aug 17 03:13:54 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37 Tue Aug 17 03:13:54 [initandlisten] diagLogging = 1 Tue Aug 17 03:13:54 [initandlisten] waiting for connections on port 27019 Tue Aug 17 03:13:54 [websvr] web admin interface listening on port 28019 [root@localhost data]# 4、配置mongos router [root@band-all local]# mongos --configdb 192.168.100.245:27019 --fork --logpath /var/log/mongodb.log --logappend [1] 6384 [root@band-all local]# Tue Aug 17 03:46:12 mongos db version v1.6.0, pdfile version 4.5 starting (--help for usage) Tue Aug 17 03:46:12 git version: 2c7f164b653f0d703947572ede064aed41cc2185 Tue Aug 17 03:46:12 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37 Tue Aug 17 03:46:12 [websvr] web admin interface listening on port 28017 Tue Aug 17 03:46:12 waiting for connections on port 27017 [root@band-all local]# 二、配置shard 集群 1、登陆mongos server,添加shard [root@localhost data]# mongo 192.168.100.229:27017/admin MongoDB shell version: 1.6.0 connecting to: 192.168.100.229:27017/admin Tue Aug 17 05:59:38 [conn4] building new index on { _id: 1 } for config.databases Tue Aug 17 05:59:38 [conn4] Buildindex config.databases idxNo:0 { name: "_id_", ns: "config.databases", key: { _id: 1 } } Tue Aug 17 05:59:38 [conn4] done for 0 records 0.025secs 添加shard 1 > db.runCommand( { addshard : "192.168.100.210:27018" } ); { "shardAdded" : "shard0000", "ok" : 1 } 添加shard 2 创建shard时,两个shard不能有相同的数据库 > db.runCommand( { addshard : "192.168.100.212:27018" } ); { "ok" : 0, "errmsg" : "trying to add shard 192.168.100.212:27018 because local database recommender exists in another shard0000:192.168.100.210:27018" } 可以在192.168.100.212:27018通过 db.dropDatabase() 删除recommender db.runCommand( { addshard : "192.168.100.212:27018" } ); 查看shard节点列表 > db.runCommand( { listshards : 1 } ); { "shards" : [ { "_id" : "shard0000", "host" : "192.168.100.210:27018" }, { "_id" : "shard0001", "host" : "192.168.100.212:27018" } ], "ok" : 1 } > 将数据库设置为shard模式 > db.runCommand( { enablesharding : "test" } ); { "ok" : 1 } > db.runCommand( { enablesharding : "recommender" } ); { "ok" : 1 } > 2、登陆mongos查看对集合进行分片 [root@band-all data]# mongo 192.168.100.229:27017/admin MongoDB shell version: 1.6.0 connecting to: 192.168.100.229:27017/admin > show dbs admin local > use admin switched to db admin > db.runCommand({listshards:1}) { "shards" : [ { "_id" : "shard0000", "host" : "192.168.100.210:27018" }, { "_id" : "shard0001", "host" : "192.168.100.212:27018" } ], "ok" : 1 } > db.printShardingStatus(); --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard0000", "host" : "192.168.100.210:27018" } { "_id" : "shard0001", "host" : "192.168.100.212:27018" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } > config = connect("192.168.100.245:27019") //登陆config server connecting to: 192.168.100.245:27019 192.168.100.245:27019 > test = db.getSisterDB("test") //新增数据库test > db.runCommand( { enablesharding : "test" } ) //将test 数据库进行分片 > db.runCommand( { shardcollection : "test.people", key : {name : 1} } ) //将数据集people进行分片 > db.printShardingStatus(); --- Sharding Status --- sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard0000", "host" : "192.168.100.210:27018" } { "_id" : "shard0001", "host" : "192.168.100.212:27018" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : true, "primary" : "shard0000" } test.people chunks: { "name" : { $minKey : 1 } } -->> { "name" : { $maxKey : 1 } } on : shard0000 { "t" : 1000, "i" : 0 } > use test switched to db test > show collections people system.indexes 批量插入数据 for (var i = 1; i <= 200000; i++) db.people.save({name:"fly_people",sex:"female",age:25}); for (var i = 1; i <= 20000000; i++) db.people.save({name:"houxuefeng",sex:"male",age:30}); > db.people.stats() { "sharded" : true, "ns" : "test.people", "count" : 462008, "size" : 33264600, "avgObjSize" : 72.00005194715243, "storageSize" : 902522624, "nindexes" : 2, "nchunks" : 4, "shards" : { "shard0000" : { "ns" : "test.people", "count" : 402006, "size" : 28944448, "avgObjSize" : 72.00003980040098, "storageSize" : 891340544, "numExtents" : 22, "nindexes" : 2, "lastExtentSize" : 155079936, "paddingFactor" : 1, "flags" : 1, "totalIndexSize" : 58966016, "indexSizes" : { "_id_" : 18694144, "name_1" : 40271872 }, "ok" : 1 }, "shard0001" : { "ns" : "test.people", "count" : 60002, "size" : 4320152, "avgObjSize" : 72.00013332888904, "storageSize" : 11182080, "numExtents" : 6, "nindexes" : 2, "lastExtentSize" : 8388608, "paddingFactor" : 1, "flags" : 1, "totalIndexSize" : 5275648, "indexSizes" : { "_id_" : 2498560, "name_1" : 2777088 }, "ok" : 1 } }, "ok" : 1 } > 至此MongoDB auto shard配置完毕 备注: 问题1、 当test.people数据量达到一定量时,这也就是32位限制,新数据无法入库,查看数据库文件也不在增加 > for (var i = 1; i <= 20000; i++) db.people.save({name:"houxuefeng",sex:"male",age:30}); can't map file memory - mongo requires 64 bit build for larger datasets > > db.people.stats(); { "ns" : "test.people", "count" : 10128817, "size" : 729274896, "avgObjSize" : 72.00000710843132, "storageSize" : 891340544, "numExtents" : 22, "nindexes" : 2, "lastExtentSize" : 155079936, "paddingFactor" : 1, "flags" : 1, "totalIndexSize" : 1241788192, "indexSizes" : { "_id_" : 420325264, "name_1" : 821462928 }, "ok" : 1 } > [root@chn-xg-5-5c1 db]# ll -lh 总用量 3.0G -rwxr-xr-x 1 root root 6 8月 18 17:42 mongod.lock drwxr-xr-x 3 root root 4.0K 8月 19 10:48 moveChunk -rw------- 1 root root 64M 8月 18 17:56 test.0 -rw------- 1 root root 128M 8月 18 17:56 test.1 -rw------- 1 root root 256M 8月 19 10:46 test.2 -rw------- 1 root root 512M 8月 19 10:48 test.3 -rw------- 1 root root 512M 8月 19 10:52 test.4 -rw------- 1 root root 512M 8月 19 10:58 test.5 -rw------- 1 root root 512M 8月 19 11:04 test.6 -rw------- 1 root root 512M 8月 19 11:12 test.7 -rw------- 1 root root 16M 8月 18 17:56 test.ns drwxr-xr-x 2 root root 4.0K 8月 18 17:58 _tmp [root@chn-xg-5-5c1 db]# 问题2、 创建shard时,两个shard不能有相同的数据库 > db.runCommand( { addshard : "192.168.100.212:27018" } ); { "ok" : 0, "errmsg" : "trying to add shard 192.168.100.212:27018 because local database recommender exists in another shard0000:192.168.100.210:27018" } 可以在192.168.100.212:27018通过 db.dropDatabase() 删除recommender 问题3、 replica set 配置完毕后才可以添加为shard,否则会出错 > db.runCommand( { addshard : "foo/192.168.100.212:27018,192.168.100.212:27019,192.168.100.212:27020" } ); { "shardAdded" : "shard0001", "ok" : 1 } > |
|
来自: CevenCheng > 《分片》