MongoDB on IBM Spectrum Scale

Transcription

1 MongoDB on IBM Spectrum Scale Version /17/2017 1

2 Contents Contents... 2 Introduction... 4 System and software configuration... 4 IBM Spectrum Scale... 6 MongoDB... 6 Configuration... 6 Java... 6 NUMA... 6 SELINUX... 7 Transparent huge page... 7 Firewall... 7 Limits... 7 Installation setup... 8 Creating the mongodb.repo file... 8 Installing MongoDB packages... 9 Deploying a single MongoDB server on IBM Spectrum Scale Modifying the MongoDB configuration file Create the directory in IBM Spectrum Scale Starting Mongod Checking startup Verifying the mongod deployment Mongo and python shell Import MongoDB sample dataset Mongo shell PyMongo installation Using python shell Deploying a replica set for High Availability Replica set in IBM Spectrum Scale Creating arbiter Shard cluster deployment Version /17/2017 2

3 Map out IBM Spectrum Scale directories for MongoDB hosts Deployment Verification Commands Starting mongod Stopping mongod Showing and removing databases in mongo shell Using the replication status method Using the shard status method Standalone mongod instance database repair Uninstalling MongoDB Summary Additional Resources Trademarks and special notices Version /17/2017 3

4 Introduction MongoDB is a cross-platform, open-source, document-based, non-relational (NoSQL) database that provides high performance and scalability for commodity hardware. IBM Spectrum Scale is a proven, scalable, flexible software-defined storage for high-performance data and file-management solutions. IBM Spectrum Scale enables the unification of virtualization, analytics, and file and object use cases into a single scale-out storage solution and provides a single namespace for all the data. IBM Spectrum Scale is designed to meet the needs of data-intensive applications including content repositories, technical computing, and big-data analysis. System and software configuration Depending on usage, there are different configurations that can be set up for MongoDB and IBM Spectrum Scale. MongoDB can run on all supported releases of IBM Spectrum Scale. At a minimum, IBM Spectrum Scale must be set up with a three-node cluster in a shared or File Placement Optimizer (FPO) environment and the MongoDB server can be set up on one node without a replica set. Considerations for MongoDB and IBM Spectrum Scale cluster: For MongoDB and IBM Spectrum Scale, turning off SELINUX is recommended. MongoDB automatically uses all the free memory that is not used by the WiredTiger cache or by other processes. Consideration is needed when allocating RAM, CPU, swap space, pagepool, and cache for MongoDB and IBM Spectrum Scale. See IBM Spectrum Scale tuning, memory, and swap space for more information and MongoDB hardware considerations, networking, and FAQ Storage. Review MongoDB chunk size and IBM Spectrum Scale file system creation considerations. MongoDB uses its own replica set mechanism for failover. To reduce replication overhead, it is recommended to use IBM Spectrum Scale in shared mode (e.g. ESS) with GPFS native RAID (GNR) and replication factor of one. Minimum recommended network is 10GB Ethernet. See MongoDB administration guide, FAQ fundamentals, limits, and wiredtiger storage engine for detail and setting information. See IBM Spectrum Scale planning and FAQ sections ( 2. Software questions, 3. Machine questions, 4. Disk questions) for details and settings information. Version /17/2017 4

5 For information about the settings, see the following documentation: Planning section of IBM Spectrum Scale ( 1in_PlanningForIBMSpectrumScale.htm ) 2. Software questions, 3. Machine questions, and 4. Disk questions under IBM Spectrum Scale FAQ () Test Environment The following table lists the software and hardware configuration packages used for the test environment to produce this documentation. The configuration specified in the table is not the only configuration that can be supported. Number of Nodes 8 TEST ENVIRONMENT Cores 16 RAM 125 G Networks 1G Ethernet and InfiniBand (IB) Data Disks System Disk SAS disks SAS disks OS RedHat 7.2 Kernel version el7.x86_64 Java OpenJDK 1.8.0_65 setuptools pymongo IBM Spectrum Scale MongoDB YCSB ycsb Note: The commands used in this document are for RedHat 7.2 Linux x86 system. This document is a quick guide to help configure MongoDB on IBM Spectrum Scale. This sample deployment is a test setup that uses fewer config servers and replica sets than the recommendation for a production deployment. The same deployment mechanism can be translated for deploying MongoDB on IBM Spectrum Scale on a larger scale. Version /17/2017 5

6 IBM Spectrum Scale IBM Spectrum Scale provides a single namespace over multiple types and tiers of storage. It further provides access to data for different types of workload access via POSIX, HDFS, and Object protocols. This document assumes that IBM Spectrum Scale is already deployed in the cluster. See the IBM Spectrum Scale documentation for more information on installation. MongoDB This document describes the deployment of a single MongoDB server and a shard cluster setup on IBM Spectrum Scale. A non-root user id is created to administer the MongoDB database. This document uses the mongod user ID. This document will list the differences, compared to a standard MongoDB install, required to install MongoDB Enterprise RH 3.4 on IBM Spectrum Scale. Configuration See the MongoDB administration documentation section for more information on optimizing the MongoDB deployment. This document highlights some important aspects from the MongoDB documentation. Java Install Java 1.8+ and Java OpenJDK. NUMA Having NUMA hardware will degrade MongoDB performance. MongoDB displays a warning during the startup in the MongoDB log. To disable NUMA behavior: # Disable the zone reclaimvalue in sysctl sudo sysctl -w vm.zone_reclaim_mode=0 # Start mongod, mongos, and config servers with numactl. Version /17/2017 6

7 # If numactl is not installed, install it using yum -y install numactl numactl --interleave=all <path><options> See MongoDB production documentation sections: NUMA Hardware and Configuring NUMA on Linux for more information. SELINUX The recommendation is to disable SELinux to avoid potential issues while running MongoDB. However, if you are using SELINUX, follow the Configure SELinux section in the MongoDB Red Hat installation guide and the IBM Spectrum Scale FAQ Q2.34 section and the IBM Spectrum Scale SELinux wiki page. Transparent huge page MongoDB recommends disabling transparent huge pages for better performance. echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag Firewall See MongoDB documentation on security and firewall for recommendations. See IBM Spectrum Scale Administering for information on firewall setting. Limits Refer to the MongoDB ulimit section to tune these settings based on your system resources. For example: Add the following limits to /etc/security/limits.conf Version /17/2017 7

8 * soft nproc * hard nproc * soft nofile * hard nofile Modify the /etc/security/limits.d /*-nproc.conf file to raise the nproc limit to at least for users $ cp -p /etc/security/limits.d/20-nproc.conf /etc/security/limits.d/20-nproc.conf.orig $ vi /etc/security/limits.d/20-nproc.conf * soft nproc root soft nproc unlimited Installation setup Installing a single MongoDB instance with all default values can be done by following the MongoDB Enterprise Red Hat installation guide. If you have sudo setup, use sudo to run the commands. To install on top of IBM Spectrum Scale, see the differences in configuration steps in Deploying a single MongoDB server. Creating the mongodb.repo file 1. Create themongodb.repo file on the MongoDB node under the /etc/yum.repos.d directory. [mongodb-enterprise] name=mongodb Enterprise Repository baseurl= gpgcheck=1 enabled=1 gpgkey= At the time of this write up when installing the test cluster, the MongoDB enterprise version 3.4 does not have the mongodb-enterprise rpm in the download link. To install without the mongodb-enterprise rpm metapackage file, run the command: yum install -y mongodb-enterprise* This document uses the MongoDB development repo: Version /17/2017 8

9 version This version has the mongodb-enterprise rpm which has the metapackage that will automatically install the four component packages. If the nodes in your system cannot access the internet, download the MongoDB rpm packages and key file and create a local repo. Example: For this test, set up the MongoDBYUM repository that resides on c902mnx09.gpfs.net under the /var/www/html/repos/mongodb/rh7_v3.4/enterprise_ _development directory. $ pwd; cat mongodb.repo /etc/yum.repos.d [mongodb-enterprise] name=mongodb Enterprise Repository baseurl= gpgcheck=1 enabled=1 gpgkey= $ Installing MongoDB packages 1. On the MongoDB node, install the MongoDB packages by running the following command: yum install -y mongodb-enterprise 2. Once the MongoDB packagesare installed, the following directories and files are created on the node: /etc/mongod.conf: This is the configuration fileused for mongod/mongos instances during startup. -rw-r--r-- 1 root root 768 Nov 22 17:59 /etc/mongod.conf /var/log/mongodb directory with the following permissions rwxr-xr-x 2 mongod mongod 23 Dec 2 11:47 mongodb Version /17/2017 9

10 # ls -ltr /var/log/mongodb -rw-r mongod mongod 0 Nov 22 17:59 mongod.log This directory corresponds to the /etc/mongod.conf systemlogpath directory. 3. /var/lib/mongo directory with the following permissions drwxr-xr-x 2 mongod mongod 6 Nov 22 17:59 mongo This directory corresponds to the /etc/mongod.conf storagedbpath directory. 4. Use a non-root user ID to manage MongoDB. If you are using root as the user ID to start mongod, a warning message is logged to not use the root user ID to start mongod. The default user ID after MongoDB deployment is mongod. If setting a non-default user ID to manage MongoDB, create a mongodb file in /etc/init.d/mongodb and set the configuration file directories with the correct user ID and group ID permissions. The MonoDB Enterprise version does not have the /etc/init.d/mongodb file, you can get one from github: This document will use the default user IDmongod to manage MongoDB. Deploying a single MongoDB server on IBM Spectrum Scale Modifying the MongoDB configuration file Note: The /etc/mongod.conf YAML file does not support tab characters for indentation. Use spaces instead. Change the MongoDB configuration file to use IBM Spectrum Scale. 1. Edit the /etc/mongod.conf file to have the Storage:dbPath point to a directory path on the IBM Spectrum Scale file system. Example: IBM Spectrum Scale file system mount point is /bigpfs # Where and how to store data. storage: #dbpath: /var/lib/mongo dbpath: /bigpfs/mongodb/data/csx01 journal: enabled: true Version /17/

11 Keep the systemlog: path and processmanagement: pidfilepath on the local disk for better performance. Note: Each MongoDB server needs its own unique directory path because IBM Spectrum Scale is a shared file system. Otherwise, data is clobbered writing to the same directory from different servers. The examples in this document will create a MongoDB main directory in IBM Spectrum Scale. The main directory for MongoDB will be set to /bigpfs/mongodb. Fill the following chart based on your configuration to help guide you during setup. The grey row presented below is the example dbpath. Hostname BindIp Role Data Path c902f05x01-ib mongod /bigpfs/mongodb/data/csx01 2. Modify the bindip to use a high-speed network that corresponds to a hostname instead of just using an IP address (ex ). A hostname is needed when creating a shard cluster. The hostname corresponds with this specified network IP address if the MongoDB server is accessed from aremote MongoDB client. This document uses an InfiniBand (IB) network. e.g c902f05x01-ib0.gpfs.net c902f05x01-ib0 where the bindip is and the short hostname is c902f05x01-ib0. Note: bindip can take in multiple IP addresses. Addresses are separated by a comma with no spaces.e.g. bindip: , See MongoDB Configuration File Options for more information on the configuration option values. 4. The /etc/mongod.conf is saved locally on the host. If the host stops functioning, the /etc/mongod.conf will not be accessible. You can save a copy of the /etc/mongod.conf in a directory on the IBM Spectrum Scale file system for recovery purposes. Create the directory in IBM Spectrum Scale Version /17/

12 The directory path set in the /etc/mongod.conf configuration file MUST be created in the IBM Spectrum Scale file system with the correct id and group permissions. Mongod does not start if the directories specified in the configuration file are not owned by the user ID managing mongod or they do not have the correct permissions. The default user ID/group ID for mongod instance is mongod/mongod. Ensure that the IBM Spectrum Scale directory used by the mongod instance is owned by the mongod user ID and mongod group ID. e.g. Create directories under IBM Spectrum Scale mount point for MongoDB. $ cd /bigpfs $ mkdir -p MongoDB/data/csx01 $ chown -R mongod.mongod MongoDB # Make sure the permissions are accessible by the user ID that will management mongod drwxr-xr-x 4 mongod mongod 4096 Dec 7 10:53 csx01 Starting Mongod 1. As the mongod user ID, start the mongod instance by running the command: mongod --config /etc/mongod.conf start Use the --config option to point to the correct dbpath since the path was modified to point to the IBM Spectrum Scale directory. If you do not use the --config option, the start fails because it is getting the default dbpath of /data/db. If you see WARNING message in the MongoDBlog to use numactl during startup on a NUMA machine, start mongod instance with numactl --interleave=all. 2. Install numactl if it is not available. yum -y install numactl 3. As mongod user ID, start the mongod instance by running the following command: numactl --interleave=all mongod --config /etc/mongod.conf Version /17/

13 Checking startup 1. Verify the startup is clean. 2. Check the PID process and the MongoDB log file. 4. PID ~]$ ps -ef grep mongo mongod :35? 00:00:02 mongod --config /etc/mongod.conf 5. Log file Handle any warning messages seen in the log file. The mongod process has started successfully if the log has the following line reading: [initandlisten] waiting for connections on port <port> where <port> is the port configured in /etc/mongod.conf, by default T13:35: I CONTROL [initandlisten] MongoDB starting : pid=29621 port=27017 dbpath=/bigpfs/mongodb/data/csx01 64-bit host=c902f05x01-ib T13:35: I CONTROL [initandlisten] db version v3.4.0-rc4-5-g7df8fe T13:35: I CONTROL [initandlisten] git version: 7df8fe d137516f1670d2a0091ace63ca T13:35: I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb T13:35: I CONTROL [initandlisten] allocator: tcmalloc T13:35: I CONTROL [initandlisten] modules: enterprise T13:35: I CONTROL [initandlisten] build environment: T13:35: I CONTROL [initandlisten] distmod: rhel T13:35: I CONTROL [initandlisten] distarch: x86_ T13:35: I CONTROL [initandlisten] target_arch: x86_ T13:35: I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindip: " ", port: }, processmanagement: { fork: true, pidfilepath: "/var/run/mongodb/mongod.pid" }, storage: { dbpath: "/bigpfs/mongodb/data/csx01", journal: { enabled: true } }, systemlog: { destination: "file", logappend: true, path: "/var/log/mongodb/mongod.log" } } T13:35: I - [initandlisten] Detected data files in /bigpfs/mongodb/data/csx01 created by the 'wiredtiger' storage engine, so setting the active storage engine to 'wiredtiger' T13:35: I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=63734m,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=( fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time= ),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), T13:35: I CONTROL [initandlisten] T13:35: I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database T13:35: I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted T13:35: I CONTROL [initandlisten] T13:35: I FTDC [initandlisten] Initializing full-time diagnostic data capture with Version /17/

14 directory '/bigpfs/mongodb/data/csx01/diagnostic.data' T13:35: I NETWORK [thread1] waiting for connections on port [mongod@c902f05x01-ib0 mongodb]$ Note: The access control warning is to remind the user to create the appropriate security authorization for the MongoDB environment. See MongoDB security documentation for more information. Verifying the mongod deployment Run the mongo shell to check the mongod deployment. 1. Import the MongoDB sample data from MongoDB. See the Import MongoDB sample dataset section. 2. Run mongo commands. See the Mongo shell section. Mongo and python shell Use the mongo shell and python shell to manage and test the MongoDB database. Follow the Getting Started with mongo and python shell editions. Note: Starting mongo without options by default connects to the localhost. Start mongo with the --host option to specify the correct database hostname connection. Otherwise, the following error is displayed: [mongod@c902f05x01-ib0 ~]$ mongo MongoDB shell version v3.4.0-rc4-5-g7df8fe1 connecting to: mongodb:// : T13:49: W NETWORK [main] Failed to connect to :27017, reason: Connection refused T13:49: E QUERY [main] Error: couldn't connect to server :27017, connection attempt failed : exception: connect failed [mongod@c902f05x01-ib0 ~]$ Import MongoDB sample dataset Version /17/

15 For more information on the MongoDB sample data, see the python import data MongoDB document. Note: If you set the bindip to a network IP address other than the default network IP address of , the corresponding hostname for the bindip IP address must be used to access the MongoDB database. For the mongoimport command, use the --host field to specify the corresponding bindip hostname. Example: [mongod@c902f05x01-ib0 ~]$ mongoimport --host c902f05x01-ib0 --db test --collection restaurants -- drop --file /tmp/primer-dataset.json Mongo shell Log in to the mongo shell by running the command: $ mongo If the network is not the default , use the --hostoption to specify the MongoDB database bindip hostname. Example: [mongod@c902f05x01-ib0 ~]$ mongo --host c902f05x01-ib0 Run mongo shell commands show dbs use <database> show collections Find database collection: db.<collection>.find() o E.g. db.restaurants.find() PyMongo installation To use the Python shell, install the python driver (PyMongo). The cluster used in this document has the following components installed: gcc python-devel setuptools - Version /17/

16 pymongo For more information on MongoDB and Python, see MongoDB website: Using python shell Perform the following steps to import the MongoClient to access the MongoDB database under the python shell: 1. Import pymongo. 2. From pymongo, import MongoClient. 3. client = MongoClient('MongoDB://<Hostname of bindip interface>:<port>/') e.g. client = MongoClient('MongoDB://c902f05x06-ib0:27017/') >>> import pymongo >>> from pymongo import MongoClient >>> client = MongoClient('mongodb://c902f05x01-ib0:27017/') >>> db = client.test >>> cursor = db.restaurants.find() >>> for document in cursor:... print(document)... Deploying a replica set for High Availability MongoDB recommends a minimum of three-member replica sets. The recommendation for a production deployment is to use three mongod instances for high availability. MongoDB uses specific write ordering between the replica sets to maintain high availability for the database usage. This is different than a file system replication which is what IBM Spectrum Scale offers. To configure automatic high availability and sharding in MongoDB, it is required to set up replica sets. When the primary mongod instance fails, the secondary mongod instance is automatically started. By using MongoDB replica set, the database files, operations log (oplog) and locks are properly maintained. For a single mongodb instance without a replica set, use a manual process to recover the mongodb instance. If the primary mongodb node is down, another node can be used to set the dbpath of the down primary. Because the dbpath is in the IBM Spectrum Scale filesystem, the data is already replicated. If there is an issue starting the new mongod instance, use mongod --repair to fix the error Version /17/

17 and remove the database lock. See Standalone mongod instance database repair section in this documentation. This section describes a non-shard cluster replica set. Note: In a shard cluster that requires configuration servers to use replica set, you cannot use an existing mongod replica set. The configuration server replica set must be created from a new replica set or upgraded from an existing configuration server deployment. If you plan to create a shard cluster, see Deploying a shard cluster. Replica set in IBM Spectrum Scale To deploy MongoDB replica sets, it is required to create a mongod instance then tie those instances together using the replication fields. It is advisable to plan the node roles before deploying the MongoDB cluster. Review the Replica set architecture and the Deploy replica set documentation on how to deploy replica sets in MongoDB. It is important to note that when you are deploying under IBM Spectrum Scale, the configuration file dbpath field points to a unique IBM Spectrum Scale directory and the directory is already created on the IBM Spectrum Scale file system with the correct ownership and permissions. When you are deploying MongoDB in IBM Spectrum Scale, use the Deploying a single MongoDB server section in this document as a guide for IBM Spectrum Scale specifics. For each mongod/mongos instance, the dbpath value must be a unique directory in the IBM Spectrum Scale file system. For example: ConfigServer0 (replica set) Host: c902f05x01 mongod instance dbpath = /bigpfs/mongodb/data/csx01 Host: c902f05x02 mongod instance dbpath = /bigpfs/mongodb/data/csx02 Host: c902f05x10 mongod instance dbpath = /bigpfs/mongodb/data/csx10 Mongos/Application Host: c902f05x03 mongod instance dbpath = /bigpfs/mongodb/data/appx03 Shard0 (replica set) Host: c902f05x04 mongod instance dbpath = /bigpfs/mongodb/data/shx04 Host: c902f05x05 mongod instance dbpath = /bigpfs/mongodb/data/shx05 Host: c902f05x07 mongod instance dbpath = /bigpfs/mongodb/data/shx07 Shard1 Host: c902f05x09 mongod instance dbpath = /bigpfs/mongodb/data/shx09 Version /17/

18 The MongoDB development/test shard architecture cluster configuration is used to set up a minimal number of components to help convey how MongoDB is set up on IBM Spectrum Scale. Adding replica configuration setting Replica sets are created from single mongod instances. Note that the dbpath must be set to the IBM Spectrum Scale directory with the directory pre-created on the IBM Spectrum Scale filesystem with the correct mongod admin permissions. See Deploying a single MongoDB server for more information. If you are dynamically creating using option fields during deploying replica sets, review the MongoDB documentation on Deploy replica set and ensure that the dbpath option is set correctly to the IBM Spectrum Scale directory. If you are using a configuration file to set up the cluster, ensure that the dbpath in the file is set correctly to the IBM Spectrum Scale directory. If you had already created single mongod instances and had verified the environment from previous steps, shut down the mongod instances to add in the replication settings. 1. As the mongod admin, shut down mongod. 2. Add replication in to the /etc/mongod.conf file for each of the mongod instance. Create a replication name for the set. replication: replsetname: csrs0 3. As the mongod admin, start mongod. Creating the replica set 1. Connect a mongo shell to a replica set member. This will become the primary. $ mongo --host c902f05x01-ib0 2. Initialize the replica set. The rs.initiate() will initialize the replica set for that node. MongoDB Enterprise >rs.initiate() 3. Check the host name and port values using rs.conf(). Because the bind_ip was set to the IP address, change the value to hostname. Version /17/

19 MongoDB Enterprise csrs0:primary>cfg = rs.conf() { "_id" : "csrs0", "version" : 1, "protocolversion" : NumberLong(1), "members" : [ { "_id" : 0, "host" : " :27017", "arbiteronly" : false,.. 4. Use the following commands to modify the host field value: MongoDB Enterprise csrs0:primary>cfg.members[0].host="c902f05x01-ib0" c902f05x01-ib0 MongoDB Enterprise csrs0:primary>rs.reconfig(cfg) { "ok" : 1 } 5. Add the additional replica set by running the following command: MongoDB Enterprise csrs0:primary>rs.add("c902f05x02-ib0") { "ok" : 1 } MongoDB Enterprise csrs0:primary> rs.add("c902f05x10-ib0") { "ok" : 1 } MongoDB Enterprise csrs0:primary> 6. Verify the added replica hosts are set by running the rs.conf() command in the replica hosts. Creating arbiter Arbiters are mongod instances that are part of a replica set, but do not hold data, and participate in elections to help break ties. Do not run an arbiter on the same system as a member of the replica set. See MongoDB Add replica set arbiter for more information. To create an arbiter on IBM Spectrum Scale: 1. Modify the configuration file (/etc/mongod.conf) with the following: a) Create the dbpath with the correct permission in IBM Spectrum Scale. b) Disable journaling. Version /17/

20 c) Set the correct bindip IP address. d) Set the arbiter replication:replsetname to be the same as the primary replica set. replication: replsetname: csrs0 2. Start the arbiter. 3. Connect to the primary replica set and add the arbiter to the replica set by using rs.addarb(). Shard cluster deployment Sharding is referred to as horizontal database partitioning where large data sets are partitioned and distributed across multiple nodes for easier management and faster performance. With MongoDB version 3.4, the configuration server is required to be in a replica set. In MongoDB, the shard cluster consists of the following components: Configuration server: This server stores the metadata and configuration settings for the cluster. Shard server: Each server contains a subset of the sharded data. Mongos: Acts as the query router to the shard servers. Note: You cannot create an existing mongod instance replica set that was not set up as a configuration server (Replica set in IBM Spectrum Scale) into a configuration server replica set for the shard cluster. Therefore, if you plan to deploy a shard cluster, create a new config server with replica set or upgrade from an existing config server replica set. The following is a list of MongoDB documentation on planning and deploying a shard cluster. ShardingDeploy a shard cluster Upgrade config servers to replica set (For MongoDB 3.4, there are instructions on how to convert an existing mirrored mongod instances as configuration servers (SCCC) mode which is depreciated to config serer replica sets (CSRS) mode.) Map out IBM Spectrum Scale directories for MongoDB hosts This section describes the deployment of a MongoDBshard cluster on IBM Spectrum Scale. It is best to map out the node roles and IBM Spectrum Scale directory names before deployment. The setup in this documentation is not for production. For production, plan to have at least 3 configuration server replicas sets, multiple shards, and mongos configurations. Version /17/

21 This setup demonstrates the setup of a shard cluster on IBM Spectrum Scale with replica sets. On IBM Spectrum Scale, each server requires a data directory path. This example creates a MongoDB main directory in IBM Spectrum Scale. The main directory for MongoDB is set to /bigpfs/mongodb. For MongoDB HA setup under IBM Spectrum Scale FPO mode, the data path directories can be created as IBM Spectrum Scale fileset and the fileset replica value can be set to one while the system default meta and data replica is set to 2 or 3. For example: Hostname BindIp Role Replica set name Data Path c902f05x01-ib Mongod(Config csrs0 /bigpfs/mongodb/data/csx01 Server) c902f05x02-ib Mongod(Config csrs0 /bigpfs/mongodb/data/csx02 Version /17/

22 Server) c902f05x03-ib mongos c902f05x04-ib Mongod (shard 0) c902f05x05-ib Mongod (shard 0) c902f05x07-ib Mongod (shard 0) c902f05x09-ib Mongod (shard 1) c902f05x10-ib Mongod(Config Server) shrs0 shrs0 shrs0 shrs1 csrs0 /bigpfs/mongodb/data/csx03 /bigpfs/mongodb/data/csx04 /bigpfs/mongodb/data/csx05 /bigpfs/mongodb/data/csx07 /bigpfs/mongodb/data/csx09 /bigpfs/mongodb/data/csx10 Fill in the following chart based on your configuration to help guide you during the setup: Hostname BindIp Role Replica set name Data Path Version /17/

23 Note: Instead of each MongoD host instance pointing to a different directory in the configuration file, you can use a local directory, /var/mongodb/data, on the host and link it to the IBM Spectrum Scale directory with the correct user ID and group ID permissions. For example: On c902f05x01-ib0 host: $ ln -s /bigpfs/mongodb/data/csx01 /var/mongodb/data Deployment Install MongoDB as stated in the MongoDB Installation setup section. This document uses the configuration file for the setup instead of using the command line options. The MongoDB documentation on deploying a shard cluster, configures the configuration server replica set, configures the shard replica set, installs mongos, connects mongos to the configuration server, and adds the shards to the cluster before enabling sharding for the collection. Note that the configuration server and the shard server can be independently created. This document lists the procedure to create the shard replica set servers, install mongos, create the configuration replica set servers, and use mongo to connect the configuration servers and shards to the cluster. This specific sequence prevents the configuration replica server from going down due to timeout of trying the shard registry. Shard server The shard replica set is created separately from the configuration replica set servers. The shard servers can be started and then can be added to the shard cluster through mongo when the configuration servers and mongos are deployed. Set-up configuration For each of the shard servers, install the MongoDB package. In the /etc/mongod.conf file, modify the following options: Version /17/

24 Storage.dbPath: Point to an existing IBM Spectrum Scale directory with the correct permission for the user id that manages the MongoDB cluster. Net.bindIP: Add the correct hostname IP address of the node. Optionally, one can add the local interface of The following example shows the adding multiple bindip addresses. Replication.replSetName: The replica set name for the shard. Sharding.clusterRole: The sharding role for the cluster. The value of the shard server is shardsvr. For example, the /etc/mongod.conf has the following changes: storage: dbpath: /bigpfs/mongodb/data/shx04 net: port: bindip: , # Listen to local interface only, comment to listen on all interfaces. replication: replsetname: shrs0 sharding: clusterrole: shardsvr Starting the shard servers 1. Start mongod. 2. Check that startup is successful through the pid and the log. Shard server replica sets initiation You can connect a mongo shell to one of the replica set members and use the rs.initiate() method to add the replica sets or run a shell script to initialize the replica sets. The following is an example on using the shell script to initialize the replica set. On one of the shard server as the mongod user ID, create the shardsvrreplicasets.js and runshardsvrreplicasets.sh files. Ensure that the files have execute permission. Execute the runshardsvrreplicasets.sh script to initialize the shard replica set. shardsvrreplicasets.js: The rs.initiate() method used to add in the shard replica sets: Version /17/

25 rs.initiate( { _id : "shrs0", members: [ { _id : 0, host : "c902f05x04-ib0.gpfs.net:27017" }, { _id : 1, host : "c902f05x05-ib0.gpfs.net:27017" }, { _id : 2, host : "c902f05x07-ib0.gpfs.net:27017" } ] } ) runshardsvrreplicasets.sh: The script to connect the mongo to a shard server (c902f05x04-ib0) to add in the shard replica sets. #!/bin/sh echo "Loading replica sets into shard server..." mongo --host c902f05x04-ib0 < /var/lib/mongo/shardsvrreplicasets.js The following is an example of running the runshardsvrreplicasets.sh script to add the replica sets to the shard servers: $./runshardsvrreplicasets.sh Loading replica sets into shard server... MongoDB shell version v3.4.0-rc4-5-g7df8fe1 connecting to: mongodb://c902f05x04-ib0:27017/ MongoDB server version: rc4-5-g7df8fe1 { "ok" : 1 } bye $ Additional shard replica sets installation There are two shard clusters, shard0 and shard1, as mapped in the Map out IBM Spectrum Scale directories for MongoDB hosts section. This section sets up the shard1 replica list of a single shard server. Replica sets should have more than one member; therefore, this single member setup is for documentation testing purpose only. Follow the Initiate the replica sets section to create the single replica set member with the following modification steps below: Version /17/

26 Modify the /etc/mongod.conf file: The replication.replsetname for shard1 must have its own replica set name. If additional hosts are to be added for shard1 replica, use the shard1 replsetname. # Where and how to store data. storage: dbpath: /bigpfs/mongodb/data/shx09 # network interfaces net: port: bindip: , # Listen to local interface only, comment to listen on all interfaces. replication: replsetname: shrs1 sharding: clusterrole: shardsvr Initiate the replica set. For example, on the shard1 cluster host c902f05x09-ib0, the rs.initiate() must contain the replica set name for shard1 and the mongo server must connect to the shard1 server c902f05x09-ib0 to add in the replica set into. On c902f05x09-ib0 as mongod user: $ cat shardsvrreplicasets.js rs.initiate( { _id : "shrs1", members: [ { _id : 0, host : "c902f05x09-ib0.gpfs.net:27017" }, ] } ) $ cat runshardsvrreplicaset.sh #!/bin/sh echo "Loading replica sets into shard server..." mongo --host c902f05x09-ib0 < /var/lib/mongo/shardsvrreplicasets.js $./runshardsvrreplicasets.sh Loading replica sets into shard server... MongoDB shell version v3.4.0-rc4-5-g7df8fe1 connecting to: mongodb://c902f05x09-ib0:27017/ MongoDB server version: rc4-5-g7df8fe1 Version /17/

27 { "ok" : 1 } bye $ Mongos instance installation Install MongoDB and configure the environment but do not start the mongos service until the configuration servers are set up and started. Install MongoDB package on the mongos instance node. Configure the mongos configuration file: o Remove the storage stanza. o Add the net.bindip hostname ip address. o Add the sharding.configdb values for the config server replica set sharding: configdb: <configreplsetname>/cfg1.example.net:27017,cfg2.example.net:27017,... For example, the mongos (c902f05x03-ib0) /etc/mongod.config file changes are: # Where and how to store data. #storage: # dbpath: /bigpfs/mongodb/data/appx03 # journal: # enabled: true # engine: # mmapv1: # wiredtiger: # network interfaces net: port: bindip: , # Listen to local interface only, comment to listen on all interfaces. sharding: configdb: csrs0/c902f05x01-ib0.gpfs.net:27017,c902f05x02-ib0.gpfs.net:27107,c902f05x10- ib0.gpfs.net:27107 Version /17/

28 Configuration server The configuration server replica set (CSRS) is created separately from the shard replica set servers. The configuration servers must be set up and started before starting the mongos instance. Set-up configuration For each of the config servers, install the MongoDB package. In the /etc/mongod.conf file, modify the following options: Storage.dbPath: Point to an existing IBM Spectrum Scale directory with the correct permission for the user ID that will manage the MongoDB cluster Net.bindIP: Add the correct hostname IP address of the node. Optionally, you can add the local interface of The following example shows the adding of multiple bindip addresses. Replication.replSetName: The replica set name for the configuration server. Each configuration server replica set has its own replica set name. Sharding.clusterRole: The sharding role for the cluster. The value for configuration server is configsvr. For example, the /etc/mongod.conf has the following changes: storage: dbpath: /bigpfs/mongodb/data/csx01 net: bindip: , replication: replsetname: csrs0 sharding: clusterrole: configsvr Starting the configuration server Start the configuration servers and ensure that the mongod pid is functioning and that the waiting for connections on port is in the log. Version /17/

29 Configuration replica sets initiation You can connect to a mongo shell to one of the replica set members and use the rs.initiate() method to add the replica sets or run a shell script to initialize the replica sets. The following is an example on using the shell script to initialize the replica set. On one of the config server as the mongod user ID, create the configsvrreplicasets.js and runaddreplicaconfigsvr.sh files. Ensure that the files have execute permission. Execute the runaddreplicaconfigsvr.sh script to initialize the configuration server replica set. configsvrreplicasets.js: The rs.initiate() method used to add in the config replica sets. rs.initiate( { _id: "csrs0", configsvr: true, members: [ { _id : 0, host : "c902f05x01-ib0.gpfs.net:27017" }, { _id : 1, host : "c902f05x02-ib0.gpfs.net:27017" }, { _id : 2, host : "c902f05x10-ib0.gpfs.net:27017" } ] } ) runaddreplicaconfigsvr.sh: The script to connect the mongo to a config server (c902f05x01-ib0) to add in the config server replica sets. #!/bin/sh echo "Loading replica sets into config server..." mongo --host c902f05x01-ib0 < /var/lib/mongo/configsvrreplicasets.js Run the runaddreplicaconfigsvr.sh script as the mongod user ID on a config server node to add the replica sets to the configuration servers. Check that the return code is { "ok" : 1 }. $./runaddreplicaconfigsvr.sh If you are not using runaddreplicaconfigsvr.sh script to initialize the replica set, use the mongo shell prompt and perform the following steps: As mongod user ID: Version /17/

30 $ mongo --host c902f05x01-ib0 MongoDB shell version v3.4.0-rc4-5-g7df8fe1 connecting to: mongodb://c902f05x01-ib0:27017/ MongoDB server version: rc4-5-g7df8fe1 Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see Questions? Try the support group Server has startup warnings: T05:40: I CONTROL [initandlisten] T05:40: I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database T05:40: I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted T05:40: I CONTROL [initandlisten] MongoDB Enterprise > rs.initiate(... {... _id: "csrs0",... configsvr: true,... members: [... { _id : 0, host : "c902f05x01-ib0.gpfs.net:27017" },... { _id : 1, host : "c902f05x02-ib0.gpfs.net:27017" },... { _id : 2, host : "c902f05x10-ib0.gpfs.net:27017" }... ]... }... ) { "ok" : 1 } MongoDB Enterprise csrs0:primary> Starting mongos and adding shard to a cluster After all the configuration server replica sets and all the shard server replica sets are functioning, the mongos can be started as the mongod user ID and the mongo shell can be connected to add the shard to the cluster environment. Starting mongos In this example, mongos instance is configured and running on node c902f05x03-ib0. $ mongos --config /etc/mongod.conf Version /17/

31 about to fork child process, waiting until server is ready for connections. forked process: child process started successfully, parent exiting $ Connecting a mongo shell to the started mongos instance $ mongo --host c902f05x03-ib0 Adding shards to the cluster in a mongo shell To add shards to the cluster in a mongo shell, use the sh.addshard() method: Shards to add Shard0: sh.addshard( "shrs0/c902f05x04-ib0.gpfs.net:27017") sh.addshard( "shrs0/c902f05x05-ib0.gpfs.net:27017") sh.addshard( "shrs0/c902f05x07-ib0.gpfs.net:27017") Shard1: sh.addshard( "shrs1/c902f05x09-ib0.gpfs.net:27017") Mongo shell MongoDB Enterprise mongos> sh.addshard( "shrs0/c902f05x04-ib0.gpfs.net:27017") { "shardadded" : "shrs0", "ok" : 1 } MongoDB Enterprise mongos> sh.addshard( "shrs0/c902f05x05-ib0.gpfs.net:27017") { "shardadded" : "shrs0", "ok" : 1 } MongoDB Enterprise mongos> sh.addshard( "shrs0/c902f05x07-ib0.gpfs.net:27017") { "shardadded" : "shrs0", "ok" : 1 } MongoDB Enterprise mongos> sh.addshard( "shrs1/c902f05x09-ib0.gpfs.net:27017") { "shardadded" : "shrs1", "ok" : 1 } Enabling sharding for a database After adding the shard to the cluster via the mongo shell, perform the following steps to enable sharding for a database and sharding for a collection sections. Note: Set enablesharding()for a database before executing the shardcollection command. Version /17/

32 For example: MongoDB Enterprise mongos>show dbs admin 0.000GB config 0.000GB MongoDB Enterprise mongos>use ycsb switched to db ycsb MongoDB Enterprise mongos>sh.enablesharding("ycsb") { "ok" : 1 } MongoDB Enterprise mongos> MongoDB Enterprise mongos>sh.shardcollection("ycsb.usertable", { _id : "hashed" } ) { "collectionsharded" : "ycsb.usertable", "ok" : 1 } MongoDB Enterprise mongos> MongoDB Enterprise mongos>db.printshardingstatus() --- Sharding Status --- sharding version: { "_id" : 1, "mincompatibleversion" : 5, "currentversion" : 6, "clusterid" : ObjectId(" a119f6ba81a") } shards: { "_id" : "shrs0", "host" : "shrs0/c902f05x04-ib0.gpfs.net:27017,c902f05x05- ib0.gpfs.net:27017,c902f05x07-ib0.gpfs.net:27017", "state" : 1 } active mongoses: "3.4.0-rc4-5-g7df8fe1" : 1 autosplit: Currently enabled: yes balancer: Currently enabled: yes Currently running: no Balancer lock taken at Tue Dec :57:41 GMT-0500 (EST) by ConfigServer:Balancer Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: { "_id" : "ycsb", "primary" : "shrs0", "partitioned" : true } ycsb.usertable shard key: { "_id" : "hashed" } unique: false balancing: true chunks: shrs0 2 { "_id" : { "$minkey" : 1 } } -->> { "_id" : NumberLong(0) } on : shrs0 Timestamp(1, 1) { "_id" : NumberLong(0) } -->> { "_id" : { "$maxkey" : 1 } } on : shrs0 Timestamp(1, 2) MongoDB Enterprise mongos> Version /17/

33 Verification The Yahoo! Cloud Serving Benchmark package [YCSB] ( was used to verify the deployment of the MongoDB cluster. The YCSB version used is and can be downloaded from: YCSB version ( tar.gz) While running YCSB, check the following: JAVA_HOME is set E.g. JAVA_HOME=/usr/lib/jvm/java openjdk b17.el7.x86_64/jre Use the mongodb.url option as an input option -p mongodb.url="mongodb://${hostnames}:27017${url_opts}" where Hostnames: If you are using a mongod instance, set to the mongod instance. E.g. Hostnames= c902f05x01-ib0 If you are using mongod instances as a replica set, set to the list of replica set. E.g. Hostnames="c902f05x01-ib0,c902f05x02-ib0,c902f05x10-ib0" If you are using a shard cluster, set the hostname to be the mongos instance host. E.g. Hostnames="c902f05x03-ib0" where url_opts: Requests acknowledgement that the write operation has propagated to the specified number of mongod instances. See MongoDB write concern for more information. E.g. url_opts="/?w=1" You can specify the replica set name for a Shard cluster. If the shard cluster set name is shrs0, specify the replicaset field to shrs0. Version /17/

34 E.g. url_opts="/?replicaset=shrs0&w=1 Use the -p mongodb.upsert=true so that the update can be completed without duplicate ID error especially while running YCSB workload d and workload e. Commands Starting mongod mongod --config /etc/mongod.conf or numactl --interleave=all mongod --config /etc/mongod.conf Stopping mongod mongod --config /etc/mongod.conf shutdown or numactl --interleave=all mongod --config /etc/mongod.conf --shutdown Showing and removing databases in mongo shell To list the databases, use the show dbs command. If you want to remove a database, switch to the database and then drop the database: MongoDB Enterprise mongos>show dbs admin 0.000GB config 0.001GB ycsb GB MongoDB Enterprise mongos>use ycsb switched to db ycsb MongoDB Enterprise mongos>db.dropdatabase() { "dropped" : "ycsb", "ok" : 1 } MongoDB Enterprise mongos> show dbs admin 0.000GB config 0.001GB MongoDB Enterprise mongos> Using the replication status method List the replication using the rs.status() method: Version /17/

35 $ mongo --host c902f05x04-ib0 MongoDB Enterprise shrs0:primary> rs.status() { "set" : "shrs0", "date" : ISODate(" T23:54:13.220Z"), "mystate" : 1, "term" : NumberLong(1), "heartbeatintervalmillis" : NumberLong(2000), "optimes" : { "lastcommittedoptime" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "appliedoptime" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "durableoptime" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) } }, "members" : [ { "_id" : 0, "name" : "c902f05x04-ib0.gpfs.net:27017", "health" : 1, "state" : 1, "statestr" : "PRIMARY", "uptime" : , "optime" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "optimedate" : ISODate(" T23:54:08Z"), "electiontime" : Timestamp( , 1), "electiondate" : ISODate(" T16:17:25Z"), "configversion" : 1, "self" : true }, { "_id" : 1, "name" : "c902f05x05-ib0.gpfs.net:27017", "health" : 1, "state" : 2, "statestr" : "SECONDARY", "uptime" : , Version /17/

36 }, { } ], "ok" : 1 "optime" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "optimedurable" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "optimedate" : ISODate(" T23:54:08Z"), "optimedurabledate" : ISODate(" T23:54:08Z"), "lastheartbeat" : ISODate(" T23:54:12.530Z"), "lastheartbeatrecv" : ISODate(" T23:54:12.312Z"), "pingms" : NumberLong(0), "syncingto" : "c902f05x04-ib0.gpfs.net:27017", "configversion" : 1 "_id" : 2, "name" : "c902f05x07-ib0.gpfs.net:27017", "health" : 1, "state" : 2, "statestr" : "SECONDARY", "uptime" : , "optime" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "optimedurable" : { "ts" : Timestamp( , 1), "t" : NumberLong(1) }, "optimedate" : ISODate(" T23:54:08Z"), "optimedurabledate" : ISODate(" T23:54:08Z"), "lastheartbeat" : ISODate(" T23:54:12.527Z"), "lastheartbeatrecv" : ISODate(" T23:54:12.690Z"), "pingms" : NumberLong(0), "syncingto" : "c902f05x05-ib0.gpfs.net:27017", "configversion" : 1 } MongoDB Enterprise shrs0:primary> See MongoDB reference documentation for more replication methods. Version /17/

37 Using the shard status method List the shard status by running the sh.status() method: $ mongo --host c902f05x03-ib0 MongoDB Enterprise mongos>sh.status() --- Sharding Status --- sharding version: { "_id" : 1, "mincompatibleversion" : 5, "currentversion" : 6, "clusterid" : ObjectId(" a119f6ba81a") } shards: { "_id" : "shrs0", "host" : "shrs0/c902f05x04-ib0.gpfs.net:27017,c902f05x05- ib0.gpfs.net:27017,c902f05x07-ib0.gpfs.net:27017", "state" : 1 } { "_id" : "shrs1", "host" : "shrs1/c902f05x09-ib0.gpfs.net:27017", "state" : 1 } active mongoses: "3.4.0-rc4-5-g7df8fe1" : 1 autosplit: Currently enabled: yes balancer: Currently enabled: yes Currently running: no Balancer lock taken at Tue Dec :57:41 GMT-0500 (EST) by ConfigServer:Balancer Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: { "_id" : "ycsb", "primary" : "shrs0", "partitioned" : false } MongoDB Enterprise mongos> See MongoDB reference documentation for more sharding methods. Standalone mongod instance database repair The database repair option must be run on a mongod instance with no replica set. For more information from MongoDB, see Recover from standalone mongod instance. /etc/mongod.conf sets the dbpath to the IBM Spectrum Scale directory. Therefore, if the mongod host, c902f05x06-ib0, becomes unavailable, a new host can be set up to replace the down mongod host. Version /17/