클라우드스토리지구축을 위한 ceph 설치및설정

Transcription

1 클라우드스토리지구축을 위한 ceph 설치및설정 Ph.D. Sun Park GIST, NetCS Lab

2 목차 Cloud Storage Services? Open Source Cloud Storage Softwares Introducing Ceph Storage Ceph Installation & Configuration Automatic Ceph Installation & Configuration with Chef Integrating Ceph with OpenStack Cloud Storage Semi-Production: Ceph Configuration on SmartX Boxes (Type D) Ceph 참고자료 2

3 Cloud Storage Services? Wikipedia: A file hosting service, cloud storage service, online file storage provider, or cyberlocker is an Internet hosting service specifically designed to host user files. 3

4 Open Source Cloud Storage Softwares A distributed object storage system for volume (QEMU VM) and container services (Swift/Amazon S3) and manages the disks and nodes intelligently A scalable network filesystem 4

5 Introducing Ceph Storage 5

6 History of Ceph ( ) Initially created by Sage Weil For his doctoral dissertation (University of California) After his graduation in fall 2007, Weil continued to work on Ceph fulltime In 2012, Weil created Inktank Storage In April 2014, Red Hat purchased Inktank Version Argonaut (July 3, 2012) Bobtail (v0.56, January 1, 2013), Cuttlefish (v0.61, May 7, 2013), Dumpling (v0.67, August 14, 2013), Emperor (v0.72, November 9, 2013) Firefly (v0.80, May 7, 2014), Giant (v0.87, October 29, 2014) Hammer (v0.94, April 7, 2015) 6

7 Why Ceph? A free software storage platform to present object, block, and file storage from a single distributed computer cluster The data is replicated Making it fault tolerant Running On commodity hardware Be designed to be both A high-level overview of the Ceph's internal organization Self-healing (Recovery) & Self-managing (Rebalancing) CRUSH (Controlled Replication Under Scalable Hashing) 7

8 New Version: Ceph Storage System Architecture & Component Ceph as a cloud storage solution RADOS: Reliable Autonomic Distributed Object Store 8

9 Ceph API Architecture 9

10 New Version: Ceph File Storage The Ceph filesystem CephFS has support For the native Linux kernel driver Ceph filesystem library (libcephfs) Ceph storage cluster protocol To store user's data to a reliable and distributed Ceph storage cluster To use CephFS: At least one Ceph metadata server (MDS) To be configured on any of your cluster nodes 10

11 New Version: Ceph Block Storage A new protocol RBD Ceph Block Device Native support for the Linux kernel Proprietary hypervisors Will be supported very soon: VMware and Microsoft HyperV. Full support: OpenStack, CloudStack with the cinder (block) and glance (imaging) components 1,000s of VMs in very little time 11

12 New Version: Ceph Object Storage A distributed object storage system provides an object storage interface via Ceph's object gateway (RADOS gateway: radosgw) The RADOS gateway librgw (the RADOS gateway library) and librados, Allowing applications to establish a connection with the Ceph object storage The most stable multitenant object storage solutions accessible via a RESTful API 12

13 Ceph Installation & Configuration 13

14 Ceph Installation & Configuration on PCs Environment Start [admin node] - ceph-deploy - mon - osd.2 - Meta Data Server [client PC] G ceph repository 추가 2. 마스터노드에 ceph-deploy 설치 3. 마스터노드를포함한모든노드에서는 ceph 계정생성 4. ceph 가 root 권한을사용할수있도록설정 5. 마스터노드 ( 어드민 ) 에서패스워드없이다른노드로접속할수있도록 key 생성 각각의노드들에 ssh key 를복사 6. 마스터노드에서 ~/.ssh/config 를수정 7. ceph 계정으로마스터노드에서 ceph 클러스터디렉토리를생성 8. 마스터노드에서 deploy 생성 9. 각노드에 ceph 를설치 10. ceph mon 설치 11. 각노드에서 OSD 생성 / 추가 /Activate 12. 마스터노드의설정파일을각노드에배포 13. keyring 파일퍼미션추가 14. ceph mds 설치 [ceph node #1] - mon - osd.3, osd-4 [ceph node #2] - mon - osd.0, osd-5 [ceph node #3] - mon - osd.1, osd-6 1. File System 설정 2. Block Device 설정 3. Object Storage 설정 End CPU RAM HDD OS Ceph Admin i7 16G SSD 120GB*1, HDD 1TB*1 Ubuntu Server LTS ceph-deploy, ceph Node #1 i3 16G SSD 120GB*1, HDD 1TB*2 Ubuntu Server LTS ceph Node #2 i3 16G SSD 120GB*1, HDD 1TB*2 Ubuntu Server LTS ceph Mode #3 i3 16G SSD 120GB*1, HDD 1TB*2 Ubuntu Server LTS ceph Client PC i7 8G SSD 128G Ubuntu Desktop LTS ceph

15 New Version: Ceph p+vbox Cluster Management Configuration Internet IBM M4 Provisioning Center Node osd1 osd2 osd3 osd4 vosd5 vosd6 vosd7 vosd8 Coordinator Box pboxes (physical boxes) vboxes (virtual boxes) Ceph p+vbox Cluster Power / Manage ment Control Data P M C D Ceph: H/W Recommendations - Networks 15

16 New Version: osd tree osd node in pbox osd node in vbox Ceph Dashboard 16

17 Rados Bench Testing Public network (1G) Public(1G)+Cluster(1G) network # rados bench -p data 10 write --no-cleanup # rados bench -p data 10 write --no-cleanup sec Cur ops started finished avg MB/s cur MB/s last lat avg lat Total time run: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat Total time run: Total writes made: Total writes made: Write size: Write size: Bandwidth (MB/sec): pub(1g) network Stddev Bandwidth: Max bandwidth (MB/sec): Min bandwidth (MB/sec): Average Latency: Stddev Latency: Max latency: Min latency: Bandwidth (MB/sec): pub(1g)+cluster(1g) network Stddev Bandwidth: Max bandwidth (MB/sec): Min bandwidth (MB/sec): Average Latency: Stddev Latency: Max latency: Min latency: # rados bench -p data 10 seq sec Cur ops started finished avg MB/s cur MB/s last lat avg lat Total time run: Total writes made: Write size: Bandwidth (MB/sec): Average Latency: Max latency: Min latency: # rados bench -p data 10 seq sec Cur ops started finished avg MB/s cur MB/s last lat avg lat Total time run: Total writes made: Write size: Bandwidth (MB/sec): Average Latency: Max latency: Min latency: # rados bench -p data 10 rand sec Cur ops started finished avg MB/s cur MB/s last lat avg lat Total time run: Total writes made: Write size: Bandwidth (MB/sec): Average Latency: Max latency: Min latency: # rados bench -p data 10 rand write (MB/s) seq read (MB/s) rand read (MB/s) sec Cur ops started finished avg MB/s cur MB/s last lat avg lat Total time run: Total writes made: Write size: Bandwidth (MB/sec): Average Latency: Max latency: Min latency:

18 Automatic Ceph Installation & Configuration with Chef 18

19 New Version: Ceph pbox Cluster with Chef Internet pboxes (physical boxes) Netgear GS608 V3 8ports Switch Node01 -client node1 Node02 -client node2 Node03 -client node2 Node04 -chef server & workstation ZyXEL 16 ports Switch Ubuntu LTS Power / Manage ment Control Data P M C D 19

20 Open Source Chef Server Installation Chef Version : installing Ubuntu Nginx: manual configuration for certification Chef Client Node Ceation $ sudo knife bootstrap node01 u chef P chef sudo $ sudo knife bootstrap node02 u chef P chef --sudo $ sudo knife bootstrap node03 u chef P chef --sudo 20

21 21

22 Ceph Installation with Chef Ceph cookbook upload to Chef Server: $ knife cookbook upload ceph Ceph Environment Creation & Configuration: Secrets, Monitor, Osd, IP, $ knife environment create Ceph Editing Ceph Environment File Uploading Roles: $ knife role from file ceph-mds.json $ knife role from file ceph-mon.json $ knife role from file ceph-osd.json $ knife role from file ceph-radosgw.json Assigning roles to nodes $ knife node run_list add node01 role[ceph-mon,role[osd] $ knife node run_list add node02 role[ceph-mon,role[osd] $ knife node run_list add node03 role[ceph-mon,role[osd] Ceph Node Environment Configuration: fsid, monitor-secret, osd_devices, $ knife node edit node01 $ knife node edit node02 $ knife node edit node03 Runing chef-client $ sudo knife ssh name:node01 x root chef-client 22

23 Ceph Environment Creation & Configuration Ceph Installation with Chef Ceph Node Environment Configuration 23

24 Ceph-cookbook Updated Libraries Issue: Freezing OSD bootstrap Installation Step 24

25 Integrating Ceph with OpenStack 25

26 PCs Environment for Integrating Ceph with OpenStack [OpenStack node] - RabbitMQ/MySQL - Keystone - Glance - Nova - Neutron - Cinder - Horizon G [Ceph Cluster node] [ceph node #1] - mon - osd.3, osd-4 [ceph node #2] - mon - osd.0, osd-5 [ceph node #3] - mon - osd.1, osd CPU RAM HDD OS Platform OpenStack i7 16G SSD 120GB*1, HDD 1TB*1 Ubuntu Server LTS OpenStack Kilo Node #1 i3 16G SSD 120GB*1, HDD 1TB*2 Ubuntu Server LTS ceph Node #2 i3 16G SSD 120GB*1, HDD 1TB*2 Ubuntu Server LTS ceph Mode #3 i3 16G SSD 120GB*1, HDD 1TB*2 Ubuntu Server LTS ceph Ceph Cluster i7 8G SSD 128G Ubuntu Desktop LTS ceph

27 OpenStack Network Design using Neutron 27

28 Manual OpenStack Kilo Installation with Ceph 28

29 Integrating Ceph with OpenStack Services Ceph with Nova Ceph with Glance Ceph with Cinder 29

30 Cloud Storage Semi-Production: Ceph Configuration on SmartX Boxes (Type D) 30

31 Ceph Installation & Configuration on SmartX Boxes (Type D) GIST Availability Zone 3 (Type D) SmartX Box M11 SmartX Box M12 SmartX Box M13 SmartX Box M14 SmartX Box M15 SmartX Box M16 24TB (76TB) VLAN ID = 601 Power / Management VLAN ID = 602 Control VLAN ID = 603 Data P M C D sm-box-m3 sm-box-m4 sm-box-m5 sm-box-m6 osd4 Ubuntu osd6 Ubuntu osd5 osd7 ceph-deploy mon MDS (meta data server) osd0&osd1 (6TGB) SSD (512GB or 1TB) * 1 HDD (3TG) * 4 mon osd2&osd3 (6TB) SSD (512GB or 1TB) * 1 HDD (3TG) * 4 mon osd4&osd5 (6TB) SSD (512GB or 1TB) * 1 HDD (3TG) * 4 osd6&osd7 (6TB) SSD (512GB or 1TB) * 1 HDD (3TG) * 4 osd0 Ubuntu osd2 Ubuntu osd1 osd3 Use cases Mounting CephFS by ceph-fuse tools Integrating Ceph with OpenStack Nova & Glance & Cinder & Swift Performance Tuning Using SSD for journal part network separation: client network (D) & cluster network (D) 31

32 Use Case: CephFS by ceph-fuse # ceph osd tree Ceph OSD Tree - sbox m3: osd.0 (3TB) / osd.1 (3TB) - sbox m4: osd.2 (3TB) / osd.3 (3TB) - sbox m5: osd.4 (3TB) / osd.5 (3TB) - sbox m6: osd.6 (3TB) / osd.7 (3TB) # ceph-deploy msd install sm-box-m3 # apt-get install ceph-fuse # ceph-fuse m :6789 /mnt/cephfs Client 2000MB Saving Test 32

33 Ceph Benchmarking using RADOS bench Public Network (1G) # rados bench p data 10 write no-cleanup sec Cur ops started finished avg MB/s cur MB/s last lat avg lat pub(1g) 2 16 pub(10g) pub(10g)+cluster(10g/jp) pub(10g/jp)+cluster(10g/jp) Total time run: Total writes made: 258 Write size: Bandwidth (MB/sec): # rados bench p data seq # rados bench p data 10 rand sec Cur ops started finished avg MB/s cur MB/s last lat avg lat sec Cur ops started finished avg MB/s cur MB/s last lat avg lat write (MB/s) seq read (MB/s) rand read (MB/s) Total time run: Total time run: Total writes made: 258 Total writes made: 293 Read size: Read size: Bandwidth (MB/sec): Bandwidth (MB/sec):

34 Ceph Cache Tiering for Performance Tuning Ceph Firefly release Creating a pool CRUSH map: decompile & compile Creating a cache tier EC-pool (erasure-coded) & cache-pool Configuring a cache tier To set cache policies 34

35 Ceph 참고자료 Googling Ceph Installation Guide Blog Ceph Book Edited By Karan Singh Published February 25,