Recursive inventory management

Transcription

1 Recursive inventory management Martin F. Krafft 13 Aug DebConf 13, Vaumarcus, Switzerland

2 Configuration management (system administration)

3 Configuration management CFEngine bcfg2 Puppet Chef Salt Ansible

4 Configuration management CFEngine bcfg2 Puppet Chef Salt Ansible

5 Two approaches Cloud provisioning System administration (classical)

6 Configuring the cloud Unprecedented ease! Scalability Orchestration Ad-hoc provisioning Homogeneous

7 Classical system administration

8 Classical system administration Longevity

9 Classical system administration Longevity Heterogeneous

10 Classical system administration Longevity Heterogeneous Themed hostnames, not canonical names and numbers

11 Classical system administration Longevity Heterogeneous Themed hostnames, not canonical names and numbers Laziness (vs. unprecedented ease)

12 Classical system administration Longevity Heterogeneous Themed hostnames, not canonical names and numbers Laziness (vs. unprecedented ease) Orchestration

13 Configuration management with reclass reclass comes from classical system administration

14 Configuration management with reclass reclass comes from classical system administration It might well suit your cloud needs

15 Configuration management with reclass reclass comes from classical system administration It might well suit your cloud needs (might require rethinking)

16 Targeting nodes vs. classifying hosts

17 Targeting nodes Mailservers

18 Targeting nodes Rack somewhere Mailservers

19 Targeting nodes Rack somewhere /28 Mailservers

20 Targeting nodes has debian fact Rack somewhere Debian /28 Mailservers

21 Targeting nodes has debian fact Debian Rack somewhere *.zurich.corp /28 Zurich Mailservers

22 Targeting nodes has debian fact Backwards! Debian /28 Rack somewhere *.zurich.corp Zurich Mailservers

23 Targeting nodes has debian fact Rack somewhere Backwards! Debian *.zurich.corp To what /28 classes does a specific node belong? Mailservers Zurich

24 Targeting nodes has debian fact Rack somewhere Backwards! Debian *.zurich.corp To what /28 classes does a specific node belong? Behaviour is partially reactive, may depend on data on host Mailservers Zurich

25 Classifying hosts blue.example.org

26 Classifying hosts mailserver blue.example.org ntpclient

27 Classifying hosts mailserver blue.example.org ntpclient server = 0.pool.ntp.org hosted@zurich

28 Classifying hosts mailserver blue.example.org ntpclient server = 0.pool.ntp.org hosted@zurich server = 0.ch.pool.ntp.org

29 Classifying hosts white.example.org

30 Classifying hosts webserver white.example.org ntpclient server = 0.pool.ntp.org hosted@munich

31 Classifying hosts webserver white.example.org ntpclient server = 0.pool.ntp.org hosted@munich server = 0.de.pool.ntp.org

32 Puppet node blue.example.org { $ntpserver = red.example.org include common include ntp }

33 Puppet node common { $ntpserver = 0.pool.ntp.org include common include ntp } node blue. example.org inherits common { }

34 Puppet node common { $ntpserver = 0.pool.ntp.org include common include ntp } node blue. example.org inherits common { $ntpserver = red.example.org }

35 Puppet Anger! node common { $ntpserver = 0.pool.ntp.org include common include ntp } node blue. example.org inherits common { $ntpserver = red.example.org }

36 Puppet Anger! node common { $ntpserver = 0.pool.ntp.org include common No multiple inheritance include ntp } node blue. example.org inherits common { $ntpserver = red.example.org }

37 Puppet Anger! node common { $ntpserver = 0.pool.ntp.org include common No multiple inheritance include ntp } Inheritance generally discouraged (cf. docs) node blue. example.org inherits common { $ntpserver = red.example.org }

38 Puppet Anger! node common { $ntpserver = 0.pool.ntp.org include common No multiple inheritance include ntp } Inheritance generally discouraged (cf. docs) node blue. example.org inherits common { $ntpserver = red.example.org } Parametrisation unnecessarily difficult

39 System administration automation principles (abridged)

40 System administration automation principles (abridged) Central control, versioned

41 System administration automation principles (abridged) Central control, versioned Parametrisation (no special casing in code)

42 System administration automation principles (abridged) Central control, versioned Parametrisation (no special casing in code) No redundancy

43 System administration automation principles (abridged) Central control, versioned Parametrisation (no special casing in code) No redundancy Use node information as parameters, don t rely on them for behaviour selection

44 recursive external (node) classifier

45 Data use by the CMS What applications should a role have? (aka. modules, states, playbooks)

46 Data use by the CMS What applications should a role have? (aka. modules, states, playbooks) How does this node differ from all other nodes that have the same application? (aka. parameters, pillar, variables)

47 Data use by the CMS What applications should a role have? (aka. modules, states, playbooks) How does this node differ from all other nodes that have the same application? (aka. parameters, pillar, variables) Which nodes belong to a group? (aka. nodegroups, clusters)

48 Usage by the system administrator

49 Usage by the system administrator Deploy and manage site-wide configuration changes

50 Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged

51 Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged Update /etc/motd on all hosts

52 Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged Update /etc/motd on all hosts Fetch logs of all hosts tagged mailserver

53 Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged Update /etc/motd on all hosts Fetch logs of all hosts tagged mailserver reclass just assembles/provides the data

54 Configuration management vs. remote execution?

55 Configuration management & remote execution!

56 Single data source

57 reclass adapters Adapters interface between configuration management system and reclass

58 reclass adapters Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.)

59 reclass adapters Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON,... )

60 reclass adapters Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON,... ) Adapters provided: Puppet (not yet re-implemented)

61 reclass adapters Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON,... ) Adapters provided: Puppet (not yet re-implemented) Salt (module)

62 reclass adapters Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON,... ) Adapters provided: Puppet (not yet re-implemented) Salt (module) Ansible (exec required)

63 reclass and Puppet Original reclass written for Puppet

64 reclass and Puppet Original reclass written for Puppet... out of frustration

65 reclass and Puppet Original reclass written for Puppet... out of frustration Rage-quit Puppet two years ago

66 reclass and Puppet Original reclass written for Puppet... out of frustration Rage-quit Puppet two years ago Rewritten reclass from scratch since then

67 reclass and Puppet Original reclass written for Puppet... out of frustration Rage-quit Puppet two years ago Rewritten reclass from scratch since then Could not be bothered to reimplement

68 reclass and Puppet Original reclass written for Puppet... out of frustration Rage-quit Puppet two years ago Rewritten reclass from scratch since then Could not be bothered to reimplement Trivially done through adapter plugin

69 reclass and Salt Provides top and pillar data

70 reclass and Salt Provides top and pillar data Adapter is a Salt module, since

71 reclass and Salt Provides top and pillar data Adapter is a Salt module, since nodegroups not yet implemented (Salt issue #5787)

72 reclass and Ansible Provides inventory and node information

73 reclass and Ansible Provides inventory and node information Implemented as external script

74 reclass and Ansible Provides inventory and node information Implemented as external script Does not yet support batched calls (recent Ansible feature)

75 Enough of this boring stuff!

76 Parametrisation is key Parametrise modules as much as possible

77 Parametrisation is key Parametrise modules as much as sensible

78 Parametrisation is key Parametrise modules as much as sensible At all cost, avoid special-casing in module code

79 Parametrisation is key Parametrise modules as much as sensible At all cost, avoid special-casing in module code Reclass allows you to keep your parameters modular

80 Parametrisation is key Parametrise modules as much as sensible At all cost, avoid special-casing in module code Reclass allows you to keep your parameters modular Define your data in one place only (no redundancy)

81 reclass node definition blue. example. org.yaml: --- applications : - postfix - ntp parameters : ntp : server : 0. pool. ntp. org

82 reclass node definition blue. example. org.yaml: --- applications : - postfix - ntp parameters : ntp : server : 0. pool. ntp. org

83 But wait! You promised recursion!

84 yaml fs YAML files for nodes in $inventory base uri/nodes

85 yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes

86 yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes Nodes and classes files may specify classes to inherit

87 yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes Nodes and classes files may specify classes to inherit You can think of classes as tags, too!

88 yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes Nodes and classes files may specify classes to inherit You can think of classes as tags, too! Smart (deep) merging on return from recursive descent walk

89 reclass node definition nodes / blue. example. org. yaml : classes : - common - mailserver parameters : ntp : server : 0.ch.pool.ntp.org classes / common. yaml : applications : - ntp parameters : ntp : server : 0. pool. ntp. org

90 reclass node definition nodes / blue. example. org. yaml : classes : - common - mailserver parameters : ntp : server : 0.ch.pool.ntp.org classes / common. yaml : applications : - ntp parameters : ntp : server : 0. pool. ntp. org

91 reclass node definition nodes / blue. example. org. yaml : classes : - common - mailserver parameters : ntp : server : 0.ch.pool.ntp.org classes / common. yaml : applications : - ntp parameters : ntp : server : 0. pool. ntp. org

92 More specific classes override data defined in less specific classes

93 reclass node definition nodes / blue. example. org. yaml : classes : - common - mailserver - hosted@zurich classes /hosted@zurich.yaml: parameters : ntp : server : 0.ch.pool.ntp.org

94 reclass node definition nodes / blue. example. org. yaml : classes : - common - mailserver - hosted@zurich classes /hosted@zurich.yaml: parameters : ntp : server : 0.ch.pool.ntp.org

95 Multiple inheritance well-defined order

96 reclass node definition nodes / blue. example. org. yaml : classes : - ssh-server classes /ssh-server.yaml parameters : permit root login: no

97 reclass node definition nodes / blue. example. org. yaml : classes : - ssh-server - backup-client classes /ssh-server.yaml parameters : permit root login: no classes /backup-client.yaml parameters : permit root login: without-password

98 reclass node definition nodes / blue. example. org. yaml : classes : - ssh-server - backup-client classes /ssh-server.yaml parameters : permit root login: no classes /backup-client.yaml parameters : permit root login: without-password classes: - ssh-server

99 Parameter interpolation nodes / diamond. example. org. yaml : classes : - motd classes /motd.yaml parameters : motd: message: ${floyd reference}

100 Parameter interpolation nodes / diamond. example. org. yaml : classes : - motd parameters: - floyd reference: Shine on, you crazy diamond classes /motd.yaml parameters : motd: message: ${floyd reference}

101 reclass future work Package it.

102 reclass future work Package it. Doh!

103 reclass future work Package it. Doh! preseed.cfg/d-i adapter?

104 reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp class mappings)

105 reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp class mappings) Membership lists

106 reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp class mappings) Membership lists Other data sources?

107 reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp class mappings) Membership lists Other data sources? Better unit testing (without philosophical debates)

108 reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp class mappings) Membership lists Other data sources? Better unit testing (without philosophical debates) Your idea here!