NetOps Coding 101 building your first robot!

Transcription

1 NetOps Coding 101 building your first robot! NANOG 66 david swafford

2 what we ll accomplish 2

3 automating 3

4 the remediation of network faults 4

5 using the examples of link flaps & line card failures 5

6 based on parsing of syslog messages 6

7 focusing today 7

8 coding in Python 8

9 interacting with network devices through software 9

10 regular expressions 10

11 keeping track of what happened 11

12 12

13 how it works 13

14 system design device 1 sends message 4 remediates syslog server 2 tails 3 messages runs parser remediation 14

15 system design device 1 sends message 4 remediates syslog server 2 tails 3 messages runs parser remediation 15

16 system design device 1 sends message 4 remediates syslog server 2 tails 3 messages runs parser remediation 16

17 system design device 1 sends message 4 remediates syslog server 2 tails 3 messages runs parser remediation 17

18 live demo! 18

19 19

20 20

21 21

22 22

23 23

24 24

25 25

26 26

27 syslog message problem identified solution suggested 27

28 28

29 the lab environment 29

30 a virtual machine of Ubuntu desktop 30

31 customized with ipython! and extras Python & MySQL server & Python libs 31

32 importing the virtual appliance 32

33 download it from netengcode.com! 33

34 34

35 35

36 36

37 37

38 TERMINAL log-in details user: demo pass: demo 38

39 keyboard shortcuts to break out of the VM f enter or leave full-screen 39

40 40

41 introducing IPython! 41

42 42

43 43

44 simple prototyping 44

45 simple prototyping 45

46 scrolling back 46

47 scrolling back 47

48 leaving 48

49 a few Python basics 49

50 the basic data structures 50

51 the basic data structures strings lists tuples dictionaries 51

52 strings in Python any sequence of characters enclosed by single or double-quotes 52

53 lists in Python comma-separated values (items) between square brackets" 53

54 tuples in Python comma-separated values 54

55 lists vs tuples? 55

56 lists vs tuples the contents of a list may be changed the contents of a tuple may not be changed 56

57 quick caveat! "...it is actually the comma which makes a tuple, not the parentheses." 57

58 dictionaries ( hash tables ) 58

59 dictionaries in Python an unordered set of key: value pairs keys must be unique 59

60 dictionaries in Python 66 is the key and San Diego the value 60

61 dictionaries in Python values are accessed using the notation of [key] 61

62 try it out! 62

63 try it out! create a variable with your neighbor's name create a list of your adjacent neighbor's names create a dictionary mapping those last names to first names 63

64 try it out! create a variable with your neighbor's name name = '' create a list of your adjacent neighbor's names names = [] create a dictionary mapping those last names to first names names = {'last': 'first'} 64

65 questions? 65

66 66

67 working with modules 67

68 what is a module? 68

69 imports, or makes available, the re module part of the standard library 69

70 re is the module s namespace 70

71 as imports the module under a different namespace 71

72 methods from a module are accessed by namespace.method() 72

73 we will be using re.match() 73

74 using re.match() two inputs a regex and thing to match against 74

75 using re.match() it returns None or a match object 75

76 try it out! 76

77 try it out! import the "re" module type help(re) type re.match('eth', 'ethernet') 77

78 questions? 78

79 79

80 reading syslog messages 80

81 reading in a file assigning the file path to a variable 81

82 reading in a file opening a file handle 82

83 reading in a file using.readlines() to get a list of lines 83

84 reading in a file 84

85 try it out! 85

86 try it out! open the file "syslog.txt" try out ".readlines()" 86

87 questions? 87

88 88

89 parsing syslog messages 89

90 a syslog message 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 90

91 looking with an abstract lens 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 91

92 looking with an abstract lens DATESTAMP 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 92

93 looking with an abstract lens DATESTAMP 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 93

94 looking with an abstract lens DATESTAMP TIMESTAMP switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 94

95 looking with an abstract lens DATESTAMP TIMESTAMP DEVICE %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 95

96 looking with an abstract lens DATESTAMP TIMESTAMP DEVICE %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 96

97 looking with an abstract lens DATESTAMP TIMESTAMP DEVICE ERROR_CODE: Interface Ethernet5/1 is down (Interface removed) 97

98 looking with an abstract lens DATESTAMP TIMESTAMP DEVICE ERROR_CODE: Interface Ethernet5/1 is down (Interface removed) 98

99 looking with an abstract lens DATESTAMP TIMESTAMP DEVICE ERROR_CODE: ERROR_MESSAGE 99

100 from log message to structured data 100

101 structured data LOG MESSAGE DATESTAMP TIMESTAMP DEVICE ERROR CODE ERROR MESSAGE 101

102 102

103 regular expressions! 103

104 a way to ask does this pattern exist in a given block of text? 104

105 where this pattern is created by you 105

106 specifying the pattern is often called building a regex 106

107 an example with BGP ip as-path access-list 1 permit "^$" 107

108 "^$" beginning of line followed by end of line (an empty line!) 108

109 getting started with regular expressions in Python 109

110 try it out! 110

111 try it out! a raw string, with an opening grouping symbol 111

112 try it out! \ to indicate that a pattern-matching character follows 112

113 try it out! d to match one digit (integer) 113

114 try it out! instead of pattern-matching, we could put exact text too -- "2015" for example 114

115 try it out! + to indicate one or more (of the previous thing) 115

116 try it out! \d+ will match

117 try it out! \s will match one space 117

118 try it out! \w+ will match one or more characters (Jun) 118

119 try it out! assigning the return of re.match() to a variable (matched) 119

120 try it out! a match object was returned (it matched) 120

121 try it out! accessing the matched data using.group() 121

122 questions? 122

123 123

124 parsing syslog 124

125 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) DATESTAMP_RE = r'(\d+\s+\w+\s+\d+)' one or \d+ more digits 125

126 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) DATESTAMP_RE = r'(\d+\s+\w+\s+\d+)' one or \s+ more spaces 126

127 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) DATESTAMP_RE = r'(\d+\s+\w+\s+\d+)' one or \w+ more characters 127

128 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) DEVICE_NAME_RE = r'(\s+)' one or more \S+ non-space characters 128

129 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) ERROR_CODE_RE = r'%(\s+):' one or more \S+ non-space characters 129

130 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) ERROR_MESSAGE_RE = r'(\.*)' one or more \.* non-newline characters 130

131 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) DATESTAMP_RE = r'(\d+\s+\w+\s+\d+)' TIMESTAMP_RE = r'(\d+:\d+:\d+)' DEVICE_NAME_RE = r'(\s+)' ERROR_CODE_RE = r'%(\s+):' ERROR_MESSAGE_RE = r'(\.*)' COLUMN_DELIMITER_RE = r'(\s+)' 131

132 132

133 using the re module to match 133

134 using the re module to match 134

135 using the re module to match 135

136 using the re module to match continue skip this log line and go to the next 136

137 using the re module to match 137

138 start of group 1 end of group 1 DATESTAMP_RE = r'(\d+\s+\w+\s+\d+)' TIMESTAMP_RE = r'(\d+:\d+:\d+)' DEVICE_NAME_RE = r'(\s+)' ERROR_CODE_RE = r'%(\s+):' ERROR_MESSAGE_RE = r'(\.*)' 138

139 try it out! 139

140 try it out! 140

141 try it out! note - SYSLOG_RE is already staged 141

142 try it out! 142

143 try it out! 143

144 try it out! 144

145 try it out! 145

146 questions? 146

147 147

148 building our first remediation 148

149 first remediation a linecard failure 149

150 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 150

151 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 151

152 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 152

153 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 153

154 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 154

155 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) Did the interface regex find a match? 155

156 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) Group "1" is the data matched by the first set of ( ) 156

157 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 157

158 2015 Apr 2 14:25:06 switch1 %ETHPORT-5- IF_DOWN_INTERFACE_REMOVED: Interface Ethernet5/1 is down (Interface removed) 158

159 try it out! 159

160 try it out! parse out only the module number from: Ethernet5/1 populate that value into the string: show module {module} 160

161 sending commands over SSH 161

162 sending commands over SSH 162

163 sending commands over SSH a wrapper around Paramiko (Python SSH client) included with our demo code 163

164 no paramiko? 164

165 installing paramiko 165

166 installing paramiko 166

167 sending commands over ssh commands over ssh 167

168 sending commands over ssh commands over ssh 168

169 sending commands over ssh commands over ssh Connection opened! 169

170 sending commands over ssh 170

171 sending commands over ssh 171

172 sending commands over ssh 172

173 try it out! 173

174 try it out! open a dummy ssh connection using: import ssh_helper ssh = ssh_helper.sshsession('switch1') inspect the output from: show module 5 174

175 inspecting the output output is a list [0:5] is a slice of the first 5 lines 175

176 inspecting the output 176

177 inspecting the output checking if one string is present in another string 177

178 inspecting the output ignoring case using.lower() 178

179 inspecting the output 179

180 inspecting the output 180

181 reacting to the output using on-device filtering with egrep 181

182 reacting to the output 182

183 reacting to the output idea! parse the uptime and react based on the value 183

184 reacting to the output 184

185 questions? 185

186 186

187 next remediation link flaps & light level checking 187

188 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 188

189 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 189

190 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 190

191 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 191

192 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 192

193 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 193

194 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 194

195 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 195

196 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 196

197 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 197

198 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 198

199 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 199

200 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 200

201 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 201

202 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 202

203 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 203

204 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 204

205 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 205

206 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 206

207 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 207

208 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 208

209 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 209

210 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) "show transceiver details" 210

211 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) example output: ' Rx Power dbm...' 211

212 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) example output: ' Rx Power dbm...' 212

213 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) example output: ' Rx Power dbm...' 213

214 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) example output: ' Rx Power dbm...' 214

215 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) example output: ' Rx Power dbm...' 215

216 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) example output: ' Rx Power dbm...' 216

217 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) converting a string to a float for mathematical comparison 217

218 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) 218

219 2015 May 18 12:43:15 switch2 %ETHPORT-5- IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure) send_ () 219

220 questions? 220

221 keeping track of what happened 221

222 options in memory (lists, dictionaries, etc) on disk (as a file) in a database 222

223 comparing a few database options sqlite mysql 223

224 an event as a database table EVENT ID DATESTAMP DEVICE ERROR CODE ERROR MESSAGE RESULT 224

225 getting started with sqlite 225

226 building the database 226

227 building the database 227

228 creating events the safe way to handle string substitution 228

229 creating events auto-generated by the database 229

230 updating events 230

231 updating events 231

232 questions? 232

233 opportunities for improvement 233

234 scoping globally scoped code is a bad design use functions, classes, and modules to keep things organized 234

235 scoping globally scoped code and code inside large functions are both hard to test concise functions are easier to test testability saves much pain and suffering later 235

236 organization remediation and system code together - it can be quite a lot to digest. "sparse is better than dense" - use smaller concise files and split logically. 236

237 scaling it's not a real-time system, but would need to be. for log parsing, continuous parsing with some intelligence re: new vs old events. 237

238 stability persistence is key! a restart to the parsing script should not re-run previously handled events / remediations. 238

239 perf one by one processing is okay for a demo... in real life though, this should break up the load within the script, or as separate scripts/instances. 239

240 system design device 1 sends message 4 remediates syslog server 2 tails 3 messages runs parser remediation 240

241 system design device 1 sends message 4 remediates syslog server 2 tails 3 messages runs parser remediation 241

242 appendix 242

243 separating backend code from remediation logic 243

244 separating the backend from remediations a dictionary where the values are functions! 244

245 separating the backend from remediations 245

246 separating the backend from remediations a function 246

247 running a function who s name is not defined in this block of code 247

248 248

249 249

250 250

251 #netengcode NANOG 66 david swafford