Azure Learning Circles

Transcription

1 Azure Learning Circles Azure Management Session 1: Logs, Diagnostics & Metrics Presented By: Shane Creamer

2 Typical Customer Narratives Most customers know how to operate on-premises, but now they need to understand the essentials of Azure management They want to leverage Azure native features They want to understand who is doing what and when They want to know how the systems performs and collects diagnostics data in case of problems They want to be alerted when things don t go as expected

3 Session Agenda Azure Infrastructure as a Service Azure Management generally includes Azure Audit Logs Azure Diagnostics Azure Alerts Analytics (extended feature sets of Azure that also incur additional cost), such as: Azure Network Watcher Azure Monitor

4 Overview Source Types Activity Logs : The Azure Activity Log is a log that provides insight into the operations that were performed on resources in your subscription. Glossary: Operational logs (ASM), Activity logs (ARM) Diagnostics Logs: Azure Diagnostic Logs are logs emitted by a resource that provide rich, frequent data about the operation of that resource Metrics: Azure Insight data that enables to gain visibility into the performance and health of your workloads on Azure. Alerts: Alerts are a method of monitoring Azure resource metrics, events or logs and then being notified when a condition you specify is met. 4

5 Diagnostics: Single Server vs. the Cloud Static Environment Single well-known instance Traceable local transactions Local Access is Feasible All in one TS session Data & tools co-located In-Place Changes Single Server Dynamic Environment Multi-instance, elastic capacity Distributed work loads Local Access is Infeasible Many instances in different nodes Distributed, scaled-out data Service Upgrades Cloud

6 Diagnostics SDK component providing distributed monitoring & data collection for cloud apps Support Standard Diagnostics APIs Trace, Debug normally Cloud Friendly Manage multiple role instances centrally Scalable Choose what to collect & when to collect it Event Logs, Trace/Debug, Performance Counters, IIS Logs, Crash Dumps, Arbitrary log files Everything is remotely configurable

7 How does it work (in a nutshell)? Role Instance Starts Diagnostic Monitor Starts Monitor is configured Imperatively at Start time Remotely any time Monitor buffers data locally User can set a quota (FIFO) User initiates transfer to storage Scheduled or On Demand Role Instance Role Diagnostic Monitor Local directory storage

8 Remote Configuration Role Instance (VM) Poll Interval Role Diagnostic Monitor Local directory storage

9 Windows Azure Diagnostics Configuration Role Instance Data collection (traces, logs, crash dumps) Role Diagnostic Monitor Quota enforcement Local directory storage Windows Data Sources IIS Logs & Failed Request Logs Perf Counters Windows Event Logs Azure Network Watcher*

10 Microsoft Azure Diagnostics: Collection Process 1. Role Instance starts 2. Diagnostic Agent starts 3. Monitor is configured 1. Configuration & Extension 4. Monitor buffers data locally 1. Users can set a quota to limit log sizes 2. Uses First In / First Out (FIFO) 5. Data transferred from local buffer to Azure Storage Blob Storage Table Storage 10

11 Microsoft Azure Diagnostics: Data Types IIS Logs Azure infrastructure logs IIS Failed Request logs Windows Event logs Performance counters Crash dumps Custom error logs.net Event Source Manifest based ETW Information about IIS web sites. Information about Diagnostics agent and other process involved in data collection Information about failed requests to an IIS site or application. Information sent to the Windows event logging system. Operating System and custom performance counters. Information about the state of the process in the event of an application crash. Logs created by your application or service. Events generated by your code using the.net EventSource class. ETW events generated by any process. 11

12 Microsoft Azure Diagnostics: Data Location Azure Storage Explorer is very useful here Diagnostics Data Storage Container Location Application Logs Table WADLogsTable Event Logs Table WADWindowsEventLogsTable Performance Counters Table WADPerformanceCountersTable Infrastructure Logs Table WADDiagnosticInfrastructureLogsTable Directories monitored by diagnostic monitor Table WADDirectoriesTable IIS Logs Blob wad-iis-logfiles IIS Failed Request Log Files Blob wad-iis-failedreqlogfiles Crash dumps Blob wad-crash-dumps Custom Log Files Blob Custom Blob Container 12

13 VM connection issues due to RDP security changes

14 Scenario affecting Dormant use VM s In March of 2018 Microsoft released a CredSSP Encryption remediation for RDP This update will prevent connection of your updated On-Prem workstations to any Cloud VM s that have not been updated (dormant or test VM s) There are (4) approaches to mitigate this: 1. Change the Encryption Oracle Remediation policy to Enabled, and Protection Level to Vulnerable: - This requires a reboot on your location Workstation to take effect which means losing a lot of work progress, so not recommended. 2. If you have OMS or SCCM agents running on the VM s, consider pushing an update script that will force them to download, install, and reboot the VM s solving the issue. 3. Use the Azure Serial Console to force the VM to install the latest update - Unwieldly but allows PowerShell scripts 4. Configure your local Workstation in System\Remote\and Deselect and apply Allow connections only from computers running Remote Desktop with Network Level Authentication This is the quickest approach

15 The fastest workaround Configure your local Workstation in System\Remote\and Deselect and apply Allow connections only from computers running Remote Desktop with Network Level Authentication This is the quickest approach REG ADD HKLM\Software\Microsoft\Windows\CurrentVer sion\policies\system\credssp\parameters\ /v AllowEncryptionOracle /t REG_DWORD /d 2 Uncheck and Apply

16 An optimal Long Term Approach Allow Microsoft to enable Update Management on your VM s. This can help prevent these kinds of scenarios, especially test VM s which can lie dormant and be affected when there is a significant Microsoft security pivot.

17

18 Alerts

19 Alerts Alerts are a method of monitoring Azure resource metrics, events or logs and then being notified when a condition you specify is met. You can receive alerts based on: Metric values - The alert triggers when the value of a specified metric crosses a threshold you assign in either direction Activity log events - An alert can trigger on every event, or, only when a certain number of events occur

20

21 Other Features worth exploring Azure Monitor Provides a single source for monitoring Azure resources. You can visualize, query, route, archive, and take action on the metrics and logs coming from your Azure resources Microsoft Azure Log Integration Enables you to integrate raw logs from your Azure resources into your on-premises Security Information and Event Management (SIEM) systems Current support: Azure Audit Logs, Virtual machine logs and Azure Security Center alerts Integration of activity log in OMS - Provides you insights into activities happening on your Azure environment across multiple subscriptions

22 Metrics

23 Metrics Azure Provides Telemetry (Performance counters) to metrics for monitoring and troubleshooting your services in the Cloud Supported Services: VMs (Azure Resource Manager based), VM Scale Sets, Batch Event Hub namespace, Service Bus namespace (premium SKU only) SQL (version 12), Elastic SQL Pool Web Sites, Web Server Farms, Logic Apps, IoT Hubs, Redis Cache Networking: Application Gateways Search

24 Metrics Metrics have the following characteristics: All metrics have 1-minute frequency. You receive a metric value every minute from your resource, giving you near real-time visibility into the state and health of your resource. Metrics are available out-of-the-box without needing to opt in or setting up additional diagnostics* You can access 30 days of history for each metric. You can quickly look at the recent and monthly trends in the performance or health of your resource

25 Metrics - Actions View all metrics via the Azure portal Alert rules based on metrics Autoscale resources. Archive metrics for longer or use them for offline reporting. Stream metrics to an Event Hub. Route all metrics to OMS

26 Metrics Storage Metrics Azure Storage Metrics allows you to track your aggregated storage usage for Blobs, Tables and Queues The metrics data can be categorized as: Capacity: Provides information regarding the storage capacity consumed for the blob service, the number of containers and total number of objects stored by the service Requests: Provides summary information of requests executed against the service. It provides total number of requests, average E2E latency etc.

27 Metrics Storage Metrics Storage metrics are stored in hidden tables, can be retrieved using the Query Tables and Query Entities APIs. You have the ability to query and/or delete records in the tables. Note that the tables themselves cannot be deleted. Capacity information: $MetricsCapacityBlob table to store capacity information for blobs Transaction metrics: $MetricsTransactionsBlob: $MetricsTransactionsTable: $MetricsTransactionsQueue: This information is available for all the services Blobs, Tables and Queues.

28

29 Activity Logs

30 Activity Log First Look Get this view by clicking on the Dashboard\Activity Log button This view has saveable queries that can be pinned to the dashboard This view also allows us to select Metrics, Diagnostic Logs, Searches, and manage Alerts

31 Activity Logs Using the Activity Log, you can determine the what, who, and when for any write (PUT, POST, DELETE) operations taken on the resources in your subscription, You can also understand the status of the operation and other relevant properties. The Activity Log does not include read (GET) operations It reports control-plane events for your subscriptions

32 Activity Logs - Actions Here are some of the things you can do with the Activity Log: Query and view it in the Azure portal. Query it via REST API, PowerShell Cmdlet, or CLI. Create an or webhook alert that triggers off an Activity Log event. Export Activity Log to a Storage Account for archival or manual inspection to an Event Hub for ingestion by a third party services Analyze it in PowerBI using the PowerBI content pack.

33 Activity Logs - Archiving Export the Activity Log with Log Profiles 90 Days default storage A Log Profile controls how your Activity Log is exported. Using a Log Profile, you can configure: Where the Activity Log should be sent (Storage Account or Event Hubs) Which event categories (eg. Write, Delete, Action) should be sent Which regions (locations) should be exported How long the Activity Log should be retained in a Storage Account

34 Activity Logs Storage Analytics Storage Analytics logs detailed information about successful and failed requests to a storage service (Minimal or Verbose) You must enable it individually for each service you want to monitor (Blobs, Tables, Queues) Log entries are created only if there are requests made against the service endpoint Stored in in block blobs in a container named $logs

35

36 Analytics Initial Configuration The management blades allow us to choose a subscription, workspace, and pricing tier that we wish

37 Activity Metrics Selection panel This allows us to configure what default dashboard data we wish to see

38 Limitation of Dashboard metrics The dashboard can only display counters that have the same sample rate, since CPU is sampled at a different rate then the disks notice below the disk counters are greyed out While selectable there is a typical 1 minute delta between real-time and graphed data in the management snap-in

39 Diagnostic Logs

40 Diagnostics Logs Provides frequent data about the operation of that resource The content of these logs varies by resource type: Windows event system logs are one category of Diagnostic Log for VMs Blob, table, and queue logs are categories of Diagnostic Logs for storage accounts Not all resources support the new type of Diagnostic Logs Available: Azure LB, NSG, AppGW, KeyVault, Azure Search, Data Lake vnet GW (PS) : Start-AzureVNetGatewayDiagnostics

41 Diagnostics Logs - Actions Here are some of the things you can do with Diagnostic Logs: Save them to a Storage Account for auditing or manual inspection. Stream them to Event Hubs for ingestion by a third-party service or custom analytics solution such as PowerBI. Analyze them with OMS Log Analytics

42 Diagnostics Logs - Enabling Diagnostic Logs for non-compute resources are configured using Diagnostic Settings. Diagnostic Settings for a resource control: Where Diagnostic Logs are sent (Storage Account, Event Hubs, and/or OMS Log Analytics). Which Log Categories are sent. How long each log category should be retained in a Storage Account a retention of zero days means that logs are kept forever

43 Diagnostics Logs - VMs Diagnostic Logs VMs are enabled using Diagnostic Settings or PowerShell It deploys a VM extension add-on: Windows: Microsoft.Azure.Diagnostics.IaaSDiagnostics Linux: Microsoft.OSTCExtensions.LinuxDiagnostic) Metrics, Events, Boot Diagnostic can be enabled Configuration is based on specific schema, Azure provides a default config file You can provide your own configuration file and enable the diagnostics during deployment or post deployment on both ASM or ARM VMs

44 Diagnostics Logs Config File Data Source IIS Logs Azure Diagnostic IIS Failed Request logs Windows Event logs Performance counters Crash dumps Custom error logs.net EventSource Manifest-based ETW Description Information about IIS websites. infrastructure logsinformation about Azure Diagnostics. Information about failed requests to an IIS website or application. Information sent to the Windows event logging system. Operating system and custom performance counters. Information about the state of the process in the event of an application crash. Logs created by your application or service. Events generated by your code by using the.net EventSource class. Event Tracing for Windows events generated by any process.

45 Diagnostics Logs VMs/Tables WADWindowsEventLogs Table stores the enabled event logs from the VM WADMetrics tables stores the Metrics (performance data) from the VM with the following naming conventions: WADMetrics : Standard prefix for all WADMetrics tables, PT1H or PT1M : Signifies that the table contains aggregate data over 1 hour or 1 minute, P10D : Signifies the table will contain data for 10 days from when the table started collecting data V2S : String constant yyyymmdd : The date at which the table started collecting data *LADMetrics for Linux

46 Diagnostics Logs Storage analytics Storage Analytics logs detailed information about successful and failed requests to a storage service You must enable it individually for each service you want to monitor Log entries are created only if there are requests made against the service endpoint Stored in in block blobs in a container named $logs

47 Diagnostics Logs Storage analytics The following types of authenticated requests are logged: Successful requests Failed requests, including timeout, throttling, network, authorization, and other errors Requests using a Shared Access Signature (SAS), including failed and successful requests Requests to analytics data

48

49 Azure Essentials Questions & Discussion