Security Edge Filter For Lync Server 2013, 2010 and Office Communications Server 2007 R2

Transcription

1 Security Edge Filter For Lync Server 2013, 2010 and Office Communications Server 2007 R2 Lync-Solutions.com

2 Version Version Date Author Remarks /05/2011 Fabian Kunz Initial draft created /29/2011 Rui Maximo Edited /1/2012 Fabian Kunz Updated screenshots in section, Security Filter in Action /2/2012 Fabian Kunz Updated screenshots order in section, Installation on the Lync Edge Server /19/2013 Rui Maximo Update PowerShell registration commands. Current Document properties Property Status Status Final Publish date 11/22/2013 Lync-Solutions.com 2013 all rights reserved. This document is intellectual property of Lync-Solutions.com. No duplication or distribution allowed without written notice of the owner. No distribution outside the customer s organization allowed.

3 Table of contents

4 1 Introduction This documentation describes the requirements and installation procedure for the Standard Edition version of the Security Filter. 1.1 Problem: Denial of Service (DoS) Why are DoS attacks disruptive to your organization? Here are the most common reasons: Each failed authentication attempt to your extranet counts in Active Directory as a failed login. It becomes trivial for a remote attacker to lock out any of your AD accounts if they know (or can guess) the login name. No further credentials or privilege is required for this attack. In severe cases such as a distributed denial of service attack, this can represent a substantial vulnerability to your network. 1.2 Solution The Security Filter augments the capabilities of Microsoft s Lync Edge Server to allow a soft lockout. Security Filter is designed to tract denied authentication attempts and block further login attempts before the AD lockout limit is reached. This provides an additional tier of account security, safely locking the account out of the extranet. Security Filter prevents password guessing on the extranet by blocking authentication attempts for that account once the number of failed authentication attempts reaches a threshold. Even when the account is locked out from the extranet, the user can still login from within your corporate network or through a VPN. Thus, the DoS risk is substantially mitigated, with a minimum inconvenience. Security Filter can enforce external users to login to Lync Server from a corporate issued computer. By blocking NTLM authentication, external users are forced to sign-in using TLS- DSK authentication, which requires a client certificate to be installed on the user s computer when connected to the corporate network.

5 2 Functionality In this extraordinarily interconnected world, companies want to allow the utmost flexibility and mobility for their employees, many of whom may work remotely. Consequently, almost every organization exposes services to the Internet. However, there s always the threat of attacks. Companies are particularly concerned with Denial of Service (DoS) and password brute-force attacks. These types of attacks can be disruptive to users and consume internal server resources. The primary trouble with DoS attacks is that they re nearly indistinguishable from legitimate signin requests. The only differentiation is the frequency of sign-in attempts and their origin. A large number of sign-in attempts in rapid succession can be indicative of a DoS attack. Most DoS attacks attempt to guess the user s password to gain unauthorized access. They often result in locking out the user account if the security policy is enabled in Active Directory Domain Services, and has a maximum number of log-in attempts. The Microsoft Lync Edge Server protects against unauthorized access using industry-standard security measures. It monitors sign-in requests and enforces account lockout at the network perimeter. All communications are encrypted and authenticated. Edge Server does not protect against DoS attacks. However, Lync Server provides a flexible programming platform you can use to create server applications to intercept Session Initiation Protocol (SIP) messages on the server and perform specialized logic using the Microsoft SIP Processing Language (MSPL). This is how the security filter operates. It inspects all incoming sign-in requests on the Edge Server. The remote user is not authenticated at the Edge Server, so the sign-in request is passed to the Director or directly to the internal pool, which then performs the authentication process. The response is then passed back to the Edge Server. The security filter inspects both the request and the response. If the sign-in fails, the security filter tracks the number of failed attempts for each user account. The next time a client attempts to sign in to the same user account, and the number of failed attempts exceeds the maximum number of allowed sign-in attempts, the security filter immediately rejects the request without passing the request to the Director or internal pool for authentication. By enforcing account lockout at the Edge Server, the security filter blocks DoS attacks at the edge of the network perimeter. As a result, the security filter protects the internal Lync Server resources. Using the security filter to prevent Windows NT LAN Manager (NTLM) version 2 authentication, companies can force users to only sign in from authorized company-issued laptops. With

6 additional security measures (like using BitLocker and Group Policy to prevent users from installing unauthorized software), the corporate-issued laptops can themselves serve as a smartcard to provide two-factor authentication. To prevent brute-force attacks on user accounts, many organizations enforce an Active Directory Group Policy to lock out the account after a certain number of failed attempts. The side effect of this countermeasure is that the attacker can lock out a user s account by simply launching multiple attempts. This amounts to a DoS attack. If the account isn t protected by an Active Directory Group Policy, the attacker can use this type of brute-force attack on the user s password. These attacks use up valuable internal server resources and deny users access to their account. Uniquely identifying the user can prevent attacks on user accounts. There are several options with which to do this. You could use the source IP address, the sign-in name (that is, the SIP URI), the account name or even a combination of any of these options. After investigating each option, it seems that rogue clients mounting a DoS attack could spoof the source IP address, eliminating this choice as a way to uniquely identify the user. The sign-in name, although required to successfully sign in to Lync Server, does not authenticate the user. A sign-in name can be varied during sign-in requests, yet still lock out the same user account. Therefore, neither the source IP address nor the sign-in name are good sources with which to identify the user. Only the account name uniquely identifies the user account. You can only extract the account name, which consists of the user name and domain name, from the authentication protocol. Remote users trying to sign in and authenticate use the NTLM v2 protocol, not Kerberos. The NTLM protocol uses a three-stage handshake authentication process. The client passes the user s credentials in the third stage of the NTLM handshake. The security filter runs as a trusted server application on the Edge Server, so it s allowed to intercept this sign-in request. The security filter decodes the user name and domain name from the NTLM authentication message. Because the account name isn t available in the response, the security filter maps the response to the request using the message ID. When either the internal pool or the Director sends the authentication response to the Edge Server, the security filter captures the Register response. If the sign-in failed, the security filter counts the failed attempts. If the sign-in succeeds, the security filter resets the count of failed attempts to zero. Every time the Edge Server receives a sign-in request, it s passed to the security filter. It checks whether the sign-in request has exceeded the maximum number allowed for that particular user

7 account. If the request has not exceeded the maximum lockout count permitted, the security filter allows the request to continue to either the internal pool or the Director. If the request exceeds the maximum lockout count permitted, the security filter blocks the request and returns a 403 response. This summarily rejects the request. Any further sign-in attempts are rejected for the duration of the lockout period. After the lockout period expires, it s reset to permit new sign-in requests. One problem can occur when users sign in from a computer not joined to the corporate Active Directory domain. Lync 2013/2010 can automatically attempt to sign in using the user s local computer credentials. Because those credentials aren t corporate domain credentials, the authentication will fail. The user will eventually be blocked from signing in to Lync Server. To prevent the security filter from locking out valid users, it doesn t count these attempts against the user. Lync Server 2010 introduces support for an additional authentication protocol called TLS-DSK. This requires users to supply a client certificate for authentication. The Lync client requests certificates from Lync Server. This is an automatic process that happens the first time the user signs in to Lync Server from within the corporate network where the user is authenticated using Kerberos. This client certificate is used for authentication with any subsequent log-in attempts. This is a selfsigned certificate issued by Lync Server, not a Certificate Authority. If that same user tries to sign in to Lync from a different computer, he s authenticated using Kerberos (if inside the corporate network) or using NTLM v2 (if outside the corporate network). The process of obtaining another client certificate starts all over. TLS-DSK provides a level of security that s very close to two-factor authentication. When combined with Windows BitLocker, the computer or laptop acts as the equivalent of a smartcard (something you have). The password that BitLocker requires to boot your computer is equivalent to the pin required to authorize the use of the smartcard (something you know). There s the remote possibility someone could steal the client certificate from the user s computer, but you can mitigate this risk. Make sure corporate-issued computers are locked down to prevent users from downloading unauthorized applications. You can force the Edge Server to negotiate the authentication protocol down from TLS-DSK to NTLM v2. In this case, the attacker can still target the user s account, as discussed earlier. To prevent this scenario, the security filter provides an option to reject all NTLM v2 authentication requests, forcing TLS-DSK-only authentication. This doesn t affect federated partner connections or PIC connections.

8 3 Security Filter Design The Security Filter registers with the Edge Server where it is collocated. It intercepts all SIP REGISTER requests, extracts the user s unique login name, and tracks the number of failed login responses sent back to the remote client. When the number of failed login attempts exceeds an administrator specified threshold, the Security Filter blocks all further login attempts until the lockout period expires. This is illustrated in Figure 1, which describes the Security Filter design. Figure Security Filter Architecture 4 Security Filter Editions The Security Filter is available in two different editions. Standard Edition The Standard Edition is the perfect choice for a single Lync Edge Server deployment. There are no other requirements like a Microsoft SQL Database in the Enterprise Edition version. There is a simple installation procedure for this Edition. Enterprise Edition Deployments with multiple Edge Servers should deploy the Enterprise Edition version of the Security Filter. The Enterprise Edition version is a two-tier architecture. Every Edge Server with the Security Filter installed logs the information about the bad login attempts

9 to a Microsoft SQL Database. This guarantees that all Edge Server shares the same information about the current bad login status. 5 Considerations 5.1 Microsoft Exchange The Security Filter prevents DoS attacks over the Lync Edge Server. You should consider that Lync clients outside of the company network also authenticate users against the Microsoft Exchange environment for accessing the Exchange Availability Service and Unified Messaging information s from the internet. It is strongly recommended to implement a solution with the same functionality for Microsoft Exchange Microsoft Unified Access Gateway (UAG) 2010 If you use Microsoft UAG 2010 for Exchange publishing you can configure similar settings as for the Security Filter in the Advanced Trunk Configuration settings. For more Information about Microsoft UAG visit the Microsoft Technet Site for UAG Microsoft Threat Management Gateway (TMG) 2010 If you use Microsoft TMG 2010 for Lync publishing, consider using the Security Web Filter for TMG to prevent similar kinds of attacks. In addition to DoS and password brute-force attacks, the Security Web Filter performs deep packet inspection for XSS and SOAP layer attacks.

10 6 Requirements The Microsoft.NET 4 Framework must be installed with the latest available patches for the.net Framework on the Lync Edge Server before you start the Security Filter installation. Create a service account on the Lync Edge Server and make this account a member of the RTC Server Applications group (see Figure 2). Figure Service account for the Security Filter 7 Install Procedure 7.1 Prepare Lync Edge Server Before you can run the Security Filter, you must first register the application with your Edge Server. You only need to do this registration once by taking the following steps. Run these Lync Server 2013/2010 Windows PowerShell cmdlets with Lync Server administrative permissions: 1. Run the following Windows PowerShell cmdlet to register the security_filter application from any Lync Server except the Edge Server. Specify the fully qualified domain name (FQDN) of the Edge Server in the parameter, <Edge Server FQDN>, and KEEP the -uri parameter value set to " new-csserverapplication -identity "EdgeServer:<Edge Server FQDN>/security_filter" -uri " -critical $false

11 2. Run the following Windows PowerShell cmdlet to initiate the replication of Central Management Store configuration to the Edge Server. invoke-csmanagementstorereplication 3. Run the following Windows PowerShell cmdlet on the Edge Server to verify the proper registration of the security_filter. get-csserverapplication -localstore 4. Run the following Windows Powershell cmdlet to enable the application from any Lync Server except the Edge Server. Specify the fully qualified domain name (FQDN) of the Edge Server in the parameter <Edge Server FQDN>. Set-CsServerApplication Identity service:<edge Server FQDN>/security_filter Enabled $true 7.2 Installation on the Lync Edge Server The Security Filter requires the installation of.net 4 Framework and all available patches of this version of the.net 4 Framework. This pre-requisite can be downloaded here. The following table details the installation steps for setting up the Security Filter on your Edge Server. - Once you ve downloaded the Security Filter, run the setup.exe with local administrator privileges. - The Security Filter comprises the following two files: SecurityFilterSetup.msi Setup.exe

12 - Click Next - This page lists the PowerShell cmdlets that must be run to register the Security Filter with your Edge Server. These steps are detailed in the previous section, Preparing Lync Edge Server. - Click Next - Accept the License Agreement - Click Next

13 The Security Filter does not overwrite any changes to Active Directory's lockout counter. That value is managed internally by the domain controllers. For the next 4 screens, specify the following parameters for each unique internal Active Directory (AD) forests you may have. If you have less than four AD forests, leave the fields blank. Nebtios domain: This field specifies one of your internal domain names. This is the domain name used by remote users to authenticate to your internal Lync Servers when connecting through your Edge Server. For example, if your company, Woodgrove Bank, has the following three internal Active Directory forests (a legacy from mergers and acquisitions), woodgrovebank.com, contoso.com and fabrikam.com, and employees have accounts from each of these AD forests, you should specify the Netbios name, "woodgrovebank", as the value for this parameter, and the other two AD forest names,,contoso and fabrikam, in the same field of the subsequent two screens. These domain names are used to verify that remote users who are trying to sign in to Lync Server are connecting using credentials from one of these three domains (e.x. contoso\bob or fabrikam\alice ). Corresponding UPN suffix: The Lync client also allows users to login using their UPN name (e.x. Fabian@contoso.com) instead of their Netbios name (e.x. contoso\fabian). Specify the UPN suffix, for example contoso.com, in this field. - Click Next

14 Specify the lockout count and lockout period. Lockout count: This is the number of failed sign-in attempts that are allowed before an account lock out is enforced. The lockout count value must be set lower than the Account lockout threshold value in Active Directory. If you have multiple domains consider the Account lockout threshold value from all your domains. Lockout period (minutes): This parameter is the account lockout period. After an account is locked out, this lockout period specifies how long the account remains locked before another sign-in attempt is allowed. Any signin attempts during this lockout period are immediately rejected at the Edge Server without being proxied to the internal Lync Server for verification. The defined value should be set to a higher value than the Reset account lockout counter after value in Active Directory. Since the clocks on Lync Edge and the domain controllers may not be synchronized exactly, you should specify a higher value in the security filter settings than in Active Directory, by 3-5 minutes. This mitigates the risk that the Security Filter allows the logon attempt to be proxied by the Edge Server to the internal Lync Server for authentication while the bad password count is still in effect by the Active Directory directory service. You should also ensure that the clocks on the Lync Edge and the domain controllers are not allowed to

15 vary by more than this amount. - Click Next Specify whether remote users are permitted to authenticate using NTLM v.2. Allow remote users to login using NTLM v.2 credentials This option allows remote users to login using NLTM v2 credentials. This means that a user can log in from every computer with a Lync client installed even if this computer is not domain joined to your enterprise Active Directory and is an authorized computer. For example, select this option if your company allows users to login from their home PC s. Disallow remote users to login using NTLM v.2 credentials This option forces remote users to login using an authorized company-issued computer. With additional security measures (using BitLocker and Group Policy to prevent users from installing unauthorized software), the corporate-issued laptops can themselves serve as a smartcard to provide two-factor authentication. - Click Next

16 - To install the Security Filter to a different folder, specify the new path in the Folder field or click Browse. - Click Next - Click Next - Click Close

17 - Open the Services Management Console. - Locate the Security Filter service. - Modify the log on settings of the Security Filter service and specify the previously created service account. - Restart the Service 8 Security Filter in Action The Security Filter logs by default all information about login attempts and service interactions in the Application Windows Logs. The following table illustrates examples of possible event log entries. 1 st bad login attempt

18 Maximum bad login attempts reached and lockout enforced Lockout still enforced for an already blocked user

19 NTLM v2 login attempt rejected. Security Filter is configured only for TLS-DSK authentication and blocks NTLM v2 authentication. 9 Configuration The Security Filter installs all files to the default location, %ProgramFiles%\MB\Security Filter, during the installation process. All of the installation settings can be modified after the installation is complete from the file, security_filter_svc.exe.config. <?xml version="1.0" encoding="utf-8"?> <configuration> <appsettings> <add key="domaina" value="contoso" />

20 <add key="upna" value="contoso.com" /> <add key="domainb" value="" /> <add key="upnb" value="" /> <add key="domainc" value="" /> <add key="upnc" value="" /> <add key="domaind" value="" /> <add key="upnd" value="" /> <add key="count" value="5" /> <add key="period" value="10" /> <add key="disablentlm" value="false" /> <add key="path" value="c:\program Files\MB\Security Filter\" /> <add key="loglevel" value="verbose" /> </appsettings> </configuration> It s important to restart the Security Filter service once a change is made to make the new settings take effect. 10 Monitoring Enterprise IT environments with System Center Operations Manager (SCOM) can easily monitor bad login attempts and lockout enforcement from the Application Event Log. The most important event log entries are listed in the section, Security filter in Action.