IBM BigFix Relays Part 1 IBM SECURITY SUPPORT OPEN MIC November 19, 2015 Revised March 2, 2018 NOTICE: BY PARTICIPATING IN THIS CALL, YOU GIVE YOUR IRREVOCABLE CONSENT TO IBM TO RECORD ANY STATEMENTS THAT YOU MAY MAKE DURING THE CALL, AS WELL AS TO IBM S USE OF SUCH RECORDING IN ANY AND ALL MEDIA, INCLUDING FOR VIDEO POSTINGS ON YOUTUBE. IF YOU OBJECT, PLEASE DO NOT CONNECT TO THIS CALL.
Panelists Presenter: Adam McDonald - L2 Support Software Engineer Panelists: Aram Eblighatian Solutions Architect Nathan Hanner L2 Support Software Engineer Moderator: Kevin Reinstein Manager, IBM BigFix & MobileFirst Protect On-Premises 2 IBM Security
BigFix Relay Introduction
Why Relays? Think of BigFix relays as mini BigFix servers that provide the following benefits in a BigFix deployment Relieves the load on the IBM BigFix Server / redistributes load away from the Server Reduces congestion on low-bandwidth connections (gather once / download once) Allows a BigFix deployment to scale in supporting a large number of clients Operations of a relay: Registers clients Gathers site content and notifies clients of these gathers Downloads and serves up files Receives and uploads files (via the upload manager) Posts client reports up to the server 4 IBM Security
Sample deployment: Notice the: Top level relays (Datacenter) Regional level relays (Large regional office) Local level relays (Small regional office) There is some planning that needs to take place in order to run an efficient relay architecture 5 IBM Security
Another sample deployment: Other sample deployment scenarios: https://ibm.biz/bd4m6i 6 IBM Security
Internet facing relays Require firewall ports to be set to allow traffic in and out on the BigFix port (52311) Client-Relay authentication configuration is recommended for public facing relays. Not recommended for internal corporate network client-relay interactions. See the Client Authentication section in the installation guide: https://ibm.biz/bd4m6y 7 IBM Security
BigFix Relay Requirements
Capacity Requirements Capacity requirements for a relay can vary widely depending on: The number of connected clients that are downloading files. The size of each download. The period of time allotted for the downloads. Machines that are candidates to become a relay: Machines with average levels of CPU and memory at a minimum are required, check vendor recommendations and ensure proper provisioning over minimum/recommended requirements for OS. Relay operations are not CPU/memory intensive. Relays do however require plenty of free disk space to cache files. Computers must have the IBM BigFix agent installed. Personal workstations/laptops are not recommended as relay machines. Computers that are powered on all the time (Workgroup file servers and other server-quality computers) are good candidates. The IBM BigFix relay machine must have a two-way TCP connection to its parent (which can be a server or another relay). The IBM BigFix relay download cache size can be configured, but is set to 1GB by default. It is recommended that you have at least 2 GB available (10+ GB preferred) free on disk. It is recommended to have at least one relay per geographic location for bandwidth reasons. 10 IBM Security
Relay child assignment capacity What is the maximum number of endpoints that should be assigned to any given relay? 1,000 (performance degrades at numbers above 1,000) https://ibm.biz/bdfpsd Recommended: between 500 and 800; to plan for relay fail over events. In the event a relay fails, the clients can comfortably fail over to another relay without overloading it. Goal: Clients: 0 clients assigned to main root server (other than the client that is installed on the server machine). Relays: Only top level relays registered to the main server. 11 IBM Security
Communication Requirements TCP/UDP (http/https) on port 52311 (inbound/outgoing) Ensure 127.0.0.1 and localhost not blocked to allow local client to work ICMP inbound/outgoing Run pings and tracert to verify Ensure any firewalls are configured to allow this traffic through Configure components to communicate through proxies: Proxy configurations between relay and parent relay/server: https://ibm.biz/bd4m6q Proxy configurations between client and parent relay: https://ibm.biz/bdsfwx Content filtering proxies and IPS systems may allow traffic through and between relay and parent; however, they might also interfere with the downloads of site and package data. This can sometimes cause Sha1 mismatches and failed gathers and downloads Ensure content filtering proxies have proper exclusions added to their configurations to allow BigFix traffic 12 IBM Security
Deploying BigFix Relays
Relay Deployment Requirements: Client first needs to be installed/deployed to the endpoint Client computer should be registered with the server and visible as having reported in the console Deploy the same version of relay as the version the client is installed at Take action on either Fixlet 2470 or 2450 in the BES Support site and target computer to be made a relay. Fixlet # 2470: Fixlet # 2450: 14 IBM Security
Relay Deployment The relay component can also be manually downloaded and installed, same requirements: Client first needs to be installed/deployed to the endpoint Client computer should be registered with the server and visible as having reported in the console Deploy the same version of relay as the version the client is installed at Download the relay component from the download site (http://support.bigfix.com/bes/release/9.2/patch5/), and manually run the installer on the endpoint: 15 IBM Security
Breaking BigFix Relays
Various ways to break a functioning relay Remove the client (either manually or by action) on the relay machine and then re-install it Use the Client Deploy Tool to upgrade a client component on a relay machine. Doing this is a great way to break the client as well. (Use upgrade fixlets to upgrade) Force an action (somehow) from the console to upgrade only the client component. The relay component needs to be upgraded ahead of the client component. Our upgrade fixlets for the relays perform upgrades for both the relay and client components in that order. 17 IBM Security
Removing a client on a relay also requires removing the relay BES Support site: Task # 328 - Uninstall BES Relay Task # 219 - TROUBLESHOOTING: Uninstall BES Client Task # 590 - TROUBLESHOOTING: Uninstall BES Clients Mac OS X The BES Remove Utility can be downloaded from IBM developerworks : http://ibmurl.hursley.ibm.com/nl1f Steps for cleanly re-installing BigFix components on a Linux relay: https://ibm.biz/bdsfwr 18 IBM Security
Relay and Client Assignments
Relay and Client Parent Selection Assignments Relay and client parent selection assignments use the same assignment methods and settings : When assigning a relay to a parent use the manual selection method only When assigning client(s) to a relay parent you can use either a manual or automatic selection method. The client running on the main BigFix server machine uses the server as its parent and communicates with it via 127.0.0.1 (do not try to change this) The client running on a relay uses the local relay component as its relay parent and communicates with it via 127.0.0.1 (do not try to change this) Login as a master console operator to make selection assignments Selection/assignment setting changes are deployed via an action Client Same Method Same Settings Relay Same Method Same Settings Relay Same Method Same Settings Main Server 20 IBM Security
Manual Selection Method The dialog when right clicking on a single computer and choosing Edit Computer Settings... The dialog when right clicking on multiple computers and choosing Edit Computer Settings... 21 IBM Security
Automatic Selection Method (requires ICMP enabled on network) The dialog when right clicking on a single computer and choosing Edit Computer Settings... The dialog when right clicking on multiple computers and choosing Edit Computer Settings... 22 IBM Security
Selection Settings (Windows) Windows OS: settings in the registry as client settings: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BigFix\EnterpriseClient\Settings\Client RelaySelect_Automatic value [REG_SZ] set to 0 for manual relay selection method (default) value [REG_SZ] set to 1 for automatic relay selection method RelayServer1 (primary relay) value [REG_SZ] set to http://<parent_name>:52311/bfmirror/downloads/ Relay Server2 (secondary relay) value [REG_SZ] set to http://<parent_name>:52311/bfmirror/downloads/ 23 IBM Security
Selection Settings (Windows) Windows OS: settings in the registry as client settings: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BigFix\EnterpriseClient\Settings\Client _BESClient_RelaySelect_TertiaryRelayList value [REG_SZ] set to a semi-colon delimited list of relays to try For example: relay1.company.com;192.168.123.32;relay2.company.com _BESClient_RelaySelect_FailoverRelayList value [REG_SZ] set to a semi-colon delimited list of relays to try For example: relay1.company.com;192.168.123.32;relay2.company.com During manual selection, the order the relays are tried is: 1. Primary 2. Secondary 3. The relays in order on the tertiary list 4. The relays in order on the failover list 5. And finally the root server 24 IBM Security
Selection Settings (Linux) Linux OS: client settings file located at: /var/opt/besclient/besclient.config [Software\BigFix\EnterpriseClient\Settings\Client\ RelaySelect_Automatic] value = 0 effective date = Thu,%2012%20Nov%202015%2020:37:02%20-0800 [Software\BigFix\EnterpriseClient\Settings\Client\ RelayServer1] value = http://adamwin7-1:52311/bfmirror/downloads/ effective date = Thu,%2012%20Nov%202015%2020:26:59%20-0800 [Software\BigFix\EnterpriseClient\Settings\Client\ RelayServer2] value = http://spt1-win2k8r2:52311/bfmirror/downloads/ effective date = Thu,%2012%20Nov%202015%2020:26:59%20-0800 25 IBM Security
Assign a relay to a client at time of client install Windows OS: Create a three line file named clientsettings.cfg, with the following similarly formatted content, and include this file in the BigFix Client installation folder next to the client's setup.exe installer: IP:http://relay.domain.com:52311/bfmirror/downloads/ RelayServer1=http://relay.domain.com:52311/bfmirror/downloads/ RelayServer2=http://relay2.domain.com:52311/bfmirror/downloads/ Note: This technique does not work for the MSI version of the BigFix Client installer package. Mac OS: Create the same file named clientsettings.cfg with the same content as the Windows method Add it to the package file, place it at BESAgent.pkg/Contents/Resources/clientsettings.cfg 26 IBM Security
Assign a relay to a client at time of client install Linux OS: Settings stored in file /var/opt/besclient/besclient.config Ahead of installing the client: 1. Create the following directory: mkdir -p /var/opt/besclient 2. Create a besclient.config file in this directory with the following lines: 27 IBM Security [Software\BigFix\EnterpriseClient] EnterpriseClientFolder = /opt/besclient [Software\BigFix\EnterpriseClient\GlobalOptions] StoragePath = /var/opt/besclient LibPath = /opt/besclient/beslib [Software\BigFix\EnterpriseClient\Settings\Client\ RelayServer1] effective date = [Enter Current Date Time In Standard Format] value = http://relay.domain.com:52311/bfmirror/downloads/ [Software\BigFix\EnterpriseClient\Settings\Client\ RelayServer2] effective date = [Enter Current Date Time In Standard Format] value = http://relay2.domain.com:52311/bfmirror/downloads/ [Software\BigFix\EnterpriseClient\Settings\Client\ RelaySelect_Automatic] effective date = [Enter Current Date Time In Standard Format] value = 0 3. Ensure file is owned by root and not writable by anyone else 4. When the client installer is run this directory and file will not be overwritten; other default client information will be written to it.
Automatic Relay Selection Automatic Relay Selection uses ICMP. ICMP packets must be allowed to traverse network for it to work. A series of ping rounds are set out to determine closest relay. Relay information is stored in the Relays.dat file within the actionsite Clients must gather the actionsite/relays.date file (inside of the clients BESData folder) ahead of performing automatic relay selection The Relays.dat file needs to be made readable to see its contents using the Relays.dat parser tool: https://ibm.biz/bdhnuu 28 IBM Security Affiliation: Unaffiliated Name: AdamWin7-1. Port: 52311 Priority: 0 Weight: 100 Affiliation: Unaffiliated Name: spt1-rhel6.sfolab.ibm.com. Port: 52311 Priority: 0 Weight: 100 Affiliation: Unaffiliated Name: spt1-win2k8r2. Port: 52311 Priority: 0 Weight: 0
Useful Tasks and Fixlets in BES Support site Task # 432 Force BES Clients to Run Manual Relay Selection Target clients that are using the manual relay selection method Forces clients that are setup to use manual relay selection method to run through their manual selection algorithm Task # 201 Force BES Clients to Run Relay Autoselection Target clients that are using the automatic relay selection method Forces clients that are setup to use automatic relay selection method to run through their automatic relay selection algorithm These tasks are useful in the situation where an event on the network or in the deployment has caused the clients (or relays) to failover over to a different relay parent or the main BigFix server. Taking action on them with force the endpoint to go through its relay selection process and re-assign itself to a desired relay. _BESClient_RelaySelect_IntervalSeconds 6 hours (default) 29 IBM Security
Checking Relay Assignments
Checking Relay Assignments You can enable any of the following properties in the Computer's view to assist in troubleshooting. Right click on the column headings and activate: BES Client Version BES Relay Version BES Relay Installed Status * BES Relay Installed Version * BES Client's Parent Relay * BES Relay's Parent Relay * Relay Manual Selection Status BES Relay Selection Method 31 IBM Security
Checking Relay Assignments The results from analysis BES Relay Status in the BES Support Site can be used to validate proper assignments AdamWin7-2 spt1-rhel6 AdamWin7-1 spt1-win2k8r2 32 IBM Security
Managing Services
Managing Relay Services Check services, are they running? Deployment Health Checks Dashboard BES Support site 34 IBM Security
Managing Relay Services Manually restarting services: Windows: Linux: service besrelay stop service besrelay start service besclient stop service besclient start /etc/init.d/besrelay stop /etc/init.d/besrelay start net stop BESRelay net start BESRelay /etc/init.d/besclient stop /etc/init.d/besclient start net stop BESClient net start BESClient 35 IBM Security
The Download Cache
The Download Cache Least Recently Used https://ibm.biz/bdj6mb How does the BigFix Server and BigFix Relay cache work? The cache is an on demand cache, a client must request something from the relay Default size of the download cache is 1GB Location of the cache: Windows Server: \Program Files (x86)\bigfix Enterprise\BES Server\wwwrootbes\bfmirror\downloads\sha1 Linux Server: /var/opt/besserver/wwwrootbes/bfmirror/downloads/sha1 Windows Relays: \Program Files (x86)\bigfix Enterprise\BES Server\wwwrootbes\bfmirror\downloads\sha1 Linux Relays: /var/opt/besrelay/wwwrootbes/bfmirror/downloads/sha1 37 IBM Security
The Download Cache Files in the cache folder are stored with their names being their sha values 38 IBM Security
Recommendations for increasing relay cache size Deployments with a lot of activity in: Patching, Software Deployment, OSD Deployment Server and top level relays: 20+ GB Relays on either side of a low bandwidth link: 20+ GB Regional and local level relays on normal bandwidth link: 5+ GB Client setting for the cache size: _BESGather_Download_CacheLimitMB Take action on: Task # 148 - BES Relay / BES Server Setting: Download Cache Size in the BES Support site : to increase size of download cache on relay: Use Analysis # 227 - BES Relay Cache Information in the BES Support site to see which relays need their download cache increased: 39 IBM Security
Test with Blank Actions to Relays
Test with blank actions to the relays Test with mailboxed actions Test with actionsite/opsite actions 41 IBM Security
Check Deployment Health Checks
Check Deployment Health Checks 43 IBM Security
Check Relay Diagnostics Pages
Relay Diagnostics page Reporting problems: 45 IBM Security
Resetting a Relay
Resetting a relay Windows: 1. Stop the relay and client services 2. Rename/delete the bfemapfile.xml and GatherState.xml files in the C:\Program Files (x86)\bigfix Enterprise\BES Relay\Mirror Server\inbox directory 3. Delete all files and folders in the C:\Program Files (x86)\bigfix Enterprise\BES Relay\wwwrootbes\bfmirror\bfsites directory 4. Start the relay service 5. Start the client service Linux: 1. Stop the relay and client services 2. Rename/delete the bfemapfile.xml and GatherState.xml files in the /var/opt/besrelay/mirror Server/inbox/ directory 3. Delete all files and folders in the /var/opt/besrelay/wwwrootbes/bfmirror/bfsites/ directory 4. Start the relay service 5. Start the client service 47 IBM Security
Avoiding Problems
Best Practices to Avoid Problems Basics Plan and design your relay infrastructure (plan for fail over redundancy) Avoid installing relays on WinXP and Win2003 machines Choose good candidate machines to act as relays (always on / local) Ensure candidate machine resources (CPU, memory, free disk space) Install relay component at same version of client Position relay on either end of WAN links Communications TCP/UDP on port 52311 Ensure 127.0.0.1 not blocked Check presence of firewall/proxy ICMP for auto relay selection Upgrading Use upgrade fixlet to upgrade relays Breaking a relay: Do not uninstall/re-install/upgrade a client out from under a relay Do not use the Client Deploy Tool to upgrade a client 49 IBM Security
Best Practices to Avoid Problems Assignments: Do not exceed 1000 endpoints per relay (500-800) Approach 0 endpoints assigned to server Always use manual relay selection for relays Troubleshooting Checks: Check to see if services are running Check # of clients per relay (Stand up new relays / re-balance clients across additional relays) Check relay assignments (fix them if incorrect) Check relay download cache size settings (increase the cache sizes as needed) Check relay deployment health checks Check relay diagnostics page Troubleshooting Actions: Restart services (relay and client) / Reboot relay machine Take action: Force BES Clients to run relay selection Reset relay Re-install relay (requires re-installing all BigFix components) Test with blank actions Turn off relay service, restart client (trace client log) 50 IBM Security
Questions for the panel Now is your opportunity to ask questions of our panelists. To ask a question now: Press *1 to ask a question over the phone or Type your question into the Meeting chat To ask a question after this presentation: You are encouraged to participate in our Forum on this topic - https://developer.ibm.com/answers/topics/bigfix/ 51 IBM Security
Where do you get more information? Questions on this or other topics can be directed to the product forum: https://developer.ibm.com/answers/topics/bigfix/ More articles you can review: BigFix forum: https://forum.bigfix.com/ IBM developerworks articles: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/tivoli%20endpoint%20manager IBM Knowledge Center: https://www.ibm.com/support/knowledgecenter/ss63nw_9.2.0/com.ibm.tivoli.tem.doc_9.2/welcome/iem92_lan ding.html Useful links: Get started with IBM Security Support IBM My Support Sign up for My Notifications Follow us: 52 IBM Security
THANK YOU FOLLOW US ON: facebook.com/ibmsecuritysupport youtube/user/ibmsecuritysupport @askibmsecurity SecurityLearningAcademy.com securityintelligence.com xforce.ibmcloud.com Copyright IBM Corporation 2018. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM's current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM does not warrant that any systems, products or services are immune from, or will make your enterprise immune from, the malicious or illegal conduct of any party.