Lightweight AccessAgent mode on Terminal Server SDK

Transcription

1 IBM Security Access Manager for Enterprise Single Sign-On Version 8.2 Lightweight AccessAgent mode on Terminal Serer SDK SC

2

3 IBM Security Access Manager for Enterprise Single Sign-On Version 8.2 Lightweight AccessAgent mode on Terminal Serer SDK SC

4 Note Before using this information and the product it supports, read the information in Notices on page 27. Edition notice Note: This edition applies to ersion 8.2 of IBM Security Access Manager for Enterprise Single Sign-On, (product number 5724 V67) and to all subsequent releases and modifications until otherwise indicated in new editions. Copyright IBM Corporation 2002, US Goernment Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Chapter 1. Oeriew Chapter 2. Deeloping a Virtual Channel Connector Build prerequisites AccessAgent Terminal Serices SDK Setting up the cross-compiler Setting up the build enironment Linking dependencies Virtual Channel Connector DLL Loading the EncVcClient library Chapter 3. Setting up Virtual Channel integration Integrating Client AccessAgent with Citrix Integrating the Serer AccessAgent with Citrix Serer Chapter 5. Troubleshooting and Support Appendix A. Terminal Serices SDK API Details Appendix B. AccessAgent Terminal Serices SDK message exchange Notices Glossary Index Chapter 4. Verifying the operation Copyright IBM Corp. 2002, 2012 iii

6 i IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

7 About this publication Intended audience IBM Security Access Manager for Enterprise Single Sign-On proides sign-on and sign-off automation, authentication management, and user tracking to proide a seamless path to strong digital identity. The IBM Security Access Manager for Enterprise Single Sign-On Lightweight AccessAgent mode on Terminal Serer SDK Guide proides information about how to deelop a irtual channel connector that integrates AccessAgent with Terminal Serices applications. This publication is for the Administrators who need to deelop the irtual channel connector. Readers must be familiar with the following topics: Virtual channel connector and integration AccessAgent Terminal Serices SDK Vendor SDK What this publication contains Publications This publication contains the following sections: Chapter 1, Oeriew, on page 1 Proides an oeriew of the Client AccessAgent and Serer AccessAgent communication. Chapter 2, Deeloping a Virtual Channel Connector, on page 3 Proides the build prerequisites, and information about the AccessAgent terminal serices SDK. Chapter 3, Setting up Virtual Channel integration, on page 11 Proides the procedure for setting up the irtual channel connector for Client AccessAgent and Serer AccessAgent. Chapter 4, Verifying the operation, on page 13 Proides the procedure for erifying if the Virtual Channel Connector is working properly. Chapter 5, Troubleshooting and Support, on page 15 Proides the requirements for troubleshooting. Appendix A, Terminal Serices SDK API Details, on page 17 Proides the details of the AccessAgent Terminal Serices SDK. Appendix B, AccessAgent Terminal Serices SDK message exchange, on page 25 Proides information about the message exchange scenario between the Client AccessAgent and Serer AccessAgent and descriptions for each message type. This section lists publications in the IBM Security Access Manager for Enterprise Single Sign-On library. The section also describes how to access Tioli publications online and how to order Tioli publications. Copyright IBM Corp. 2002, 2012

8 IBM Security Access Manager for Enterprise Single Sign-On library The following documents are aailable in the IBM Security Access Manager for Enterprise Single Sign-On library: IBM Security Access Manager for Enterprise Single Sign-On Quick Start Guide, CF38DML Read this guide for a quick start on the main installation and configuration tasks to deploy and use IBM Security Access Manager for Enterprise Single Sign-On. IBM Security Access Manager for Enterprise Single Sign-On Planning and Deployment Guide, SC Read this guide before you do any installation or configuration tasks. This guide helps you to plan your deployment and prepare your enironment. It proides an oeriew of the product features and components, the required installation and configuration, and the different deployment scenarios. It also describes how to achiee high aailability and disaster recoery. IBM Security Access Manager for Enterprise Single Sign-On Installation Guide, GI Read this guide for the detailed procedures on installation, upgrade, or uninstallation of IBM Security Access Manager for Enterprise Single Sign-On. This guide helps you to install the different product components and their required middleware, and also do the initial configurations required to complete the product deployment. It coers procedures for using irtual appliance, WebSphere Application Serer Base editions, and Network Deployment. IBM Security Access Manager for Enterprise Single Sign-On Configuration Guide, GC Read this guide if you want to configure the IMS Serer settings, the AccessAgent user interface, and its behaior. IBM Security Access Manager for Enterprise Single Sign-On Administrator Guide, SC This guide is intended for the Administrators. It coers the different Administrator tasks. This guide proides procedures for creating and assigning policy templates, editing policy alues, generating logs and reports, and backing up the IMS Serer and its database. Use this guide together with the IBM Security Access Manager for Enterprise Single Sign-On Policies Definition Guide. IBM Security Access Manager for Enterprise Single Sign-On Help Desk Guide, SC This guide is intended for Help desk officers. The guide helps Help desk officers to manage queries and requests from users usually about their authentication factors. Use this guide together with the IBM Security Access Manager for Enterprise Single Sign-On Policies Definition Guide. IBM Security Access Manager for Enterprise Single Sign-On Policies Definition Guide, SC Read this guide for the detailed descriptions of the different user, machine, and system policies that Administrators can configure in AccessAdmin. Use this guide along with the IBM Security Access Manager for Enterprise Single Sign-On Administrator Guide. IBM Security Access Manager for Enterprise Single Sign-On Troubleshooting and Support Guide, GC Read this guide if you hae any issues with regards to installation, upgrade, and product usage. This guide coers the known issues and limitations of the i IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

9 product. It helps you determine the symptoms and workaround for the problem. It also proides information about fixes, knowledge bases, and support. IBM Security Access Manager for Enterprise Single Sign-On AccessStudio Guide, SC Read this guide if you want to create or edit profiles. This guide proides procedures for creating and editing standard and adanced AccessProfiles for different application types. It also coers information about managing authentication serices and application objects, and information about other functions and features of AccessStudio. IBM Security Access Manager for Enterprise Single Sign-On Proisioning Integration Guide, SC Read this guide for information about the different Jaa and SOAP API for proisioning. It also coers procedures for installing and configuring the Proisioning Agent. IBM Security Access Manager for Enterprise Single Sign-On Web API for Credential Management Guide, SC Read this guide if you want to install and configure the Web API for credential management. IBM Security Access Manager for Enterprise Single Sign-On Lightweight AccessAgent mode on Terminal Serer SDK Guide, SC Read this guide for the details on how to deelop a irtual channel connector that integrates AccessAgent with Terminal Serices applications. IBM Security Access Manager for Enterprise Single Sign-On Serial ID SPI Guide, SC IBM Security Access Manager for Enterprise Single Sign-On has a Serice Proider Interface (SPI) for deices that contain serial numbers, such as RFID. See this guide to know how to integrate any deice with serial numbers and use it as a second authentication factor with AccessAgent. IBM Security Access Manager for Enterprise Single Sign-On Context Management Integration Guide, SC Read this guide if you want to install and configure the Context Management solution. IBM Security Access Manager for Enterprise Single Sign-On User Guide, SC This guide is intended for the end users. This guide proides instructions for using AccessAgent and Web Workplace. IBM Security Access Manager for Enterprise Single Sign-On Error Message Reference Guide, GC This guide describes all the informational, warning, and error messages associated with IBM Security Access Manager for Enterprise Single Sign-On. Accessing terminology online The IBM Terminology Web site consolidates the terminology from IBM product libraries in one conenient location. You can access the Terminology Web site at the following Web address: Accessing publications online IBM posts publications for this and all other Tioli products, as they become aailable and wheneer they are updated, to the Tioli Information Center Web site at About this publication ii

10 Note: If you print PDF documents on other than letter-sized paper, set the option in the File > Print window that allows Adobe Reader to print letter-sized pages on your local paper. Ordering publications You can order many Tioli publications online at You can also order by telephone by calling one of these numbers: In the United States: In Canada: In other countries, contact your software account representatie to order Tioli publications. To locate the telephone number of your local representatie, perform the following steps: 1. Go to 2. Select your country from the list and click Go. 3. Click About this site in the main panel to see an information page that includes the telephone number of your local representatie. Accessibility Accessibility features help users with a physical disability, such as restricted mobility or limited ision, to use software products successfully. With this product, you can use assistie technologies to hear and naigate the interface. You can also use the keyboard instead of the mouse to operate all features of the graphical user interface. Tioli technical training Tioli user groups Support information For additional information, see "Accessibility features" in the IBM Security Access Manager for Enterprise Single Sign-On Planning and Deployment Guide. For Tioli technical training information, see the following IBM Tioli Education Web site at Tioli user groups are independent, user-run membership organizations that proide Tioli users with information to assist them in the implementation of Tioli Software solutions. Through these groups, members can share information and learn from the knowledge and experience of other Tioli users. Tioli user groups include the following members and groups: 23,000+ members 144+ groups Access the link for the Tioli Users Group at If you hae a problem with your IBM software, you want to resole it quickly. IBM proides the following ways for you to obtain the support you need: iii IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

11 Online Go to the IBM Software Support site at support/probsub.html and follow the instructions. IBM Support Assistant The IBM Support Assistant is a free local software sericeability workbench that helps you resole questions and problems with IBM software products. The IBM Support Assistant proides quick access to support-related information and sericeability tools for problem determination. To install the IBM Support Assistant software, go to Troubleshooting Guide For more information about resoling problems, see the IBM Security Access Manager for Enterprise Single Sign-On Troubleshooting and Support Guide. Conentions used in this publication This publication uses seeral conentions for special terms and actions, operating system-dependent commands and paths, and margin graphics. Typeface conentions This publication uses the following typeface conentions: Bold Italic Lowercase commands and mixed case commands that are otherwise difficult to distinguish from surrounding text Interface controls (check boxes, push buttons, radio buttons, spin buttons, fields, folders, icons, list boxes, items inside list boxes, multicolumn lists, containers, menu choices, menu names, tabs, property sheets) and labels (such as Tip: and Operating system considerations:) Keywords and parameters in text Citations (examples: titles of publications, diskettes, and CDs) Words defined in text (example: a nonswitched line is called a point-to-point line) Emphasis of words and letters (words as words example: "Use the word that to introduce a restrictie clause."; letters as letters example: "The LUN address must start with the letter L.") New terms in text (except in a definition list): a iew is a frame in a workspace that contains data. Variables and alues you must proide:... where myname represents... Monospace Examples and code examples File names, programming keywords, and other elements that are difficult to distinguish from surrounding text Message text and prompts addressed to the user Text that the user must type Values for arguments or command options About this publication ix

12 Operating system-dependent ariables and paths This publication uses the UNIX conention for specifying enironment ariables and for directory notation. When using the Windows command line, replace $ariable with % ariable% for enironment ariables and replace each forward slash (/) with a backslash (\) in directory paths. The names of enironment ariables are not always the same in the Windows and UNIX enironments. For example, %TEMP% in Windows enironments is equialent to $TMPDIR in UNIX enironments. Note: You can use the UNIX conentions if you are using the bash shell on a Windows system. x IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

13 Chapter 1. Oeriew IBM Security Access Manager for Enterprise Single Sign-On supports single sign-on and authentication serices for applications hosted on Citrix/Terminal Serers Citrix XenApp Serers or Microsoft Remote Desktop. You need to build a irtual channel connector for both the client and the serer to establish communication between the Client AccessAgent and Serer AccessAgent. Use Microsoft Visual Studio.NET 2008 with Serice Pack 1 to deelop Virtual Channel Connectors by using AccessAgent Terminal Serices SDK. You can deelop a connector dynamic link library (DLL) file either as a 32-bit or 64-bit application depending on the operating system where the DLL file is located. The following diagram illustrates the communication between the Terminal Serer and the client machine. Terminal Serer TSVCSerer ISAM ESSO Components Serer Virtual Channel Connector Implements TS SDK Serer SPI Virtual Channel Client Machine ISAM ESSO Components Client Virtual Channel Connector Terminal Serices Client Application Uses TS SDK Client API Client Virtual Channel Connector The client irtual channel connector DLL file must be registered with the client application through the registry editor. When the client application connects to a terminal serer, the client application loads and initializes the connector DLL file. The DLL file then communicates with its serer counterpart oer the irtual channel. See Chapter 3, Setting up Virtual Channel integration, on page 11. Serer Virtual Channel Connector The serer irtual connector DLL file must be registered with AccessAgent by updating the AccessAgent configuration. For more information, see Chapter 3, Setting up Virtual Channel integration, on page 11. Copyright IBM Corp. 2002,

14 At runtime, AccessAgent determines the aailable irtual channel framework for the current terminal session. AccessAgent also loads the appropriate connector DLL file for the irtual channel communication. 2 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

15 Chapter 2. Deeloping a Virtual Channel Connector Build prerequisites AccessAgent includes the required irtual channel connector DLL files for Microsoft Remote Desktop Serices. For other terminal serice products like Citrix, you hae to deelop the DLL files. See the following topics for information about deeloping the DLL files and setting up the irtual channel integration. Build prerequisites AccessAgent Terminal Serices SDK Setting up the cross-compiler on page 4 Setting up the build enironment on page 4 Linking dependencies on page 6 Virtual Channel Connector DLL on page 6 Loading the EncVcClient library on page 7 Before you deelop the Virtual Channel Connector, make sure that you comply with the requirements. The following are the prerequisites for deeloping a Virtual Channel Connector DLL file: Install the following software: Microsoft Visual Studio 2008.NET 2008 with Serice Pack 1 for 32-bit. AccessAgent Terminal Serices SDK. See AccessAgent Terminal Serices SDK for more information. Vendor Virtual Channel SDK. For example, Citrix. See Vendor SDKs on page 4 for more information. If you are building a 64-bit DLL file, install a 64-bit cross-compiler. See Setting up the cross-compiler on page 4 for more information. AccessAgent Terminal Serices SDK The AccessAgent Terminal Serices SDK is included in the AccessAgent installer. The Terminal Serices SDK is located in <aa_home>\tssdk\. For example, C:\Program Files\IBM\ISAM ESSO\AA\TSSDK\. AccessAgent Terminal Serices SDK contains the following files: File EncVcSerer.h EncVcClient.h Description Header file containing the AccessAgent Serer SPI Header file containing the AccessAgent Client SDK interface See Appendix A, Terminal Serices SDK API Details, on page 17 for the AccessAgent Terminal Serices SDK API details. Copyright IBM Corp. 2002,

16 Vendor SDKs Citrix Virtual Channel SDK includes seeral sample codes. For example, the "Ping" project located in VCSDK. The client sample codes can be found at \VCSDK\Win32\src\examples\c\client\dping\ while the serer sample codes can be found at \VCSDK\Win32\src\examples\c\serer\ctxping\. The required Citrix SDKs can be found at the following URLs: Citrix Virtual Channel SDK (VCSDK) 10.x WinFrame Application Programming Interface SDK (WFAPI SDK) Setting up the cross-compiler Set up the cross-compiler before you build the DLL file through Microsoft Visual Studio. Procedure 1. Close all the Microsoft Visual Studio 2008 Solutions. 2. Insert the Microsoft Visual Studio 2008 CD in your computer. 3. Select Repair. 4. Select 64-bit cross-compiler. 5. Continue with the installation. 6. Remoe the Microsoft Visual Studio 2008 CD after the installation. 7. Insert the Microsoft Visual Studio 2008 Serice Pack 1 CD. 8. Follow the steps to install Serice Pack 1. Setting up the build enironment Set up the build enironment for both 32-bit and 64-bit. Procedure 1. Create two separate Visual C++ Win32 DLL files for both the Client and Serer projects. In this procedure, the client project is called ICAVCClient, while the serer project is called ICAVCSerer. 2. For each project, set the Project Properties. a. Select ICAVCClient > Property Pages. b. Select Configuration Properties > General. c. In the Character Set field, select Use Unicode Character Set. d. Select C/C++ > Code Generation. e. In the Runtime Library field, select Multi-threaded DLL. f. Select ICAVCSerer > Property Pages. g. Select Configuration Properties > General. h. In the Character Set field, select Use Unicode Character Set. i. Select C/C++ > Code Generation. j. In the Runtime Library field, select Multi-threaded. 3. Define the ENCVCCLIENT_EXPORTS preprocessor macro in the client connector DLL project. The DLL exports the functions expected by AccessAgent. 4. Define the ENCVCSERVER_EXPORTS preprocessor macro in the serer connector DLL project. The DLL exports the functions expected by AccessAgent. 4 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

17 5. Add linking dependencies for each Virtual Channel Connection framework. For more information, see Linking dependencies on page Optional: Follow these steps for x64 connectors: a. Verify if the x64 cross-compiler installation is successful. b. Open the Project Properties for any project. c. Click the Platform list and check if the x64 operating system is in the list. d. Enable the configuration of the x64 operating system in the new project. After creating the project, the x64 operating system is not automatically added to the Platform list. There are two options to add x64 in the Platform list: Task Build the project in Win32 first if you hae a stable Win32 project. Modify the Win32 and the x64 operating systems in parallel. Steps 1. Open Project Properties. 2. Click Configuration Manager. 3. Select your project in the list. 4. In the Platform column, select Win32, and then click New. A pop-up window is displayed. 5. In the New Platform field, select x In the Copy Settings From field, select Win Clear the Create New Solutions Platform check box. 8. Click OK. All Win32 project settings are copied to the x64 configuration. 9. In the Project Properties, make the necessary changes such as Input Libraries. 1. Open Project Properties. 2. Click Configuration Manager. 3. Select your project in the list. 4. In the Platform column, select Win32, and then click New. A pop-up window is displayed. 5. In the New Platform field, select x In the Copy Settings From field, select Win Clear the Create new solutions platform check box. 8. Click OK. e. If you hae already configured the x64 operating system, wheneer you change Project Properties, synchronize these configurations: Solution configuration Debug Release Debug Release Solution platform Win32 Win32 x64 x64 Chapter 2. Deeloping a Virtual Channel Connector 5

18 Linking dependencies Add a linking dependency on the library files to build the Client and Serer connector DLL files. When you build a client connector DLL, add a linking dependency for these library files: For Citrix XenApp plug-in (for 32-bit DLL): clibdll.lib (from Citrix Virtual Channel SDK ) dapi.lib (from Citrix Virtual Channel SDK ) For Citrix XenApp Serer: wfapi.lib from WinFrame Application Programming Interface (for 32-bit DLL) wfapi64.lib from WinFrame Application Programming Interface (for 64-bit DLL) The required Citrix SDKs can be found at the following URLs: Citrix Virtual Channel SDK (VCSDK) 10.x WinFrame Application Programming Interface SDK (WFAPI SDK) Virtual Channel Connector DLL This topic coers information about the Virtual Channel Client and Serer connector DLL. Virtual Channel Client Connector DLL The Virtual Channel client framework has four abstract functions that: Opens the Virtual Channel drier Writes data to Virtual Channel Reads data from Virtual Channel Closes the Virtual Channel drier The following pseudo codes proide an example on where to call the AccessAgent client SDK API. DrierOpen() //This function is called when drier is loaded { Call LoadEncVcClientLib to load EncVcClient library; Perform required initialization based on the specific VC framework for DLL drier; Call InitEncVcClient to initialize EncVcClient; } See Loading the EncVcClient library on page 7 for instructions on how to load the EncVcClient library. DrierPoll() //This function is called wheneer drier wants to send a message on the Virtual Channel using endor API 6 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

19 { Call GetNextClientMessage to retriee next message to send; Send message ia VC; Call ReleaseMessage to wipe message from memory; } DataArrial() //This function is called when drier receies a message on the Virtual Channel { Call ProcessSererMessage to process receied message; } DrierClose() //This function is called when the drier is unloaded { Call FinalizeEncVcClient to de-initialize EncVcClient; Call FreeEncVcClientLib to free EncVcClient library; } See Appendix A, Terminal Serices SDK API Details, on page 17 for additional information about the APIs. Virtual Channel Serer Connector DLL Implement all the wrapper functions within EncVcSerer.h. EncVcSerer.h is located in C:\Program Files\IBM\ISAM ESSO\TSSDK\. Call the corresponding Virtual Channel framework serer API. Loading the EncVcClient library The EncVcClient library is not proided in the AccessAgent Terminal Serices SDK. The EncVcClient library is dynamically loaded within the client connector DLL file. Perform a dynamic loading of EncVcClient library for AccessAgent. Procedure 1. Include the following codes in the client connector: //Typedef of EncVcClient function pointers typedef E_VCCError (*PF_InitEncVcClient)(); typedef E_VCCError (*PF_FinalizeEncVcClient)(); typedef E_VCCError (*PF_ProcessSererMessage)(const STsMessage* pmessage); typedef E_VCCError (*PF_GetNextClientMessage)(STsMessage* pmessage); typedef E_VCCError (*PF_ReleaseMessage)(STsMessage* pmessage); typedef oid (*PF_AADebugLog)(E_VCCLogLeel logleel, const wchar_t* wszmethodname, const wchar_t* wszmessage); HINSTANCE g_hencvclib = NULL; PF_InitEncVcClient g_pfinitencvcclient = NULL; PF_FinalizeEncVcClient g_pffinalizeencvcclient = NULL; PF_ProcessSererMessage g_pfprocessserermessage = NULL; PF_GetNextClientMessage g_pfgetnextclientmessage = NULL; PF_ReleaseMessage g_pfreleasemessage = NULL; PF_AADebugLog g_pfaadebuglog = NULL; /** * \brief Initalization of dynamic run time loading for EncVcClient.dll */ BOOL LoadEncVcClientLib() { Chapter 2. Deeloping a Virtual Channel Connector 7

20 g_hencvclib = LoadLibrary(L"EncVcClient.dll"); if(g_hencvclib == NULL) { ::OutputDebugString(L"::LoadEncVcClientLib, Error: EncVcClient.dll NOT found"); return FALSE; } g_pfinitencvcclient = (PF_InitEncVcClient)::GetProcAddress (g_hencvclib, "InitEncVcClient"); g_pffinalizeencvcclient = (PF_FinalizeEncVcClient)::GetProcAddress (g_hencvclib, "FinalizeEncVcClient"); g_pfprocessserermessage = (PF_ProcessSererMessage)::GetProcAddress (g_hencvclib, "ProcessSererMessage"); g_pfgetnextclientmessage = (PF_GetNextClientMessage)::GetProcAddress (g_hencvclib, "GetNextClientMessage"); g_pfreleasemessage = (PF_ReleaseMessage)::GetProcAddress (g_hencvclib, "ReleaseMessage"); g_pfaadebuglog = (PF_AADebugLog)::GetProcAddress (g_hencvclib, "AADebugLog"); if(!g_pfinitencvcclient!g_pffinalizeencvcclient!g_pfprocessserermessage!g_pfgetnextclientmessage!g_pfreleasemessage!g_pfaadebuglog) { ::OutputDebugString(L"::LoadEncVcClientLib, Error: Required APIs NOT found in EncVcClient.dll"); FreeLibrary(g_hEncVcLib); g_hencvclib = NULL; return FALSE; } g_pfaadebuglog(e_vcc_log_moreinfo, L"::LoadEncVcClientLib", L"Load library successful"); return TRUE; } /** * \brief Free EncVcClient library */ BOOL FreeEncVcClientLib() { BOOL bret = FreeLibrary(g_hEncVcLib); g_pfinitencvcclient = NULL; g_pffinalizeencvcclient = NULL; g_pfprocessserermessage = NULL; g_pfgetnextclientmessage = NULL; g_pfreleasemessage = NULL; g_pfaadebuglog = NULL; g_hencvclib = NULL; return bret; }/** * \brief Below are API functions for ESSO TS SDK */ E_VCCError InitEncVcClient() { return g_pfinitencvcclient(); } E_VCCError FinalizeEncVcClient() { return g_pffinalizeencvcclient(); } E_VCCError ProcessSererMessage(const STsMessage* pmessage) { return g_pfprocessserermessage(pmessage); 8 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

21 } E_VCCError GetNextClientMessage(STsMessage* pmessage) { return g_pfgetnextclientmessage(pmessage); } E_VCCError ReleaseMessage(STsMessage* pmessage) { return g_pfreleasemessage(pmessage); } oid AADebugLog(E_VCCLogLeel logleel, const wchar_t* wszmethodname, const wchar_t* wszmessage) { g_pfaadebuglog(logleel, wszmethodname, wszmessage); } 2. Call the LoadEncVcClientLib() function only once during the opening of the drier before calling of any of the functions listed 3. Call the FreeEncVcClientLib() function only once during the closing of the drier as the last function call before the drier closes 4. Sae and recompile the connector DLL file. Note: Check the leel 3 log. If the loading of library is successful, there is an entry that state LoadEncVcClientLib, Load library successful in the AccessAgent log file. See "Log policies" in the IBM Security Access Manager for Enterprise Single Sign-On Policies Definition Guide. Chapter 2. Deeloping a Virtual Channel Connector 9

22 10 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

23 Chapter 3. Setting up Virtual Channel integration You can integrate Client AccessAgent with the Citrix XenApp Plug-in and the Serer AccessAgent with terminal serices. See the following topics for more information. Integrating Client AccessAgent with Citrix Integrating the Serer AccessAgent with Citrix Serer on page 12 Note: AccessAgent already includes the required irtual channel connector DLL files for Microsoft Remote Desktop Serices. Integrating Client AccessAgent with Citrix The Client Connector DLL file has 32-bit Citrix clients, which means it is built in 32-bit only. This procedure is required when you install or upgrade the client AccessAgent ersion. Procedure 1. Copy the client connector DLL file. For example ICAVCClient.dll, to <Citrix XenApp Plug-in installation folder>. 2. Copy EncVcClient.dll into the <Citrix XenApp Plug-in installation folder>. 3. Configure the Citrix client registry for the client connector DLL file. a. On your Windows desktop, click Start > Run. b. In the Open field, enter regedit and click OK. The Registry Editor is displayed. Note: For a Windows 32-bit operating system, the Citrix registry hie is HKEY_LOCAL_MACHINE\SOFTWARE\Citrix. For a Windows x64 operating system, the Citrix registry hie is HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\. c. Modify VirtualDrierEx. 1) Select <Citrix registry hie>\ica Client\Engine\Configuration\ Adanced\Modules\ICA 3.0\. 2) Double-click VirtualDrierEx. 3) In the Value data field, add ICAVCClient. 4) Click OK. d. Create an ICAVCClient. 1) Open <Citrix registry hie>\ica Client\Engine\Configuration\ Adanced\Modules\. 2) Right-click and select New > Key. A new folder is created. 3) Rename the folder. The new name is ICAVCClient. 4) Repeat these steps until you create all the alues in the table. a) In the <Citrix registry hie>\ica Client\Engine\Configuration\ Adanced\Modules\ICAVCClient\ folder, right-click and select New > String Value. b) Specify the following alue names and alue data. Copyright IBM Corp. 2002,

24 Value Name DrierName DrierNameWin16 DrierNameWin32 Value Data ICAVCClient.dll ICAVCClient.dll ICAVCClient.dll Integrating the Serer AccessAgent with Citrix Serer If the Serer AccessAgent is 32-bit, a 32-bit connector DLL file is needed. If the Serer AccessAgent is 64-bit, a 64-bit connector DLL file is needed. Procedure 1. Copy the serer connector file ICAVCSerer.dll into the C:\Program Files\ISAM ESSO folder. 2. Register the serer connector file with AccessAgent. a. On your Windows desktop, click Start > Run. b. In the Open field, enter regedit and click OK. The Registry Editor is displayed. c. Open HKEY_LOCAL_MACHINE\SOFTWARE\ISAM ESSO. d. Right-click and select New > Key. A new folder is created. e. Rename the folder; the new name is AccessAgent. f. Click the AccessAgent folder. g. Right-click and select New > Key. A new folder is created. h. Rename the folder; the new name is Integration. i. Click the Integration folder. j. Right-click and select New > Key. A new folder is created. k. Rename the folder; the new name is TerminalSerices. l. Click the TerminalSerices folder. m. Right-click and select New > Key. A new folder is created. n. Rename the folder; the new name is SererSPI. You can continue to create new keys until you hae created your folder structure. o. In the HKLM/SOFTWARE/IBM/ISAM ESSO/AccessAgent/Integration/ TerminalSerices/SererSPI folder, right-click and select New > String Value. p. Set ICA as the Value name. q. In the Value data field, enter the serer connector file name. For example, ICAVCSerer.dll. r. Click OK. 12 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

25 Chapter 4. Verifying the operation After building and compiling the required DLL file for Citrix XenApp, erify if the DLL file is working properly. Citrix XenApp 1. Set the AccessAgent log leel to Start an application by using the Citrix Online plug-in. 3. Connect to the terminal serer. 4. In the AccessAgent log on the client machine, erify that the log entries generated by EncVcClient.dll exists. 5. In the AccessAgent log on the serer machine, erify that the following entries generated by tscserer.exe exists: [GetTsSpiDllForSession], TsSpi dll [ICAVCSerer.dll] found for Session [ICA], [CTSSerer::Initialize()], Polling Thread started,[ctsserer::initialize()], Polling Thread started Copyright IBM Corp. 2002,

26 14 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

27 Chapter 5. Troubleshooting and Support Troubleshooting is necessary to determine the cause and required solution for the error encountered. If the irtual channel integration fails, contact IBM support and proide them the required information. You must proide the following files for reference: Client AccessAgent log files Serer AccessAgent log files Client and Serer Visual Studio projects Follow this procedure to ensure that the log data are collected correctly: 1. Close unrelated applications. 2. Delete all existing logs in the log folder. 3. Set the log leel to Proceed to step through the procedures for collection of log data. 5. Set the log leel to 1 after the log collection. Copyright IBM Corp. 2002,

28 16 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

29 Appendix A. Terminal Serices SDK API Details Know the different client SDK API and Serer SDK SPI constants, structures, and functions. See this topic when deeloping the code for the Virtual Channel Connector DLL file. Client SDK API The following are the client SDK functions and their corresponding details. Constants Error Types: E_VCCError enum Error Code E_VCC_SUCCESS E_VCC_FAIL E_VCC_INVALID_ARG E_VCC_NOT_INITIALIZED E_VCC_NO_DATA Definition The function is successful. An unexpected error occurred. One or more arguments are inalid. InitializeVcClient() has not been called successfully. There is no more data aailable to be sent to the serer. Log Leels: E_VCCLogLeel enum E_VCC_LOG_SEVERE E_VCC_LOG_BASEINFO E_VCC_LOG_MOREINFO E_VCC_LOG_DEBUG Structures typedef struct { unsigned char* m_pdata; unsigned long m_ulen; }STsMessage; //Structure to hold binary messages handled by Client SDK Functions The Client SDK proides the following set of functions that the irtual channel drier can call to communicate with the local AccessAgent. AADebugLog Definition Description Definition oid AADebugLog (E_VCCLogLeel logleel, const wchar_t* wszmethodname, const wchar_t* wszmessage); Produces a debug log line to AccessAgent log files. Copyright IBM Corp. 2002,

30 AADebugLog Parameters logleel [in] Log leel Return Values wszmethodname [in] NULL-terminated name of the module/class/method name generating the log statement. wszmessage [in] NULL-terminated descriptie text. None InitEncVcClient Definition Description Parameters E_VCCError InitEncVcClient(); Initializes the EncVcClient modules loaded in the irtual drier. This function must be called when the terminal client establishes a connection with the serer and creates a irtual channel. The function internally sets up a named pipe serer so that it can receie messages from the local AccessAgent. None Return Values E_VCC_SUCCESS E_VCC_FAIL FinalizeEncVcClient Definition Description Parameters Return Values E_VCCError FinalizeEncVcClient(); Shuts down the EncVcClient modules loaded in the irtual drier. This function must be called when the terminal session is disconnected and the irtual channel is closed. None E_VCC_SUCCESS ProcessSererMessage Definition Description Parameters E_VCCError ProcessSererMessage (const STsMessage* pmessage); Processes the data packet receied from the irtual channel serer. When a data packet is receied from the serer, the irtual channel connector calls this function to pass the message to the local AccessAgent for processing. pmessage [in] Data packet receied from the serer AccessAgent Return Values E_VCC_SUCCESS E_VCC_NOT_INITIALIZED E_VCC_INVALID_ARG 18 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

31 GetNextClientMessage Definition Description Parameters E_VCCError GetNextClientMessage (STsMessage* pmessage); Returns a data packet, if any, from the local AccessAgent to the serer. The function looks up the queue in which the packets from the local AccessAgent are put in. The irtual drier calls this function periodically to check if there are any packets to be sent to the serer. The irtual drier calls this function within a period less than one second. pmessage [out] Data packet from the client AccessAgent to be sent to the serer AccessAgent Return Values E_VCC_SUCCESS E_VCC_NOT_INITIALIZED E_VCC_INVALID_ARG E_VCC_NO_DATA ReleaseMessage Definition Description Parameters E_VCCError ReleaseMessage (STsMessage* pmessage); Wipes the memory buffer allocated to the data packet and frees the memory. A client message is obtained from the GetNextClientMessage function. After sending the client message to the irtual channel serer, the irtual drier passes the message to this function to release the resources. pmessage [out] Message whose memory buffer is to be released. Return Values E_VCC_SUCCESS E_VCC_INVALID_ARG Serer SDK SPI The following are the Terminal Serices Serer SPI specifications. These specifications are essentially a wrapper around serer-side functions of Terminal Serices API for Windows and WinFrame API for Citrix. The following functions must be implemented by the connector DLL file. Constants Error Types: E_VCCError enum Error Code E_VCC_SUCCESS E_VCC_FAIL E_VCC_INVALID_ARG Definition The function is successful. An unexpected error occurred. One or more arguments are inalid. Appendix A. Terminal Serices SDK API Details 19

32 Query Types: EN_INFO_CLASS enum EnInitialProgram EnApplicationName EnWorkingDirectory EnOEMId EnSessionId EnUserName EnWinStationName EnDomainName EnConnectState EnClientBuildNumber EnClientName EnClientDirectory EnClientProductId EnClientHardwareId EnClientAddress EnClientDisplay EnClientProtocolType EN_TS_EVENT - Eent flags for EnWaitSystemEent() Values identical to WTS_EVENT_xxx and WF_EVENT_xxx Eent EN_TS_EVENT_NONE EN_TS_EVENT_CREATE EN_TS_EVENT_DELETE EN_TS_EVENT_RENAME EN_TS_EVENT_CONNECT EN_TS_EVENT_DISCONNECT EN_TS_EVENT_LOGON EN_TS_EVENT_LOGOFF EN_TS_EVENT_STATECHANGE EN_TS_EVENT_LICENSE EN_TS_EVENT_ALL EN_TS_EVENT_FLUSH Description // return no eent // new WinStation created // existing WinStation deleted // existing WinStation renamed // WinStation connect to client // WinStation logged on without client // user logged on to existing WinStation // user logged off from existing WinStation // WinStation state change // license state change // wait for all eent types // unblock all waiters Structures typedef struct _EN_CLIENT_ADDRESS { //Returned by QueryCurrentSessionInfo(EnClientAddress) DWORD AddressFamily; // AF_INET, AF_IPX, AF_NETBIOS, AF_UNSPEC BYTE Address[20]; // client network address } EN_CLIENT_ADDRESS, * PEN_CLIENT_ADDRESS; Functions 20 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

33 EnQueryCurrentSessionInformation Definition Description E_VCSError EnQueryCurrentSessionInfo (EN_INFO_CLASS infoclass, LPWSTR * ppbuffer, DWORD * pbytesreturned); Queries information about the current terminal session. The parameters that can be queried are: EnSessionId (SessionId) EnUserName (UserName) EnDomainName (DomainName) EnConnectState (ConnectionState) EnClientName (Client machine name) EnClientAddress (Client IP address) Parameters infoclass [in] - Specifies the type of the information to retriee. Return Values E_VCS_SUCCESS ppbuffer [out] - Pointer to a ariable that receies a pointer to the requested information. pbytesreturned [out] - Pointer to a ariable that receies the size, in bytes, of the data returned ppbuffer. E_VCS_FAIL EnVirtualChannelOpen Definition Description Parameters HANDLE EnVirtualChannelOpen(); Opens a irtual channel with a gien name. None Return Values A alid irtual channel HANDLE on success NULL on failure EnVirtualChannelClose Definition Description Parameters E_VCSError EnVirtualChannelClose (IN HANDLE hchannelhandle); Closes the irtual channel opened with a preious call to EnVirtualChannelOpen(). hchannelhandle [in] - Handle obtained from EnVirtualChannelOpen call Return Values E_VCS_SUCCESS E_VCS_FAIL Appendix A. Terminal Serices SDK API Details 21

34 EnVirtualChannelRead Definition Description E_VCSError EnVirtualChannelRead (IN HANDLE hchannelhandle, IN ULONG utimeout, OUT PCHAR pbuffer, IN ULONG ubuffersize, OUT PULONG pubytesread); Reads data from a gien irtual channel. Parameters hchannelhandle [in] - Handle to a irtual channel opened by the nvirtualchannelopen() function. Return Values E_VCS_SUCCESS utimeout [in] - Specifies the timeout, in milliseconds. pbuffer [out] - Pointer to a buffer that receies a chunk of data read from the serer end of the irtual channel. UbufferSize [in]- Specifies the size, in bytes, of pbuffer pubytesread [out] - Pointer to a ariable that receies the number of bytes read. E_VCS_FAIL EnVirtualChannelWrite Definition Description E_VCSError EnVirtualChannelWrite (IN HANDLE hchannelhandle, IN PCHAR Buffer, IN ULONG Length, OUT PULONG pbyteswritten); Writes data to a gien irtual channel. Parameters hchannelhandle [in] - Handle to a irtual channel opened by the EnVirtualChannelOpen() function. Return Values E_VCS_SUCCESS Buffer [in] - Pointer to a buffer containing the data to write to the irtual channel. Length [in] - Specifies the size, in bytes, of the data to write. pbyteswritten [out] - Pointer to a ariable that receies the number of bytes written. E_VCS_FAIL EnDisconnectCurrentSession Definition Description Parameters E_VCSError EnDisconnectCurrentSession (IN BOOL bwait); Disconnects the current Windows session. bwait [in] Indicates whether the operation is synchronous. Specify TRUE to wait for the completion of the operation, or FALSE to return immediately. 22 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

35 EnDisconnectCurrentSession Return Values E_VCS_SUCCESS E_VCS_FAIL EnLogoffCurrentSession Definition Description Parameters E_VCSError EnLogoffCurrentSession (BOOL bwait); Logs off the current Windows session. bwait [in] Indicates whether the operation is synchronous. Specify TRUE to wait for the completion of the operation, or FALSE to return immediately. Return Values E_VCS_SUCCESS E_VCS_FAIL EnWaitSystemEent Definition Description E_VCSError EnWaitSystemEent (IN DWORD EentMask,OUT DWORD * peentflags); Blocks on a system eent before returning. The system eent can be any one of the following: EN_TS_EVENT_CREATE (Session creation) EN_TS_EVENT_DELETE (Session deletion) EN_TS_EVENT_CONNECT (Session connect) EN_TS_EVENT_DISCONNECT (Session disconnect) EN_TS_EVENT_LOGON (User log on) EN_TS_EVENT_LOGOFF (User log off) Parameters EentMask [in] Bitwise OR of the eents to wait for peentflags [out] Eent that fired Return Values E_VCS_SUCCESS E_VCS_FAIL EnFreeMemory Definition Description Parameters Return Values VOID EnFreeMemory(IN PVOID pmemory); Frees memory allocated by other SPI functions. pmemory [in] Pointer to the memory to be freed None Appendix A. Terminal Serices SDK API Details 23

36 24 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

37 Appendix B. AccessAgent Terminal Serices SDK message exchange Messages are exchanged between the Client AccessAgent and the Serer AccessAgent during the initialization of the serer-side irtual channel application. The following diagram shows a typical message exchange scenario between the Client AccessAgent and the Serer AccessAgent. The following are the messages that the Client AccessAgent and the Serer AccessAgent can send. Table 1. Serer to Client Messages Message Type Session Query Credential Request Full Synchronize Request Description Queries the client AccessAgent session status. Requests the user AccessAgent credential. Request client AccessAgent to perform full synchronization with the IMS Serer. Table 2. Client to Serer Messages Message Type Session Ready Session Not Ready Credential Data Start Session Description Response to Session Query message. Indicates that the client session is logged on. Response to Session Query message. Indicates that the client session is not logged on. Response to Credential Request which contains the user AccessAgent credential. Requests the Serer AccessAgent to start the user session. Copyright IBM Corp. 2002,

38 Table 2. Client to Serer Messages (continued) Message Type Description Logoff Session Requests the Serer AccessAgent to log off from the user session. Full Synchronize Request Requests the Serer AccessAgent to perform full synchronize with the IMS Serer. Enable Single Sign-on Requests the Serer AccessAgent to enable single sign-on for the user session. Disable Single Sign-on Requests the Serer AccessAgent to disable single sign-on for the user session. Lock Desktop Requests the Serer AccessAgent to lock the user desktop. Unlock Desktop Requests the Serer AccessAgent to unlock the user desktop. 26 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK

39 Notices This information was deeloped for products and serices offered in the U.S.A. IBM may not offer the products, serices, or features discussed in this document in other countries. Consult your local IBM representatie for information on the products and serices currently aailable in your area. Any reference to an IBM product, program, or serice is not intended to state or imply that only that IBM product, program, or serice may be used. Any functionally equialent product, program, or serice that does not infringe any IBM intellectual property right may be used instead. Howeer, it is the user's responsibility to ealuate and erify the operation of any non-ibm product, program, or serice. IBM may hae patents or pending patent applications coering subject matter described in this document. The furnishing of this document does not gie you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drie Armonk, NY U.S.A. For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan, Ltd , Shimotsuruma, Yamato-shi Kanagawa Japan The following paragraph does not apply to the United Kingdom or any other country where such proisions are inconsistent with local law : INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement might not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are proided for conenience only and do not in any manner sere as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. Copyright IBM Corp. 2002,

40 IBM may use or distribute any of the information you supply in any way it beliees appropriate without incurring any obligation to you. Licensees of this program who wish to hae information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation 2Z4A/ Burnet Road Austin, TX U.S.A. Such information may be aailable, subject to appropriate terms and conditions, including in some cases payment of a fee. The licensed program described in this document and all licensed material aailable for it are proided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equialent agreement between us. Any performance data contained herein was determined in a controlled enironment. Therefore, the results obtained in other operating enironments may ary significantly. Some measurements may hae been made on deelopment-leel systems and there is no guarantee that these measurements will be the same on generally aailable systems. Furthermore, some measurement may hae been estimated through extrapolation. Actual results may ary. Users of this document should erify the applicable data for their specific enironment. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly aailable sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objecties only. All IBM prices shown are IBM's suggested retail prices, are current and are subject to change without notice. Dealer prices may ary. This information is for planning purposes only. The information herein is subject to change before the products described become aailable. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of indiiduals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on arious operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to 28 IBM Security Access Manager for Enterprise Single Sign-On: Lightweight AccessAgent mode on Terminal Serer SDK