NCSA Security R&D Last Updated: 11/03/2005

Stage Design Project: Community Accounts Stage #1: Accounts NCSA Security R&D Last Updated: 11/03/2005 Summary The Accounts stage of the Community Accounts project consists of creating all tools necessary for implementation of a community account on a UNIX-like system. These utilities will be highly configurable to facilitate a wide variety of possible implementation styles. This stage of the project does not encompass concerns such as long-term storage and job auditing. Please see the Community Accounts Stages document for an overview of all stages involved in this project. Deliverables 1. A suite of chroot environment management utilities 2. A community shell application 3. A community environment management utility 4. A combined package in source and binary formats 5. A GRAM patch to execute jobs through the user's shell People Von Welch <vwelch@ncsa.uiuc.edu> Kevin Price <kjprice@ncsa.uiuc.edu> Project Manager Designer; Programmer Deliverable #1: A suite of chroot environment management utilities. The chroot_jail suite of utilities can create, manipulate, and remove chroot jail environments. Also included will be an init.d-style script that can properly mount filesystems into jail environments (including special filesystems, such as proc, devfs, and devptsfs), as well as a daemon that can repeat a jail's /dev/log into the root /dev/log for proper syslog logging. See attachment #1: Detailed list of jail utilities and parameters. Jail creation and management utilities are necessary because creation of a properly configured jail environment is a non-trivial task. For example, shared libraries must be included correctly, the /etc of the jailed environment must be configured correctly, and certain shared libraries that are not specifically dependencies of any binaries must be included for some general UNIX functions to work correctly. Additionally, certain Project Deliverables 1 of 18

conditions (such as setuid root binaries) must be avoided in order to keep the jailed environment secure. See attachment #2: Required tasks and security concerns for jail creation. These utilities use a database to track dependencies so that files can be easily added and removed from the jail. The means of storing this database will be configurable. Commonly used SQL-driven database environments (such as MySQL, Postgres, and Oracle), as well as for DBM-style file-based databases will be supported. See attachment #3: Database structure for jail dependency management. Configuration files for all jails (as well as any global options) reside in /etc/jails by default, but a different location can be specified at compile-time. It should be noted that utilities already exist to generate jail environments, such as jailkit and vserver, however these applications do not allow for the wide variety of configuration and management options, nor are they as widely supported among UNX systems. The chroot_jail suite will function on any UNIX system with perl 5.8 installed. Deliverable #2: A community shell application. The commsh general-purpose community shell includes the abilities to perform a change root (chroot) operation upon startup; create a dynamic account (by setting the user's uid to a dynamically allocated uid); directly launch a command from a configurable list of valid commands; and launch a more conventional (or restricted) shell. Each of these options is fully configurable. All logging takes place using the syslog logging mechanism. See attachment #4: commsh flow diagram. Configuration takes place by parsing /etc/commsh.conf (although a different default configuration location can be specified at compile-time) for a set of configuration directives. Two of these directives allow you to load additional configuration files (optionally based on the community user running the shell.) If a directive that takes only one value appears twice, the latter value will overwrite any prior values. See attachment #5: commsh configuration directives. Of particular note are the dynamic account options. If enabled, commsh has the ability to manage a range of valid uid values that it will assign to sessions dynamically. Each dynamic account instantiation can create a copy of a template home directory, and will expire and be automatically removed after a configurable length of time has passed. Also, if configured a specific uid can be selected at login time, allowing a user to re-use a dynamic account for multiple sessions. Expired dynamic accounts can either be reaped on every run of commsh (if configured), or they can be reaped by running commsh --reap at any time (such as from a cron job). Certain options, such as changed root environments and dynamic accounts, are only available if the commsh binary is setuid root. If the binary is not setuid root, Project Deliverables 2 of 18

configuration directives related to these options will be ignored. The setuid root bit is dropped immediately after selecting a dynamic uid and changing to a new root directory. Deliverable #3: A community environment management utility A simple shell-script utility facilitates generation of community shell environments. This script will generate configuration files, use the chroot_jail suite of utilities to create jailed environments (if desired), and make appropriate additions to /etc/passwd and /etc/group. This utility can generate multiple community accounts, and can be used later to manage these accounts. Deliverable #4: A combined package in source and binary formats The above utilities will be packaged and available as a combined package. This package will be available both in source format as a tarball, and in binary format in a variety of UNIX package styles. A combination of configure scripts and Makefiles will allow the package to be generated with different features enabled and disabled. Deliverable #5: A GRAM patch to execute jobs through the user's shell. It is unclear at this point exactly how this patch will need to be constructed. The most likely candidate appears to be modifications to fork_starter.c coupled with a new configuration parameter for globus-fork.conf. This parameter should be of the form "use_user_shell=true", and if set it should force fork_starter to execute commands through the user's defined shell. Assuming that all job managers do (or can be made to) use fork_starter when they launch jobs, this should function as desired. Further research into the GRAM execution flow and how it interacts with existing and potential new job schedulers is required. Notes This document, and all attachments included, should be considered as works in progress. They are neither definitive nor complete, and may change with final implementation. Some of the functionality implemented by these deliverables is already partially available in existing software. Examples of such include jailkit (http://olivier.sessink.nl/jailkit/), Linux- VServer (http://linux-vserver.org/) and chrsh (http://www.adg.us/computers/chrsh.html). However, these utilities do not meet all requirements of this stage of the Community Accounts project. Where possible, we will utilize already existent code and ideas (with proper permission and citation) in the development of these utilities. Project Deliverables 3 of 18

Attachment #1: Detailed list of jail utilities: jail_create [-nedpl] [-fc FLATFILE] PATH NAME Create a fresh jail at the specified PATH, and give it the specified NAME. A database for this jail will be initialized. -n Add login library files (typically libnss) -e Add /etc login files (nsswitch.conf, passwd, shadow, group) -d Add common device (/dev/tty, /dev/null, et cetera) -l Use hard links if possible -p Preserve original owner and permissions if possible -f Create the jail using FLATFILE, which should be the same format saved by jail_archive or jail_destroy. If used in conjunction with -n or -d, the login libraries and/or devices take precedence over FLATFILE -c Check for any addition dependencies for files added with -f. Without this option, it is assumed that the dependency information contained by FLATFILE is completely correct jail_add [-qlrp] NAME/PATH FILE/PATH [FILE/PATH...] jail_add [-qlrp] NAME/PATH --manifest MANIFEST jail_add [-qlp] NAME/PATH FILE/PATH --depend FILE... Add one or more files to a given jail, specified either by NAME or by PATH. -q Do not add dependencies -l Use hard links if possible -p Preserve original owner and permissions if possible -r If any of the arguments are a path, add them recursively --manifest Specifies a manifest file that contains a list of files and paths to add. --depend Force dependencies on a file. This can also be used to add a dependency to a file that already exists within the jail. These dependencies will be added even if -q is specified. Attachment #1: Detailed list of jail utilities 5 of 18

jail_rm [-qr] NAME/PATH FILE/PATH [FILE/PATH...] jail_rm [-qr] NAME/PATH --manifest MANIFEST Works as jail_add, but instead removes files from the given jail. By default, dependencies that are no longer necessary are also removed from the jail. -q Do not remove any dependencies -r Recursively remove subdirectories of non-empty directories --manifest Specifies a manifest file that contains a list of files and paths to remove. jail_destroy [-k] [-d FLATFILE] [-tzb TARBALL] NAME/PATH Completely remove a jail from the system. By default, all files will be deleted, the database tables will be dropped, and all metadata will be removed. -k Do not delete physical files -d Save a copy of the database in FLATFILE -t Archive all files to TARBALL before deleting -z Compress the tarball using gzip if available -b Compress the tarball using bzip2 if available, defaults to -z if not jail_archive [-zb] NAME/PATH FLATFILE TARBALL Archive a jail. The dependency database will be written to FLATFILE, and the physical files will be tarred up and saved as TARBALL. -z Compress the tarball using gzip if available -b Compress the tarball using bzip2 if available, defaults to -z if not jail_restore NAME/PATH FLATFILE TARBALL Restore an archived database. If the database exists, it will be emptied and then restored from the saved copy. If the database does not exist, it will be initialized and then restored. FLATFILE and TARBALL should both be in the same format created by jail_archive or jail_destroy. Compression of the tarball is automatically detected. Attachment #1: Detailed list of jail utilities 6 of 18

jail_mount [-u] [NAME/PATH [NAME/PATH...]] Mount all mount points listed in the /etc/mtab files of the specified jails. If no jails are listed, this action will be performed on all jails. This is especially useful in mounting special dev and proc file systems. -u Unmount all mount points instead jail_syslogd [-l LOG] [-r RLOG] [NAME/PATH [NAME/PATH...]] jail_syslogd [-h -k] Launch a daemon that listens to /dev/log on the specified jails (or all jails, if no list is given) and repeats what it hears to the root /dev/log. It should be noted that most syslogd daemons could be configured directly to listen to all of these /dev/log files simultaneously, so this utility is not strictly needed in all environments. -l Listen to LOG inside of the specified jails -r Repeat to RLOG in the root file system -h Send all jail_syslogd processes the HUP (restart) signal -k Send all jail_syslogd processes the TERM (terminate) signal Attachment #1: Detailed list of jail utilities 7 of 18

Attachment #2: Required tasks and security concerns for jail creation: Required tasks: A new directory structure must be created, including (at a minimum) /bin, /lib, /etc and other such directories. Certain /etc files must be created or copied into the jail for authentication to be successful. Exactly which files depend on the platform, but typically nsswitch.conf, passwd, group and shadow are minimally required. Other possible files include host and mtab. Certain shared libraries must be copied into the jail for authentication to be successful. Exactly which files depend on the platform, but typically libnss is minimally required. All desired applications must be copied into the directory structure. Dependency information must be determined for these applications, and all dependencies must also be copied o Types of dependencies include: Shared libraries (determined by ld.so or equivalent) Interpreters/shells (determined by #! at start of application) If the jail includes any mounts (such as special dev or proc file systems), an init script must be added to the startup scripts that mounts these devices/shares. Either the system's syslog daemon must be configured to read from the jail's /dev/log, or a log repeater daemon must be configured and set to run at system boot Security concerns: There must be no setuid root binaries within the jailed directory. Libraries and binaries (especially libc) should be owned by root with permissions set so that no other user can modify them. There should be no devices allowing a user within the jailed environment direct access to memory or any hard drive, especially the root partition. Compilers should not be included within the jailed directory unless absolutely necessary. Likewise, powerful scripting languages (such as perl) should be avoided unless necessary. Attachment #2: Required tasks and security concerns for jail creation 9 of 18

Attachment #3: Database structure for jail dependency management: // An entry in this table represents a file included in the // jail. Also recorded is whether or not the file was // included explicitly or is only a dependency for another // file. (This information is used to intelligently remove // dependencies as appropriate.) TABLE files { integer id primary key, } varchar(1024) file_name, bool explicit // root-system path to file // included explicitly or as // a dependency? // Each entry in this table represents the dependency of // one file upon another. If nothing depends on a file that // was not included explicitly, then it should be removed // from the jail. TABLE dependencies { integer id primary key, } integer file_id references files.id, integer depends_on reference files.id // depender // dependee Attachment #3: Database structure for jail dependency management 11 of 18

Attachment #4: commsh flow diagram: Attachment #4: commsh flow diagram 13 of 18

Attachment #5: commssh configuration directives: Configuration will be read from /etc/commsh.conf by default, but this can be changed at compile time. Lines in the configuration file will be of the form: Directive PARAMETER PARAMETER... Blank lines and comments (lines beginning with a hash (#)) will be ignored. For each path, file or shell parameter, a tilde (~) will expand to the home path of the user as defined in /etc/passwd (or other authoritative user information repository). The following configuration directives will be supported: General ReadConfig FILE Reads the configuration file found at FILE. Note that tildes in FILE are expanded. ReadUserConfig UID/USERNAME FILE Read the configuration file found at FILE, but only if the real uid or user name matches UID/USERNAME. As above, tildes in FILE are expanded. CacheInformation PATH Specifies the location used to store management information for temporary user IDs and other shell extensions. By default, /var/cache/commsh/ is used. Change Root ChangeRoot [PATH] Change root to PATH before proceeding. If PATH is not specified, ~ is assumed. CheckRootSecurity LEVEL Specify the level of security checking performed before changing root to a new path. LEVEL is a numeric value that indicates the degree of checking, with lower numbers being less secure. By default, this is set to 2, and can range from 0 (no checking) to 4 (complete checking). If ChangeRoot is enabled and the path specified does not pass this security check, the login will fail. Attachment #5: commsh configuration directives 15 of 18

Dynamic Accounts AssignDynamicUID MIN MAX Enable assignment of a dynamic uid to the user. The range of allowed uids is [MIN, MAX] (inclusive). If you begin either MIN or MAX with a + or sign, then the uid range will be determined by adding MIN or MAX to the actual uid of the user. For example, if the community account has uid 10000, then a range of [+1, +1000] would correlate to [10001, 11000]. If all dynamic uids are assigned, then further login attempts will fail until a uid is freed up. DynamicHome PATH Set the path to use as the home directory for a temporary account. A hash (#) character within the path will be replaced by the temporary uid assigned. PATH is relative to the new root, if the root was changed. DynamicTemplateHome PATH Set the path used as a template for the home directory of a dynamic account. This parameter has no effect unless DynamicHome has been set. If no template directory is specified, a dynamically generated home directory will be empty. DynamicShell SHELL Set the shell to launch when using a dynamic account. If this option is not specified, the shell of the original account will be used. Note that in the default case, the shell must be specified correctly within the new root path's /etc/passwd. If no shell can be determined, the login will fail. This directive has no effect if LaunchShell is not enabled. DynamicExpiration TIMESPAN Set the length of time until a dynamic account is deleted and the temporary uid is reclaimed. This is measured from logout. Accounts are not guaranteed to be deleted at this exact time, but they will be cleaned up on the next run of commsh --reap, or the next run of commsh if DynamicAutoReap is enabled. (See below: TIMESPANS) DynamicAutoReap [On/Off] Enable (or disable) automatic reaping of dynamic accounts. If enabled, commsh will try to reclaim any expired dynamic accounts every time it is run. If disabled, accounts will be reclaimed only when commsh --reap is run, or when no more dynamic accounts are available. By default, this directive will enable automatic reaping. Attachment #5: commsh configuration directives 16 of 18

AllowSpecificUID Allow assignment of a specific dynamic uid that may or may not already exist. If enabled, this allows for multiple connections to use the same dynamic account by specifying the uid at the beginning of the passed command line (preceded by a hash (#)). The uid specified is still required to fall within the range of allowed dynamic uids. This option has no effect of AssignTempUID is not enabled. SECURITY NOTE: This option, if enabled, poses a significant security risk if a gateway application allows an end-user to pass through an arbitrary command string. Gateway servers must be intelligent enough to strip a leading hash character off the start of a user-issued command. Access Methods DirectAccess CMD Grant direct access to a given CMD. This option has no effect if LaunchShell is enabled. (See below: COMMAND PARAMETERS) AllowSftp Allow access to files via sftp. The sftp subsystem will not be launched until after the root has been changed and the temporary UID has been assigned. It is therefore required that the sftp subsystem be installed inside of the new root path, if ChangeRoot is enabled. LaunchShell Enable execution of another shell. If enabled, after all authentication is completed the new user's shell will be launched. If disabled, only DirectAccess commands will be allowed. If LaunchShell is not enabled and no command is given, login will fail. The external shell must be listed in /etc/shells. PassCommandToShell Enable passing of commands to an external shell. If disabled, any commands passed to commsh will be discarded when another shell is launched. This has no effect if LaunchShell is not enabled. Attachment #5: commsh configuration directives 17 of 18

NOTES: TIMESPANS: A timespan can either be an English-style time span ("2 days"), or can be a number of minutes ("1440" would equate to "1 day"). COMMAND PARAMETERS: The command parameters used by the DirectAccess parameter conform to the following specifications: The command issued must match the specified command exactly. If command parameters are expected, they must be included explicitly in the format of the command. Wildcard matching is included, but wildcards will not match spaces. Optional parameters can be specified by enclosing them in brackets ([]). If a wildcarded parameter is preceded by a tilde (~), then only files in the user's home directory can be specified. A final parameter of ** specifies that the user can add any number of additional parameters to the end of the command. Examples: DirectAccess /bin/rbash Allow the user to launch a restricted bash environment without parameters DirectAccess /usr/bin/whoami [--*] Allow the user to execute whoami with an optional -- option DirectAccess /bin/cat ~* Allow the user to execute cat on any file in the user's home directory DirectAccess /usr/bin/jobmanager ** Allow the user to execute jobmanager with any parameters Attachment #5: commsh configuration directives 18 of 18