Server guides for the GIRAF project A joint guide produced by SW611 & SW616 Aalborg University
Contents 1 Accessing the GIRAF servers (SW611)........................... 3 2 Using Docker (SW611).......................................... 4 3 Add database user (SW616).................................... 5 4 Links to all services (SW616).................................... 6 5 Expand disk with LVM (SW616).................................. 9 6 Restore from backup (SW616)................................. 10 6.1 Jenkins 10 6.2 ApplicationDB 11 6.3 Phabricator 12 7 Service Administration........................................ 13 7.1 Git (SW611) 13 7.2 Jenkins (SW616) 14 7.3 Artifactory (SW611) 15 7.4 Phabricator (SW616) 15 Bibliography................................................. 16
1. Accessing the GIRAF servers (SW611) There are four (4) servers accessible to the designated server group(s). This section will specify general information with respect to these servers, and conventions used when managing these servers. The services running on each server are specified in section 4. web.giraf.cs.aau.dk The web server can be accessed from outside the AAU internal network, through SSH using port 22. git.giraf.cs.aau.dk The git server can be accessed from outside the AAU internal network, through SSH using port 22. jenkins.giraf.cs.aau.dk The jenkins server can be accessed from outside the AAU internal network, through SSH using port 22. backup01.giraf.cs.aau.dk The backup server cannot be accessed from outside the AAU internal network. If you wish to access the server from your private network, without using a VPN to AAU, you may SSH to the backup server from either of the other three (3) servers. When first accessing the servers, we recommend that all created accounts be fitted with an SSH key immediately, to ensure good security. If you are not experienced with SSH keys, this is a fitting guide. For accessing the servers, the SSH client PuTTY is recommended. PuTTY includes an SSH key generator, PuTTYgen, that may be used to generate the SSH keys. We have disabled SSH login to root on all servers, and recommend that you do not change this. If you wish to access root, use the sudo command. When you want to access each of the servers for the first time, we have to setup a new account for you on all of them. Contact either of the following, for support with the initial user creation process: mclc13@student.aau.dk - Mikkel sbjerg13@student.aau.dk - Søren sela12@student.aau.dk - Simon
2. Using Docker (SW611) All servers, except for the backup server, uses the Docker platform for the services running on them. We recommend that you read up on the documentation for Docker Engine in order to understand how Docker works. Further documentation specific to GIRAF, and argumentation for choosing Docker, may be found in the reports from groups SW611 and SW616.
3. Add database user (SW616) One of the most requested ad hoc task is to add database users for programmers. This is done using the following commands. mysql -hweb.giraf.cs.aau.dk -P3333 -uroot -p (connects to the database server as root) CREATE USER username @ % IDENTIFIED BY password (create user with username username and password password. User has access from all ip addresses (%)); GRANT SELECT ON giraf.* TO username @ % ; (grant user access to select on all tables in the database giraf) GRANT SELECT,INSERT,DELETE,UPDATE,ALTER ON girafdev.* TO username @ % ; (grant user access to select, insert, delete, update and alter table in the database girafdev) SHOW GRANTS FOR username @ % (Show grants assigned to user);
4. Links to all services (SW616) This chapter provides information about all running services and how to access these. Jenkins URL: http://jenkins.giraf.cs.aau.dk Login using AAU Access Control Yes Yes Yes Phabricator URL: http://web.giraf.cs.aau.dk Login using AAU Access Control Yes Yes Yes Git URL: http://giraf.giraf.cs.aau.dk Login using credentials from AAU account Yes Yes Yes ApplicationDB Address: web.giraf.cs.aau.dk:3333 Login using credentials provided by server group Yes Yes Yes SymmetricDS Address: web.giraf.cs.aau.dk:31415 Login using credentials provided by server group Yes Yes No
7 Webadmin URL: http://web.giraf.cs.aau.dk:8081 Login using credentials provided by server group Yes Yes Yes Wildfly Public URL: http://web.giraf.cs.aau.dk:8080 No login Yes Yes Yes Wildfly Administration Console URL: http://web.giraf.cs.aau.dk:9990 Login using credentials provided by server group Yes Yes No Artifactory URL: http://jenkins.giraf.cs.aau.dk:8080 No login for normal users. Admin-login with credentials provided by server group Yes Yes Yes Glances URLs: http://{jenkins,web,git}.giraf.cs.aau.dk:61208 No login needed Yes Yes No Filestore URL: http://jenkins.giraf.cs.aau.dk:8083/files SFTP: jenkins.giraf.cs.aau.dk:2222 No login needed Yes Yes No Javadocs URL: http://git.giraf.cs.aau.dk:8080 No login needed Yes Yes Yes
8 Chapter 4. Links to all services (SW616) Adminer URL: http://git.giraf.cs.aau.dk:10080 Login using credentials provided by server group Yes Yes No
5. Expand disk with LVM (SW616) To expand the volume with LVM the following commands has been used [1]: reboot (first reboot to make CentOS detect the new space) fdisk -l (show disks - find which disk which needs to be added / has added space) fdisk /dev/sdb (change partition table on /dev/sdb/) Select n (new partition) Select p (primary) Select 1 (select next number available) Enter (First cylinder) Enter (Last cylinder) t (select partition) 8e (Linux LVM filesystem) w (Write changes) fdisk -l (show disks - if new LVM is not presents - reboot) pvcreate /dev/sda3 (create new volume for use with LVM) vgdisplay (Show status and name for volume group which needs to be expanded) vgextend centos /dev/sda3 (Extend volume group centos) pvscan (Scan for new volume group) lvdisplay (Show LVM status, also show name of LVM volume) lvextend -L+2G /dev/centos/root /dev/sda3 (Extend volume /dev/centos/root with 2 GB from /dev/sda3 (you can exclude -L+2G if you want to use all space)) xfs_growfs /dev/centos/root (Grow file system - detect new space) df -h (Show status for file system)
6. Restore from backup (SW616) Let s assume that everything has crashed and you are forced to reinstall the servers. This guide will explain how to get everything setup again if you are in this situation. Please note that all commands are assumed to be run as root. The first step is to install Docker on the server. Docker s official installation guide is found here: https://docs.docker.com/engine/installation/linux/ubuntulinux 6.1 Jenkins Now that Docker is installed, we can create a container for Jenkins. We will be using the official image for this, this is installed with: docker run --name jenkins -p 80:8080 -v /srv/jenkins_home:/var/jenkins_home jenkins Jenkins should now be accessible at port 80, check that this is the case. We now have a fresh, working Jenkins. In order to retrieve the old data, we need to do this from the backup server. We need to stop the Jenkins container before this. This is done with: docker stop jenkins Now we delete all files in /srv/jenkins_home with command cd /srv/jenkins_home; rm -rf * Now we have to get into the backup-server, called backup01.giraf.cs.aau.dk and copy the backup to the Jenkins server with the following commands: cd /srv/backup/jenkins/jenkins scp "jenkins.{date}-{time}.tar.gz" {USER}@{IP}:/srv/jenkins_home Now we go back to the new server running Jenkins and extracts the backup with this command: cd /srv/jenkins_home tar -zxvf jenkins.{date}-{time}.tar.gz cd /srv/jenkins_home/srv/backup/jenkins/jenkins/jenkins.{date}-{time}
6.2 ApplicationDB 11 mv * /srv/jenkins_home All files are copied and placed in the correct directory, and we are ready to run docker start jenkins 6.2 ApplicationDB First step is to install the MariaDB container with this command: docker run --name applicationdb -v /srv/applicationdb:/var/lib/mysql -e MYSQL_ROOT_PASSWORD={PASSWORD} mariadb When the database is running, we have to stop it before copying from the backup: docker stop applicationdb We can now copy the neccessary files from the backup with: cd /srv/backup/web/applicationdb scp "applicationdb.{date}-{time}.tar.gz" {USER}@{IP}:/srv/application Back on the new server we extract and move the neccessary files: cd /srv/applicationdb tar -zxvf applicationdb.{date}-{time}.tar.gz cd /srv/applicationdb/srv/backup/web/applicationdb/applicationdb.{date}-{time} mv giraf /srv/applicationdb mv girafdev /srv/applicationdb mv girafrestdev /srv/applicationdb Everything is ready for starting the container again with: docker start applicationdb
12 Chapter 6. Restore from backup (SW616) 6.3 Phabricator In order to restore Phabricator, a little more work has to be done since this is split into to containers - one for the database, and one for the web server. Let us start by installing both of these: docker run --name phabricator-database yesnault/docker-phabricator-mysql docker run --name phabricator-web -p 80:80 --link phabricator-database:database yesnault/docker-phabricator Since we need to modify data files, we need to stop them: docker stop phabricator-database phabricator-web We now have to clean up the generated database files by running: cd $(docker inspect -f {{ (index.mounts 0).Source }} phabricator-database); rm -rf default_* mysql performance_schema We can now copy the neccessary files from the backup with: cd /srv/backup/web/phabricator-database scp "phabricator-database.{date}-{time}.tar.gz" {USER}@{IP}:$(docker inspect -f {{ (index.mounts 0).Source }} phabricator-database) Back on the new server we extract and move the neccessary files: cd ~ tar -zxvf phabricator - database.{ DATE } -{ TIME }. tar.gz cd ~/ srv / backup / web / phabricator - database.{ DATE } -{ TIME } mv mysql $( docker inspect -f {{ ( index. Mounts 0). Sourc e }} phabricator - database ) mv default_ * $( docker inspect -f {{ ( index. Mounts 0). S ource }} phabricator - database )} mv phabricator_ * $( docker inspect -f {{ ( index. Mounts 0). Source }} phabricator - database )} mv performance \ _ schema $( docker inspect -f {{ ( index. Mounts 0). Source }} phabricator - database )} The database server is ready to be started again, and then it s time to repeat the above steps for phabricator-web
7. Service Administration 7.1 Git (SW611) Accessible at: http://git.giraf.cs.aau.dk The system has been configured to only allow access to members of specific groups. Obviously this is currently defined as those participating in the GIRAF multi project in 2016. Specific rules are defined for members of the two server groups, as well as for Ulrik, to ensure administrative rights. To gain access to the system, the easiest way is to contact tbendi13@student.aau.dk, although any of the server group members should be able to assist. User sign up procedure When a new user signs into GOGS for the first time, their account is created, but they aren t automatically added to the Giraf organization and its Developers team. To give users write access to the repositories, an admin must add them to the team manually. It s a bit of a hassle, but it doesn t take long, and the process is basically: Open the Giraf organization. The easiest way to do this is to find it in the list of organizations on the dashboard, in the right hand side. Open the Developers team, which can be found in the right hand side on the organization page. It defaults to the list of members. At the end of this list there s a text field which allows you to add someone to the team. Pretty simple once you know about it. New repositories, special teams, and access modifications When adding a new repository it will automatically become available to the Owners team, but in order to give developers access, it must be added to the Developers team. This is done in much the same way as adding a user to the team, only the Repositories link in the left hand side info bar (on the team page) needs to be clicked first. There are a couple of special teams as well. Here s a quick rundown of all the teams, for good measure: Owners: Server group members as well as Ulrik should be added to this one. They should be able to add themselves to it, though, so that shouldn t be a problem. All repositories under the Giraf organization are automatically available with write access to all members of this group. Developers:
14 Chapter 7. Service Administration As the name suggests, this is where the regular members should be. It will give them full access to all the repositories that have been added there. DeprecatedRepos: There were a bunch of seemingly unused repositories on the old server, but rather than deleting them, they have been added to this group. These are the repositories that were specifically marked as deprecated and deletable, but for some reason they weren t deleted, so it shall now be up to you to decide on whether or not they should stay or go. UndecidedFate: These are more or less in the same category as the deprecated repositories, except that the former server group didn t mark them as deprecated specifically. AdminSpecific: Hasn t been maintained as well as it should be. The idea was that repositories that were meant only for server admins eyes should be added here. Admins wouldn t need to be added to the group, so it d mostly just be to get a quick overview of the admin repos. DevRO: Not currently used, but if there s ever a repository that the developers should have read only access to, then that s what this group is for. Obviously the developers in question would also need to be added to the group, so there s that. JavaDocs: The documentation generation application uses API access to GOGS to find the repositories that it needs to clone and generate for. So adding a repository to this team will enable it to generate documentation for it, and I m sure you can work out what removing a repo from the team will do. 7.2 Jenkins (SW616) Accessible at: http://jenkins.giraf.cs.aau.dk Jenkins is viewable and editable for everyone with a AAU login. However, administration is only possible for the server group. If you wish to gain administration access, please contact the server group from 2016. There are a few important things that you need to know about. First of all, the location for key files to Google Play is located at /srv/jenkins_home/google_play_keys. This directory contains key files as well as credentials for these. Another important thing to keep in mind is that all configs for the project are saved via version control. This means that every single change to the config will be saved, and it s possible to select an older version. A last remark about Jenkins is how this is restarted correctly. It s important that the container receives the correct appends to the hosts file in order for Artifactory to work correctly. This means that instead of running docker start jenkins, you need to use the bash script located at /home/mclc/restart_jenkins.sh
7.3 Artifactory (SW611) 15 7.3 Artifactory (SW611) Accessible at: http://jenkins.giraf.cs.aau.dk:8080 Artifactory is viewable by anyone able to access the URL. Artifactory administration can be done manually using an admin login, credentials for which is stored in the /srv/jenkins_home/credentials/artifactory.properties file. When logged in as admin, it is possible to import a system and repository to Artifactory, using the Admin panel, and this is where restoring the service from a backup can potentially be performed. Artifactory has not needed any maintenance from either of our groups, ever since we migrated to the current servers, and we do not expect this to change in the future. 7.4 Phabricator (SW616) Accessible at: http://web.giraf.cs.aau.dk/ Phabricator can be accessed using AAU login, which creates a new Phabricator user the first time you log in. The Phabricator account s username is the first part of your AAU mail. We suggest that each user also sets their real name, and include their group number, e.g. sela12 (Simon Ellegaard Larsen (616)). Regular users are permitted to do everything that is required for normal operation, so the administrator account is not needed. However, there is a shared admin account in case a situation arises where it is needed. The login credentials for this account can be found on the codesheet handed to Ulrik (ulrik@cs.aau.dk).
Bibliography [1] RootUsers. (May 10, 2016). How to increase the size of a linux lvm by adding a new disk, [Online]. Available: https://www.rootusers.com/how-to-increase-the-size-ofa-linux-lvm-by-adding-a-new-disk (cited on page 9).