4.1.0-Beta 2 Python Cartridge Agent Guide

Transcription

1 4.1.0-Beta 2 Python Cartridge Agent Guide Overview Congurations Logging Methods of installing and starting a CA Docker le scenario Puppet scenario Setting up the cartridge agent Step 1 - Check out the source Step 2 - Build using Maven Step 3 - Install and start the cartridge agent Docker le scenario - Building the Docker image Puppet scenario Running the Python cartridge agent without Puppet or Docker Overview The cartridge agent (CA) is a python based component that resides within a cartridge instance and handles the communication between the cartridge and Stratos. The Python cartridge agent communicates with the message broker to subscribe and publish topics, communicates with a remote Git repository to synchronize the artifacts in the cartridge with the remote repository and also publishes health statistics to the CEP. The Python cartridge agent source can be found the / components/org.apache.stratos.python.cartridge.agent directory. Congurations The Python cartridge agent is automatically congured based on the congurations in the <AGENT_HOME>/agent. conf le. The agent.conf le contains parameters to congure the message broker, Thrift receiver, monitoring server and more. In the case of VM's, Puppet will handle setting the values in this le. While, in the Kubernete's based container scenario, the Docker entry point script will handle the process of setting these values. The values for the agent.conf le parameters that do not have default values should be provided by the calling party; otherwise, the cartridge agent will not start. Click here to see the agent.conf Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 1

2 agent.conf mb.ip = mb.port = listen.address = thrift.receiver.ip = thrift.receiver.port = thrift.server.admin.username = thrift.server.admin.password = cep.stats.publisher.enabled = lb.private.ip = lb.public.ip = enable.artifact.update = auto.commit = auto.checkout = artifact.update.interval = port.check.timeout = enable.data.publisher = monitoring.server.ip = monitoring.server.port = monitoring.server.secure.port = monitoring.server.admin.username = monitoring.server.admin.password = log.le.paths = super.tenant.repository.path =/repository/deployment/server/ tenant.repository.path =/repository/tenants/ extension.instance.started =instance-started.sh extension.start.servers =start-servers.sh extension.instance.activated =instance-activated.sh extension.artifacts.updated =artifacts-updated.sh extension.clean =clean.sh extension.mount.volumes =mount_volumes.sh extension.member.started =member-started.sh extension.member.activated =member-activated.sh extension.member.suspended =member-suspended.sh extension.member.terminated =member-terminated.sh extension.complete.topology =complete-topology.sh extension.complete.tenant =complete-tenant.sh extension.subscription.domain.added =subscription-domain-added.sh extension.subscription.domain.removed =subscription-domain-removed.sh extension.artifacts.copy =artifacts-copy.sh extension.tenant.subscribed =tenant-subscribed.sh extension.tenant.unsubscribed =tenant-unsubscribed.sh The python cartridge agent persist the payload parameters, similar to how the Java agent worked. When the cartridge agent is initialized, it will read both the agent conguration les ( agent.conf and logging.ini ), and as required, will provide these values to the modules. In addition, the logging parameters can be congured via the <AGENT_HOME>/logging.ini le. Logging Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 2

3 The log les associated with the agent are located in the following two directories; the agent.screen log is in the /tmp directory and the other agent logs are in /var/log/apache-stratos/ directory. The agent.log is the standard cartridge agent log le; while, the error.log contains all the ERROR level logs. The agent.log is very verbose when the debug level logging ( DEBUG) is enabled. Therefore, it is useful to refer the error.log when the a gent.log is lled with a large number of DEBUG log entries. As the agent.screen.log le contains all the output of the Python Agent process's standard output (stdout), including all the uncaught exceptions, the agent. screen.log is useful in situations when the agent.log fails to include process related output. Methods of installing and starting a CA The cartridge agent installation and starting process will be done automatically; however, it will vary based on the approach used to deploy a cartridge in Stratos. In a Docker scenario this will be done by the entry point script while in a VM scenario this will be done by Puppet. Docker le scenario The approach used with the a Docker le is to install the needed service (e.g., PHP, MySQL, Tomcat etc.) and then add the cartridge agent to the image. The entrypoint script is the run script le. This script replaces the mandatory and optional values with the values that are passed to the container as environment variables and starts the Python process. Additionally, the run script will also persist the payload parameters in the <AGENT_HOME> /payload/launch.params le. Click here to see the run script run script source /root/.bashrc /usr/local/bin/populate-user-data.sh #mandatory parameters sed -i "s/mb-ip/${mb_ip}/g" /mnt/apache-stratos-python-cartridge-agent SNAPSHOT/agent.conf sed -i "s/mb-port/${mb_port}/g" /mnt/apache-stratos-python-cartridgeagent snapshot/agent.conf #parameters that can be empty #default values have to be set if [ -z "${LISTEN_ADDR}" ]; then sed -i "s/listen_addr/localhost/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/listen_addr/${listen_addr}/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf # defaults to the message broker IP if not set if [ -z "${CEP_IP}" ]; then sed -i "s/cep-ip/${mb_ip}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/cep-ip/${cep_ip}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf if [ -z "${CEP_PORT}" ]; then sed -i "s/cep-port/7711/g" /mnt/apache-stratos-python-cartridge- Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 3

4 agent snapshot/agent.conf sed -i "s/cep-port/${cep_port}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf if [ -z "${CEP_USERNAME}" ]; then sed -i "s/cep-admin-username/admin/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/cep-admin-username/${cep_username}/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf if [ -z "${CEP_PASSWORD}" ]; then sed -i "s/cep-admin-password/admin/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/cep-admin-password/${cep_password}/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf if [ -z "${ENABLE_HEALTH_PUBLISHER}" ]; then sed -i "s/enable_health_publisher/true/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/enable_health_publisher/${enable_health_publisher}/g" /mnt/apache-stratos-python-cartridge-agent snapshot/agent.conf if [ -z "${LB_PRIVATE_IP}" ]; then sed -i "s/lb_private_ip/ /g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/lb_private_ip/${lb_private_ip}/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf if [ -z "${LB_PUBLIC_IP}" ]; then sed -i "s/lb_public_ip/ /g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/lb_public_ip/${lb_public_ip}/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf if [ -z "${ENABLE_ARTFCT_UPDATE}" ]; then sed -i "s/enable_artfct_update/true/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/enable_artfct_update/${enable_artfct_update}/g" /mnt /apache-stratos-python-cartridge-agent snapshot/agent.conf if [ -z "${COMMIT_ENABLED}" ]; then Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 4

5 sed -i "s/commit_enabled/false/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/commit_enabled/${commit_enabled}/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf if [ -z "${CHECKOUT_ENABLED}" ]; then sed -i "s/checkout_enabled/true/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/checkout_enabled/${checkout_enabled}/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf if [ -z "${ARTFCT_UPDATE_INT}" ]; then sed -i "s/artfct_update_int/15/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s/artfct_update_int/${artfct_update_int}/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf if [ -z "${PORT_CHECK_TIMEOUT}" ]; then sed -i "s/port_check_timeout/600000/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/port_check_timeout/${port_check_timeout}/g" /mnt /apache-stratos-python-cartridge-agent snapshot/agent.conf if [ -z "${ENABLE_DATA_PUBLISHER}" ]; then sed -i "s/enable-data-publisher/false/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/enable-data-publisher/${enable_data_publisher}/g" /mnt /apache-stratos-python-cartridge-agent snapshot/agent.conf # defaults to the message broker IP if not set if [ -z "${MONITORING_SERVER_IP}" ]; then sed -i "s/monitoring-server-ip/${mb_ip}/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/monitoring-server-ip/${monitoring_server_ip}/g" /mnt /apache-stratos-python-cartridge-agent snapshot/agent.conf if [ -z "${MONITORING_SERVER_PORT}" ]; then sed -i "s/monitoring-server-port/7611/g" /mnt/apache-stratospython-cartridge-agent snapshot/agent.conf sed -i "s/monitoring-server-port/${monitoring_server_port}/g" Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 5

6 /mnt/apache-stratos-python-cartridge-agent snapshot/agent.conf if [ -z "${MONITORING_SERVER_SECURE_PORT}" ]; then sed -i "s/monitoring-server-secure-port/7711/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf sed -i "s/monitoring-server-secure-port /${MONITORING_SERVER_SECURE_PORT}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf if [ -z "${MONITORING_SERVER_ADMIN_USERNAME}" ]; then sed -i "s/monitoring-server-admin-username/admin/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf sed -i "s/monitoring-server-admin-username /${MONITORING_SERVER_ADMIN_USERNAME}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf if [ -z "${MONITORING_SERVER_ADMIN_PASSWORD}" ]; then sed -i "s/monitoring-server-admin-password/admin/g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf sed -i "s/monitoring-server-admin-password /${MONITORING_SERVER_ADMIN_PASSWORD}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf if [ -z "${LOG_FILE_PATHS}" ]; then sed -i "s/log_file_paths/ /g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf sed -i "s#log_file_paths#${log_file_paths}#g" /mnt/apachestratos-python-cartridge-agent snapshot/agent.conf if [ -z "${APP_PATH}" ]; then sed -i "s/app-path/ /g" /mnt/apache-stratos-python-cartridgeagent snapshot/agent.conf sed -i "s#app-path#${app_path}#g" /mnt/apache-stratos-pythoncartridge-agent snapshot/agent.conf if [ -z "${LOG_LEVEL}" ]; then sed -i "s/log_level/info/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/logging.ini sed -i "s/log_level/${log_level}/g" /mnt/apache-stratos-pythoncartridge-agent snapshot/logging.ini Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 6

7 SYS_LOG="/var/log/apache2/error.log" SAML_ALIAS="/var/www/" #sed -i "s#sys-log#${sys_log}#g" /etc/apache2/apache2.conf #sed -i "s#app-path#${app_path}#g" /etc/apache2/sites-available/default #sed -i "s#saml-alias#${saml_alias}#g" /etc/apache2/sites-available /default cd /mnt/apache-stratos-python-cartridge-agent snapshot python agent.py > /tmp/agent.screen.log 2>&1 & /usr/sbin/apache2ctl -D FOREGROUND Stratos will provide these values via Kubernetes using the values provided via the application denition JSON. Click here to see a sample Docker le FROM ubuntu:14.04 MAINTAINER chamilad@wso2.com RUN apt-get update RUN apt-get install -y wget WORKDIR /opt/ ################################# # Enable ssh - This is not good. /docker-ssh-considered-evil/ # For experimental purposes only ################################## RUN apt-get install -y openssh-server RUN mkdir -p /var/run/sshd RUN echo 'root:g' chpasswd RUN sed -i "s/permitrootlogin without-password/#permitrootlogin withoutpassword/" /etc/ssh/sshd_cong ################## # Install PHP ################## RUN apt-get install -y apache2 php5 zip unzip stress RUN rm /etc/apache2/sites-enabled/000-default.conf ADD les/000-default.conf /etc/apache2/sites-enabled/000-default.conf #ADD les/apache2/apache2.conf /etc/apache2/apache2.conf #ADD les/apache2/sites-available/default-ssl /etc/apache2/sitesavailable/default-ssl #ADD les/apache2/sites-available/default /etc/apache2/sites-available /default ################################## # Install Dependent Python Libs ################################## RUN apt-get install -y git python python-pip python-dev gcc RUN pip install paho-mqtt RUN pip install GitPython==0.3.1-beta2 RUN pip install psutil RUN pip install gittle RUN pip install pexpect ################## Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 7

8 # Congure Agent ################## WORKDIR /mnt/ RUN mkdir -p /mnt/packs RUN mkdir -p /mnt/apache-stratos-python-cartridge-agent snapshot ADD packs/apache-stratos-python-cartridge-agent snapshot.zip /mnt /packs/apache-stratos-python-cartridge-agent snapshot.zip WORKDIR /mnt/apache-stratos-python-cartridge-agent snapshot RUN unzip -q../packs/apache-stratos-python-cartridge-agent SNAPSHOT.zip WORKDIR /mnt/ RUN rm -rf packs RUN mkdir -p /mnt/apache-stratos-python-cartridge-agent snapshot /payload RUN mkdir -p /mnt/apache-stratos-python-cartridge-agent snapshot /extensions ADD packs/extensions /mnt/apache-stratos-python-cartridge-agent SNAPSHOT/extensions RUN chmod +x /mnt/apache-stratos-python-cartridge-agent snapshot /extensions/* RUN mkdir -p /var/log/apache-stratos/ RUN touch /var/log/apache-stratos/cartridge-agent-extensions.log EXPOSE ################### # Setup run script ################### ADD run /usr/local/bin/run RUN chmod +x /usr/local/bin/run ADD les/populate-user-data.sh /usr/local/bin/populate-user-data.sh RUN chmod +x /usr/local/bin/populate-user-data.sh ENTRYPOINT /usr/local/bin/run /usr/sbin/sshd -D This sample Docker le is not recommended in a production scenario as SSHD is strongly not recommended for Docker containers. Puppet scenario In Virtual Machine (VM) scenario, Puppet will be used to congure the VM. Puppet congurations are located in the /tools/puppet3/modules/python_agent/ directory. The old agent and java modules are not used anymore, unless they are explicitly needed in cartridges Java related cartridges (e.g., tomcat etc.). The use of the Java cartridge agent has been dropped from the Apache Stratos release onwards. For more information, see About this Release m4 developer Setting up the cartridge agent Follow the steps below to setup the cartridge agent: Step 1 - Check out the source Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 8

9 1. Checkout the source from Apache Stratos remote repository to a folder of your choice. git clone <local-foldername> For example: git clone mylocalrepo Make sure that you have setup Git properly. For more information, go to /Getting-Started-First-Time-Git-Setup 2. If required, you can execute the following command to get details on the deployable cartridge agent ZIP le. ls -l products/python-cartridge-agent/target If Stratos has been built successfully, the deployable cartridge agent ZIP le named apache-stratospython-cartridge-agent-<version>-snapshot.zip (e.g., apache-stratos-pythoncartridge-agent snapshot.zip ) can be found in the /products/python-cartridge-agent /target/ directory. Step 2 - Build using Maven 1. Go to the top level of the directory in which you checked out the source. cd <local-folder-name> 2. For example: cd mylocalrepo Use Maven to build the source distribution of the release. mvn clean install Step 3 - Install and start the cartridge agent The cartridge agent installation and starting process will be done automatically; however, it will vary based on the approach used to deploy a cartridge in Stratos. In a Docker scenario this will be done by the enrtrypoint script while in a VM scenario this will be done by Puppet. For more information, see Methods of installing and starting a CA. Docker le scenario - Building the Docker image 1. Checkout the Docker le content. git clone 2. Navigate to the stratos-container-image-php directory. cd stratos-container-image-php 3. Copy the cartridge agent artifact ZIP le from the products directory to the packs directory. cp <STRATOS_HOME>/products/python-cartridge-agent/target/apache-stratos-pythoncartridge-agent SNAPSHOT.zip packs/ 4. Build the Docker image. sudo docker build -t chamilad/stratos-php:4.1.0-alpha Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0 9

10 5. Verify the built image sudo docker images Puppet scenario Follow the steps below to congure the Puppet module for the Python cartridge agent: 1. Copy the Python agent distribution to the Puppet master. For more information, see Conguring Puppet Master. 2. Set the agent conguration values in the base.pp le, found in the /etc/puppet/manifests/nodes/ director y. The Puppet module for the Python cartridge agent will congure the cartridge agent based on the values provided. Running the Python cartridge agent without Puppet or Docker Follow the steps below to run the python cartridge agent without Puppet or Docker: 1. Navigate to the apache-stratos-python-cartridge-agent snapshot directory. cd <STRATOS_HOME>/products/ python-cartridge-agent/target/apache-stratos-pythoncartridge-agent snapshot 2. Start the cartridge agent by running the agent.py le. nohup python agent.py & Copyright 2015 The Apache Software Foundation, Licensed under the Apache License, Version