Lesson: Web Programming(3) Omid Jafarinezhad Sharif University of Technology

Lesson: Web Programming(3) Omid Jafarinezhad Sharif University of Technology

Materials HTTP, JavaScript, CSS, HTML5, ReactJs, Flow, Progressive Web App Golang, NodeJs, MongoDB, PostgreSQL, Redis Docker, Git, NPM, YUIDoc, Jest, WebPack, Gulp, Browserify, Locust (Optional/Research) Kubernetes, InfluxDB, RabbitMQ, grpc, Ansible

Git Helps software developers to work together and maintain a complete history of their work

Git A version control system (VCS) allows you to track the history of a collection of files Git is a distributed revision control and source code management system with an emphasis on speed > sudo apt-get install git > sudo yum install git > git --version

Git - Life Cycle A Git repository contains the history of a collection of files starting from a certain directory Amend: After committing, if you realize something is wrong, then you correct the last commit and push the changes to the repository

Working tree The working tree contains the set of working files for the repository You can modify the content and commit the changes as new commits to the repository

Commit When you commit your changes into a repository this creates a new commit object in the Git repository. This commit object uniquely identifies a new revision of the content of the repository This revision can be retrieved later, for example, if you want to see the source code of an older version Each commit object contains the author and the committer. This makes it possible to identify who did the change. The author and committer might be different people. The author did the change and the committer applied the change to the Git repository. This is common for contributions to open source projects

Staging area The staging area is the place to store changes in the working tree before the commit The staging area contains a snapshot of the changes in the working tree (changed or new files) relevant to create the next commit and stores their mode (file type, executable bit)

The staging area (aka index) is a container where git collects all changes which will be part of the next commit. If you are editing a versioned file on your local machine, git recognizes that your file is modified - but it will not be automatically part of your next commit and is therefore unstaged. Staging the file will put the file into the staging area (index). The next git commit will transfer all items from staging area into your repository

Revision Represents a version of the source code Git implements revisions as commit objects (or short commits ). These are identified by an SHA-1 hash

Tag A tag points to a commit which uniquely identifies a version of the Git repository. With a tag, you can have a named point to which you can always revert to. You can revert to any point in a Git repository, but tags make it easier. The benefit of tags is to mark the repository for a specific reason, e.g., with a release. Branches and tags are named pointers, the difference is that branches move when a new commit is created while tags always point to the same commit. Tags can have a timestamp and a message associated with them

Branch Git supports branching which means that you can work on different versions of your collection of files, A branch allows the user to switch between these versions so that he can work on different changes independently from each other. One of the branches is the default (typically named _master ). The default branch is the one for which a local branch is automatically created when cloning the repository For example, if you want to develop a new feature, you can create a branch and make the changes in this branch. This does not affect the state of your files in other branches. For example, you can work independently on a branch called production for bugfixes and on another branch called feature_123 for implementing a new feature

Brach (2) A branch is a named pointer to a commit. Selecting a branch in Git terminology is called to checkout a branch. If you are working in a certain branch, the creation of a new commit advances this pointer to the newly created commit ([p:231]... branches move when a new commit is created...) Each commit knows their parents (predecessors). Successors are retrieved by traversing the commit graph starting from branches or other refs, symbolic references (for example: HEAD) or explicit commit objects. This way a branch defines its own line of descendants in the overall version graph formed by all commits in the repository

Example terminal prompt A working directory

To initialize a Git repository here

to see what the current state of our project

I created a file called octocat.txt in the octobox repository

To tell Git to start tracking changes made to octocat.txt, we first need to add it to the staging area by using git add

To store our staged changes we run the commit command with a message describing what we've changed

You also can use wildcards if you want to add many files of the same type I put some(.txt file) in a directory named "octofamily" and some others ended up in the root of our "octobox" directory

So we've made a few commits. Now let's browse them to see what we changed. Think of Git's log as a journal that remembers all the changes we've committed so far, in the order we committed them

To push our local repo to the GitHub server we'll need to add a remote repository The push command tells Git where to put our commits > Push our local changes to our origin repo

Let's pretend some time has passed. We've invited other people to our GitHub project who have pulled your changes, made their own commits, and pushed them. We can check for changes on our GitHub repository and pull down any new changes Fast-Forward

Compare HEAD by Pulled files :-) 1,4 range (line) - removed + added

You can unstage files by using the git reset command create a branch called clean_up You can switch branches using the git checkout <branch> command

Switching Back to master We're already on the master branch, we can merge our changes from the clean_up branch into the master branch

Basic Branching and Merging Example Let s go through a simple example of branching and merging with a workflow that you might use in the real world. You ll follow these steps: Do some work on a website Create a branch for a new story you re working on Do some work in that branch At this stage, you ll receive a call that another issue is critical and you need a hotfix. You ll do the following: Switch to your production branch Create a branch to add the hotfix After it s tested, merge the hotfix branch, and push to production Switch back to your original story and continue working

You ve decided that you re going to work on issue #53 in whatever issue-tracking system your company uses

Now you get the call that there is an issue with the website, and you need to fix it immediately. With Git, you don t have to deploy your fix along with the iss53 changes you ve made, and you don t have to put a lot of effort into reverting those changes before you can work on applying your fix to what is in production

You ll notice the phrase fast-forward in that merge. Because the commit C4 pointed to by the branch hotfix you merged in was directly ahead of the commit C2 you re on, Git simply moves the pointer forward. To phrase that another way, when you try to merge one commit with a commit that can be reached by following the first commit s history, Git simplifies things by moving the pointer forward because there is no divergent work to merge together this is called a fast-forward.

After your super-important fix is deployed, you re ready to switch back to the work you were doing before you were interrupted

Suppose you ve decided that your issue #53 work is complete and ready to be merged into your master branch. In order to do that, you ll merge your iss53 branch into master, much like you merged your hotfix branch earlier This looks a bit different than the hotfix merge you did earlier. In this case, your development history has diverged from some older point. Because the commit on the branch you re on isn t a direct ancestor of the branch you re merging in, Git has to do some work. In this case, Git does a simple three-way merge, using the two snapshots pointed to by the branch tips and the common ancestor of the two. Instead of just moving the branch pointer forward, Git creates a new snapshot that results from this three-way merge and automatically creates a new commit that points to it. This is referred to as a merge commit, and is special in that it has more than one parent It s worth pointing out that Git determines the best common ancestor to use for its merge base

Now that your work is merged in, you have no further need for the iss53 branch. You can close the ticket in your ticket-tracking system, and delete the branch Occasionally, this process doesn t go smoothly. If you changed the same part of the same file differently in the two branches you re merging together, Git won t be able to merge them cleanly. If your fix for issue #53 modified the same part of a file as the hotfix branch, you ll get a merge conflict that looks something like this

Your file contains a section that looks something like this This means the version in HEAD (your master branch, because that was what you had checked out when you ran your merge command) is the top part of that block (everything above the =======), while the version in your iss53 branch looks like everything in the bottom part. In order to resolve the conflict, you have to either choose one side or the other or merge the contents yourself

Cherry-pick Problem is found on the Production branch. A fix for the problem is developed in commit H, but commit G does not need to be applied to the Development branch Commit H is cherry-picked onto the Development branch, resulting in commit H'. Note that the changes made in commit G are not included on the Development branch

Reset On the commit-level, resetting is a way to move the tip of a branch to a different commit This can be used to remove commits from the current branch (a simple way to undo changes)

> git reset HEAD~~ HEAD is now at 326c9f commit message > git reset HEAD~2 HEAD is now at 326c9f commit message

Revert Reverting undoes a commit by creating a new commit. This is a safe way to undo changes, as it has no chance of re-writing the commit history

> git revert HEAD~2

Rebase Rebasing is the process of moving or combining a sequence of commits to a new base commit. Rebasing is most useful and easily visualized in the context of a feature branching workflow The primary reason for rebasing is to maintain a linear project history For example, consider a situation where the master branch has progressed since you started working on a feature branch. You want to get the latest updates to the master branch in your feature branch, but you want to keep your branch's history clean so it appears as if you've been working off the latest master branch. This gives the later benefit of a clean merge of your feature branch back into the master branch.

Docker a helpful tool for packing, shipping, and running applications within containers

What are containers and VMs? Containers and VMs are similar in their goals: to isolate an application and its dependencies into a self-contained unit that can run anywhere remove the need for physical hardware, allowing for more efficient use of computing resources, both in terms of energy consumption and cost effectiveness The main difference between containers and VMs is in their architectural approach Docker share the host system s kernel with other containers

Virtual machines run guest operating systems note the OS layer in each box. This is resource intensive, and the resulting disk image and application state is an entanglement of OS settings, system-installed dependencies, OS security patches, and other easy-to-lose, hard-to-replicate ephemera guest machine host machine Containers can share a single kernel, and the only information that needs to be in a container image is the executable(app A, B, C) and its package dependencies (Bin/Lib), which never need to be installed on the host system. Because they contain all their dependencies, there is no configuration entanglement; a containerized app runs anywhere

VM New Setting and Install Start Docker Pull Run (Stop/start)

Where does Docker come in? Docker is an open-source project based on Linux containers. It uses Linux Kernel features like namespaces and control groups to create containers on top of an operating system Containers are far from new; Google has been using their own container technology for years. Others Linux container technologies include Solaris Zones, BSD jails, and LXC, which have been around for many years

So why is Docker all of a sudden gaining steam? Ease of use: Docker has made it much easier for anyone developers, systems admins, architects and others to take advantage of containers in order to quickly build and test portable applications Speed: Docker containers are very lightweight and fast Docker Hub: Docker users also benefit from the increasingly rich ecosystem of Docker Hub, which you can think of as an app store for Docker images Modularity and Scalability: Docker makes it easy to break out your application s functionality into individual containers

Install Docker (CE/EE) Docker is available in two editions: Community Edition (CE) and Enterprise Edition (EE) > curl -fssl https://download.docker.com/linux/ubuntu/gpg sudo apt-key add > sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" > sudo apt-get update > sudo apt-get install -y docker-ce > sudo systemctl status docker docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2016-05-01 06:53:52 CDT; 1 weeks 3 days ago Docs: https://docs.docker.com Main PID: 749 (docker) -- Ubuntu 16.04

Docker Command Without Sudo (Optional, Dev) By default, running the docker command requires root privileges that is, you have to prefix the command with sudo. It can also be run by a user in the docker group, which is automatically created during the installation of Docker. If you attempt to run the docker command without prefixing it with sudo or without being in the docker group, you'll get an output like this If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. See 'docker run --help'. > sudo usermod -ag docker ${USER} > su - ${USER} 1. 2. Add your username to the docker group apply the new group membership(without logout)

A brief explanation of containers An image is a lightweight, stand-alone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files A container is a runtime instance of an image what the image becomes in memory when actually executed. It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so Containers run apps natively on the host machine s kernel. Containers can get native access, each one running in a discrete process, taking no more memory than any other executable

Docker image repositories https://hub.docker.com https://store.docker.com

Pull an image from Docker Hub > docker pull debian Using default tag: latest latest: Pulling from library/debian fdd5d7827f33: Pull complete a3ed95caeb02: Pull complete Digest: sha256:e7d38b3517548a1c71e41bffe9c8ae6d6d29546ce46bf62159837aad072c90aa Status: Downloaded newer image for debian:latest

Pull an image from Docker Hub (2) > docker pull ubuntu:14.04 14.04: Pulling from library/ubuntu 5a132a7e7af1: Pull complete fd2731e4c50c: Pull complete 28a2f68d1120: Pull complete a3ed95caeb02: Pull complete Digest: sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2 Status: Downloaded newer image for ubuntu:14.04

Displaying Docker Images To see the list of Docker images on the system, you can issue the following command > docker images ubuntu-java latest ubuntu latest consul latest mongo latest influxdb latest postgres latest node latest rabbitmq latest nginx latest 389560217a19 3 weeks ago 889MB 8b72bba4485f 3 weeks ago 120MB c6f7042bd0f8 2 months ago 51.7MB b39de1d79a53 2 months ago 359MB d913baeaf36b 2 months ago 227MB 33b13ed6b80a 2 months ago 269MB 045ea30913bb 2 months ago 667MB bad16bdb4e74 2 months ago 124MB b8efb18f159b 2 months ago 107MB

Images and layers A Docker image is built up from a series of layers Each layer represents an instruction in the image s Dockerfile When you use docker pull to pull down an image from a repository, each layer is pulled down separately, and stored in Docker s local storage area, which is usually /var/lib/docker/ on Linux hosts. You can see these layers being pulled in this example:

> docker pull debian Using default tag: latest latest: Pulling from library/debian fdd5d7827f33: Pull complete a3ed95caeb02: Pull complete Digest: sha256:e7d38b3517548a1c71e41bffe9c8ae6d6d29546ce46bf62159837aad072c90aa Status: Downloaded newer image for debian:latest

Docker run Running of containers is managed with the Docker run command

Assign name and allocate pseudo-tty ( name, -it) --interactive, -i --name --tty, -t Keep STDIN open even if not attached Assign a name to the container Allocate a pseudo-tty (pseudoterminal) > docker run --name test -it debian root@d6c0fe130dba:/# exit > docker ps -a grep test d6c0fe130dba debian:7 "/bin/bash" This example runs a container named test using the debian:latest image. The -it instructs Docker to allocate a pseudo-tty connected to the container s stdin; creating an interactive bash shell in the container 26 seconds ago Exited (13) 17 seconds ago test

Listing of Containers One can list all of the containers on the machine via the docker ps command. This command is used to return the currently running containers > docker ps -a This command is used to list all of the containers on the system

Run in background and remove if exits --detach, -d --rm Run container in background and print container ID Automatically remove the container when it exits >docker run --name ubuntu_bash --rm -i -t ubuntu bash root@4f47613a44e3:/# exit exit > >docker run --name ubuntu_bash -d --rm -i -t ubuntu bash 327f7e6b08d6bd1404ec88330df6af2638be3b85d055a66177b5a61b4ff04f75 >

Publish or expose port (-p, expose) --publish, -p Publish a container s port(s) to the host $ docker run -p 127.0.0.1:80:8080 ubuntu bash This binds port 8080 of the container to port 80 on 127.0.0.1 of the host machine

Set environment variables (-e, env, env-file) --env, -e Set environment variables --env-file Read in a file of environment variables Use the -e, --env, and --env-file flags to set simple (non-array) environment variables in the container you re running, or overwrite variables that are defined in the Dockerfile of the image you re running. $ docker run --env MYVAR2=foo --env-file./env.list ubuntu bash

Set metadata on container (-l, label, label-file) --label, -l Set meta data on a container --label-file Read in a line delimited file of labels A label is a key=value pair that applies metadata to a container $ docker run -l my-label --label com.example.foo=bar ubuntu bash The my-label key doesn t specify a value so the label defaults to an empty string(""). To add multiple labels, repeat the label flag (-l or --label)

Set ulimits in container ( ulimit) $ docker run --ulimit nofile=1024:1024 debian sh -c "ulimit -n" 1024 <type>=<soft limit>[:<hard limit>] A hard limit is the maximum allowed to a user/process;increasing hard limit can be done only by root A soft limit is the effective value right now for that user/process; a soft limit may be increased up to the value of the hard limit $ docker run -ti node /bin/bash user@4d04d06d5022:/# ulimit -a... max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 cpu time (seconds, -t) unlimited max user processes (-u) 58729.

Mount volume --volume, -v Bind mount a volume $ docker run -v /data:/path-in-container/data -i -t ubuntu bash

Memory Constraints $ docker run -it -m 300M ubuntu:16.04 /bin/bash

CPU Constraints You can limit CPU, either using a percentage of all CPUs, or by using specific cores. The setting is a bit strange -- 1024 means 100% of the CPU, so if you want the container to take 50% of all CPU cores, you should specify 512 You can also only use some CPU cores using cpuset-cpus $ docker run -ti --c 512 ubuntu:16.04 /bin/bash $ docker run -ti --cpuset-cpus=0,4,6 ubuntu:16.04 /bin/bash

Listing All Docker Networks > docker network ls This command can be used to list all the networks associated with Docker on the host > docker network ls NETWORK ID NAME DRIVER 2ededf290eeb bridge bridge c8fec94ed780 docker-net bridge 58a207b3a0d2 host host 365568e1e2bf my-mongo-cluster bridge SCOPE local local local local

> sudo docker network inspect bridge >sudo docker run it ubuntu:latest /bin/bash > sudo docker network inspect bridge

Creating Your Own New Network > docker network create --driver bridge new_nw d0f4e81ce4d629e904f3780840c2c097ba669c8d9abb4e0f18ab700f59683429 > sudo docker run -it --network=new_nw ubuntu:latest /bin/bash > sudo docker network inspect new_nw [ { "Name": "new_nw", "Id": "d0f4e81ce4d629e904f3780840c2.",... "Config": [ { "Subnet": "172.19.0.0/16", "Gateway": "172.19.0.1" }

Connect a container to a network ( network) --network Connect a container to a network You can also choose the IP addresses for the container with --ip and --ip6 flags when you start the container on a user-defined network $ docker run -itd --network=my-net busybox $ docker run -itd --network=my-net --ip=10.10.9.75 busybox If you do not specify a different network, new containers are automatically connected to the default bridge network

docker inspect Return low-level information on Docker objects $ docker ps -a CONTAINER ID NAMES 77c6e841638e mongo1 IMAGE COMMAND mongo "docker-entrypoint..." 10 days ago $ docker inspect mongo1 CREATED STATUS PORTS Exited (0) 10 days ago

Useful Docker Commands docker start starts a container so it is running docker stop stops a running container docker restart stops and starts a container docker kill sends a SIGKILL to a running container docker rm remove one or more containers docker rmi remove one or more images docker exec run a command in a running container docker save save one or more images to a tar archive (streamed to STDOUT by default) docker commit create a new image from a container s changes

$ docker ps -a CONTAINER ID NAMES 77c6e841638e mongo-docker $ $ $ $ $ IMAGE COMMAND mongo "docker-entrypoint..." 10 days ago docker start 77c6e841638e docker start 77 docker start 77c docker start 77c6 docker start mongo-docker CREATED STATUS PORTS Exited (0) 10 days ago

> docker run --name ubuntu_bash --rm -i -t ubuntu bash root@4f47613a44e3:/# exit exit > > docker run --name ubuntu_bash -d --rm -i -t ubuntu bash 327f7e6b08d6bd1404ec88330df6af2638be3b85d055a66177b5a61b4ff04f75 > docker exec -it ubuntu_bash bash root@327f7e6b08d6:/# exit exit > docker exec -it ubuntu_bash ls bin dev home lib64 mnt proc run srv tmp var boot etc lib media opt root sbin sys usr > > docker save busybox > busybox.tar

Container Orchestration Docker Swarm is a native Docker container orchestrator that allows users to treat a group of Docker hosts as a single Docker Engine Kubernetes is an open-source platform for container deployment automation, scaling, and operations across clusters of hosts. The production ready orchestrator draws on Google s extensive experience of years of working with Linux containers

Node Package Manager (NPM) Install Node.js, npm, and stay up-to-date

What is npm? npm makes it easy for JavaScript developers to share and reuse code, and makes it easy to update the code that you re sharing, so you can build amazing things npm is installed with Node.js -- ubuntu > curl -sl https://deb.nodesource.com/setup_8.x sudo -E bash > sudo apt-get install -y nodejs > curl -sl https://deb.nodesource.com/setup_7.x sudo -E bash > sudo apt-get install -y nodejs

Installing NPM... root@73eee145286b:/# which node /usr/local/bin/node root@73eee145286b:/# node --version v8.2.1 root@73eee145286b:/# which npm /usr/local/bin/npm root@73eee145286b:/# npm --version 5.3.0

Other Nifty Commands $ npm init creates a package.json in root for you $ npm list lists all installed packages $ npm prune removes packages not depended on by your project according to your package.json $ npm outdated tells you which installed packages are outdated with respect to what is current in the npm registry but allowable by the version definition in your package.json

Installing Modules using NPM $ npm install <Module Name> example: $ npm install express installed express module, it created node_modules directory in the current directory where it installed the express module.

Installing Packages in Global Mode $ npm install express -g

Installing a Specific Version of a Package

Uninstalling Local Packages

package.json

package.json (2)

npm run... $ npm run test $ npm run start

The Dependencies property The dependencies property of a module's package.json is where dependencies - the other modules that this module uses - are defined. The dependencies property takes an object that has the name and version at which each dependency should be used

Version syntax ~version ^version version >version >=version <version <=version 1.2.x * latest Approximately equivalent to version Compatible with version matches the most recent Must match version exactly minor version (the middle Must be greater than version number). ~1.2.3 will match all 1.2.x versions but will miss 1.3.0. 1.2.0, 1.2.1, etc., but not 1.3.0 Matches any version Obtains latest release Matches the most recent major version (the first number). ^1.2.3 will match any 1.x.x release including 1.3.0, but will hold off on 2.0.0

The devdependencies property The devdependencies property of a package.json is almost identical to the dependencies property in terms of structure, with a key difference. The dependencies property is used to define the dependencies that a module needs to run in production. The devdependencies property is usually used to define the dependencies the module needs to run in development

--save vs --save-dev $ npm install [package_name] --save $ npm install [package_name] --save-dev --save-dev is used to save the package for development purpose. Example: unit tests, minification.. --save is used to save the package required for the application to run.

thanks for your attention any questions?