Lesson: Web Programming(3) Omid Jafarinezhad Sharif University of Technology

Transcription

1 Lesson: Web Programming(3) Omid Jafarinezhad Sharif University of Technology

2 Materials HTTP, JavaScript, CSS, HTML5, ReactJs, Flow, Progressive Web App Golang, NodeJs, MongoDB, PostgreSQL, Redis Docker, Git, NPM, YUIDoc, Jest, WebPack, Gulp, Browserify, Locust (Optional/Research) Kubernetes, InfluxDB, RabbitMQ, grpc, Ansible

3 Git Helps software developers to work together and maintain a complete history of their work

4 Git A version control system (VCS) allows you to track the history of a collection of files Git is a distributed revision control and source code management system with an emphasis on speed > sudo apt-get install git > sudo yum install git > git --version

5 Git - Life Cycle A Git repository contains the history of a collection of files starting from a certain directory Amend: After committing, if you realize something is wrong, then you correct the last commit and push the changes to the repository

6 Working tree The working tree contains the set of working files for the repository You can modify the content and commit the changes as new commits to the repository

7 Commit When you commit your changes into a repository this creates a new commit object in the Git repository. This commit object uniquely identifies a new revision of the content of the repository This revision can be retrieved later, for example, if you want to see the source code of an older version Each commit object contains the author and the committer. This makes it possible to identify who did the change. The author and committer might be different people. The author did the change and the committer applied the change to the Git repository. This is common for contributions to open source projects

8 Staging area The staging area is the place to store changes in the working tree before the commit The staging area contains a snapshot of the changes in the working tree (changed or new files) relevant to create the next commit and stores their mode (file type, executable bit)

9 The staging area (aka index) is a container where git collects all changes which will be part of the next commit. If you are editing a versioned file on your local machine, git recognizes that your file is modified - but it will not be automatically part of your next commit and is therefore unstaged. Staging the file will put the file into the staging area (index). The next git commit will transfer all items from staging area into your repository

10 Revision Represents a version of the source code Git implements revisions as commit objects (or short commits ). These are identified by an SHA-1 hash

11 Tag A tag points to a commit which uniquely identifies a version of the Git repository. With a tag, you can have a named point to which you can always revert to. You can revert to any point in a Git repository, but tags make it easier. The benefit of tags is to mark the repository for a specific reason, e.g., with a release. Branches and tags are named pointers, the difference is that branches move when a new commit is created while tags always point to the same commit. Tags can have a timestamp and a message associated with them

12

13 Branch Git supports branching which means that you can work on different versions of your collection of files, A branch allows the user to switch between these versions so that he can work on different changes independently from each other. One of the branches is the default (typically named _master ). The default branch is the one for which a local branch is automatically created when cloning the repository For example, if you want to develop a new feature, you can create a branch and make the changes in this branch. This does not affect the state of your files in other branches. For example, you can work independently on a branch called production for bugfixes and on another branch called feature_123 for implementing a new feature

14

15 Brach (2) A branch is a named pointer to a commit. Selecting a branch in Git terminology is called to checkout a branch. If you are working in a certain branch, the creation of a new commit advances this pointer to the newly created commit ([p:231]... branches move when a new commit is created...) Each commit knows their parents (predecessors). Successors are retrieved by traversing the commit graph starting from branches or other refs, symbolic references (for example: HEAD) or explicit commit objects. This way a branch defines its own line of descendants in the overall version graph formed by all commits in the repository

16 Example terminal prompt A working directory

17 To initialize a Git repository here

18 to see what the current state of our project

19 I created a file called octocat.txt in the octobox repository

20 To tell Git to start tracking changes made to octocat.txt, we first need to add it to the staging area by using git add

21 To store our staged changes we run the commit command with a message describing what we've changed

22 You also can use wildcards if you want to add many files of the same type I put some(.txt file) in a directory named "octofamily" and some others ended up in the root of our "octobox" directory

23 So we've made a few commits. Now let's browse them to see what we changed. Think of Git's log as a journal that remembers all the changes we've committed so far, in the order we committed them

24 To push our local repo to the GitHub server we'll need to add a remote repository The push command tells Git where to put our commits > Push our local changes to our origin repo

25 Let's pretend some time has passed. We've invited other people to our GitHub project who have pulled your changes, made their own commits, and pushed them. We can check for changes on our GitHub repository and pull down any new changes Fast-Forward

26 Compare HEAD by Pulled files :-) 1,4 range (line) - removed + added

27 You can unstage files by using the git reset command create a branch called clean_up You can switch branches using the git checkout <branch> command

28 Switching Back to master We're already on the master branch, we can merge our changes from the clean_up branch into the master branch

29 Basic Branching and Merging Example Let s go through a simple example of branching and merging with a workflow that you might use in the real world. You ll follow these steps: Do some work on a website Create a branch for a new story you re working on Do some work in that branch At this stage, you ll receive a call that another issue is critical and you need a hotfix. You ll do the following: Switch to your production branch Create a branch to add the hotfix After it s tested, merge the hotfix branch, and push to production Switch back to your original story and continue working

30 You ve decided that you re going to work on issue #53 in whatever issue-tracking system your company uses

31 Now you get the call that there is an issue with the website, and you need to fix it immediately. With Git, you don t have to deploy your fix along with the iss53 changes you ve made, and you don t have to put a lot of effort into reverting those changes before you can work on applying your fix to what is in production

32

33 You ll notice the phrase fast-forward in that merge. Because the commit C4 pointed to by the branch hotfix you merged in was directly ahead of the commit C2 you re on, Git simply moves the pointer forward. To phrase that another way, when you try to merge one commit with a commit that can be reached by following the first commit s history, Git simplifies things by moving the pointer forward because there is no divergent work to merge together this is called a fast-forward.

34 After your super-important fix is deployed, you re ready to switch back to the work you were doing before you were interrupted

35 Suppose you ve decided that your issue #53 work is complete and ready to be merged into your master branch. In order to do that, you ll merge your iss53 branch into master, much like you merged your hotfix branch earlier This looks a bit different than the hotfix merge you did earlier. In this case, your development history has diverged from some older point. Because the commit on the branch you re on isn t a direct ancestor of the branch you re merging in, Git has to do some work. In this case, Git does a simple three-way merge, using the two snapshots pointed to by the branch tips and the common ancestor of the two. Instead of just moving the branch pointer forward, Git creates a new snapshot that results from this three-way merge and automatically creates a new commit that points to it. This is referred to as a merge commit, and is special in that it has more than one parent It s worth pointing out that Git determines the best common ancestor to use for its merge base

36

37 Now that your work is merged in, you have no further need for the iss53 branch. You can close the ticket in your ticket-tracking system, and delete the branch Occasionally, this process doesn t go smoothly. If you changed the same part of the same file differently in the two branches you re merging together, Git won t be able to merge them cleanly. If your fix for issue #53 modified the same part of a file as the hotfix branch, you ll get a merge conflict that looks something like this

38 Your file contains a section that looks something like this This means the version in HEAD (your master branch, because that was what you had checked out when you ran your merge command) is the top part of that block (everything above the =======), while the version in your iss53 branch looks like everything in the bottom part. In order to resolve the conflict, you have to either choose one side or the other or merge the contents yourself

39 Cherry-pick Problem is found on the Production branch. A fix for the problem is developed in commit H, but commit G does not need to be applied to the Development branch Commit H is cherry-picked onto the Development branch, resulting in commit H'. Note that the changes made in commit G are not included on the Development branch

40 Reset On the commit-level, resetting is a way to move the tip of a branch to a different commit This can be used to remove commits from the current branch (a simple way to undo changes)

41 > git reset HEAD~~ HEAD is now at 326c9f commit message > git reset HEAD~2 HEAD is now at 326c9f commit message

42 Revert Reverting undoes a commit by creating a new commit. This is a safe way to undo changes, as it has no chance of re-writing the commit history

43 > git revert HEAD~2

44 Rebase Rebasing is the process of moving or combining a sequence of commits to a new base commit. Rebasing is most useful and easily visualized in the context of a feature branching workflow The primary reason for rebasing is to maintain a linear project history For example, consider a situation where the master branch has progressed since you started working on a feature branch. You want to get the latest updates to the master branch in your feature branch, but you want to keep your branch's history clean so it appears as if you've been working off the latest master branch. This gives the later benefit of a clean merge of your feature branch back into the master branch.

45

46 Docker a helpful tool for packing, shipping, and running applications within containers

47

48

49 What are containers and VMs? Containers and VMs are similar in their goals: to isolate an application and its dependencies into a self-contained unit that can run anywhere remove the need for physical hardware, allowing for more efficient use of computing resources, both in terms of energy consumption and cost effectiveness The main difference between containers and VMs is in their architectural approach Docker share the host system s kernel with other containers

50 Virtual machines run guest operating systems note the OS layer in each box. This is resource intensive, and the resulting disk image and application state is an entanglement of OS settings, system-installed dependencies, OS security patches, and other easy-to-lose, hard-to-replicate ephemera guest machine host machine Containers can share a single kernel, and the only information that needs to be in a container image is the executable(app A, B, C) and its package dependencies (Bin/Lib), which never need to be installed on the host system. Because they contain all their dependencies, there is no configuration entanglement; a containerized app runs anywhere

51 VM New Setting and Install Start Docker Pull Run (Stop/start)

52

53

54 Where does Docker come in? Docker is an open-source project based on Linux containers. It uses Linux Kernel features like namespaces and control groups to create containers on top of an operating system Containers are far from new; Google has been using their own container technology for years. Others Linux container technologies include Solaris Zones, BSD jails, and LXC, which have been around for many years

55 So why is Docker all of a sudden gaining steam? Ease of use: Docker has made it much easier for anyone developers, systems admins, architects and others to take advantage of containers in order to quickly build and test portable applications Speed: Docker containers are very lightweight and fast Docker Hub: Docker users also benefit from the increasingly rich ecosystem of Docker Hub, which you can think of as an app store for Docker images Modularity and Scalability: Docker makes it easy to break out your application s functionality into individual containers

56 Install Docker (CE/EE) Docker is available in two editions: Community Edition (CE) and Enterprise Edition (EE) > curl -fssl sudo apt-key add > sudo add-apt-repository "deb [arch=amd64] $(lsb_release -cs) stable" > sudo apt-get update > sudo apt-get install -y docker-ce > sudo systemctl status docker docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Sun :53:52 CDT; 1 weeks 3 days ago Docs: Main PID: 749 (docker) -- Ubuntu 16.04

57 Docker Command Without Sudo (Optional, Dev) By default, running the docker command requires root privileges that is, you have to prefix the command with sudo. It can also be run by a user in the docker group, which is automatically created during the installation of Docker. If you attempt to run the docker command without prefixing it with sudo or without being in the docker group, you'll get an output like this If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. See 'docker run --help'. > sudo usermod -ag docker ${USER} > su - ${USER} Add your username to the docker group apply the new group membership(without logout)

58 A brief explanation of containers An image is a lightweight, stand-alone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files A container is a runtime instance of an image what the image becomes in memory when actually executed. It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so Containers run apps natively on the host machine s kernel. Containers can get native access, each one running in a discrete process, taking no more memory than any other executable

59 Docker image repositories

60 Pull an image from Docker Hub > docker pull debian Using default tag: latest latest: Pulling from library/debian fdd5d7827f33: Pull complete a3ed95caeb02: Pull complete Digest: sha256:e7d38b a1c71e41bffe9c8ae6d6d29546ce46bf aad072c90aa Status: Downloaded newer image for debian:latest

61 Pull an image from Docker Hub (2) > docker pull ubuntu: : Pulling from library/ubuntu 5a132a7e7af1: Pull complete fd2731e4c50c: Pull complete 28a2f68d1120: Pull complete a3ed95caeb02: Pull complete Digest: sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f c5cb2 Status: Downloaded newer image for ubuntu:14.04

62 Displaying Docker Images To see the list of Docker images on the system, you can issue the following command > docker images ubuntu-java latest ubuntu latest consul latest mongo latest influxdb latest postgres latest node latest rabbitmq latest nginx latest a19 3 weeks ago 889MB 8b72bba4485f 3 weeks ago 120MB c6f7042bd0f8 2 months ago 51.7MB b39de1d79a53 2 months ago 359MB d913baeaf36b 2 months ago 227MB 33b13ed6b80a 2 months ago 269MB 045ea30913bb 2 months ago 667MB bad16bdb4e74 2 months ago 124MB b8efb18f159b 2 months ago 107MB

63 Images and layers A Docker image is built up from a series of layers Each layer represents an instruction in the image s Dockerfile When you use docker pull to pull down an image from a repository, each layer is pulled down separately, and stored in Docker s local storage area, which is usually /var/lib/docker/ on Linux hosts. You can see these layers being pulled in this example:

64 > docker pull debian Using default tag: latest latest: Pulling from library/debian fdd5d7827f33: Pull complete a3ed95caeb02: Pull complete Digest: sha256:e7d38b a1c71e41bffe9c8ae6d6d29546ce46bf aad072c90aa Status: Downloaded newer image for debian:latest

65 Docker run Running of containers is managed with the Docker run command

66 Assign name and allocate pseudo-tty ( name, -it) --interactive, -i --name --tty, -t Keep STDIN open even if not attached Assign a name to the container Allocate a pseudo-tty (pseudoterminal) > docker run --name test -it debian root@d6c0fe130dba:/# exit > docker ps -a grep test d6c0fe130dba debian:7 "/bin/bash" This example runs a container named test using the debian:latest image. The -it instructs Docker to allocate a pseudo-tty connected to the container s stdin; creating an interactive bash shell in the container 26 seconds ago Exited (13) 17 seconds ago test

67 Listing of Containers One can list all of the containers on the machine via the docker ps command. This command is used to return the currently running containers > docker ps -a This command is used to list all of the containers on the system

68 Run in background and remove if exits --detach, -d --rm Run container in background and print container ID Automatically remove the container when it exits >docker run --name ubuntu_bash --rm -i -t ubuntu bash exit exit > >docker run --name ubuntu_bash -d --rm -i -t ubuntu bash 327f7e6b08d6bd1404ec88330df6af2638be3b85d055a66177b5a61b4ff04f75 >

69 Publish or expose port (-p, expose) --publish, -p Publish a container s port(s) to the host $ docker run -p :80:8080 ubuntu bash This binds port 8080 of the container to port 80 on of the host machine

70 Set environment variables (-e, env, env-file) --env, -e Set environment variables --env-file Read in a file of environment variables Use the -e, --env, and --env-file flags to set simple (non-array) environment variables in the container you re running, or overwrite variables that are defined in the Dockerfile of the image you re running. $ docker run --env MYVAR2=foo --env-file./env.list ubuntu bash

71 Set metadata on container (-l, label, label-file) --label, -l Set meta data on a container --label-file Read in a line delimited file of labels A label is a key=value pair that applies metadata to a container $ docker run -l my-label --label com.example.foo=bar ubuntu bash The my-label key doesn t specify a value so the label defaults to an empty string(""). To add multiple labels, repeat the label flag (-l or --label)

72 Set ulimits in container ( ulimit) $ docker run --ulimit nofile=1024:1024 debian sh -c "ulimit -n" 1024 <type>=<soft limit>[:<hard limit>] A hard limit is the maximum allowed to a user/process;increasing hard limit can be done only by root A soft limit is the effective value right now for that user/process; a soft limit may be increased up to the value of the hard limit $ docker run -ti node /bin/bash user@4d04d06d5022:/# ulimit -a... max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) cpu time (seconds, -t) unlimited max user processes (-u)

73 Mount volume --volume, -v Bind mount a volume $ docker run -v /data:/path-in-container/data -i -t ubuntu bash

74 Memory Constraints $ docker run -it -m 300M ubuntu:16.04 /bin/bash

75 CPU Constraints You can limit CPU, either using a percentage of all CPUs, or by using specific cores. The setting is a bit strange means 100% of the CPU, so if you want the container to take 50% of all CPU cores, you should specify 512 You can also only use some CPU cores using cpuset-cpus $ docker run -ti --c 512 ubuntu:16.04 /bin/bash $ docker run -ti --cpuset-cpus=0,4,6 ubuntu:16.04 /bin/bash

76 Listing All Docker Networks > docker network ls This command can be used to list all the networks associated with Docker on the host > docker network ls NETWORK ID NAME DRIVER 2ededf290eeb bridge bridge c8fec94ed780 docker-net bridge 58a207b3a0d2 host host e1e2bf my-mongo-cluster bridge SCOPE local local local local

77 > sudo docker network inspect bridge >sudo docker run it ubuntu:latest /bin/bash > sudo docker network inspect bridge

78 Creating Your Own New Network > docker network create --driver bridge new_nw d0f4e81ce4d629e904f c2c097ba669c8d9abb4e0f18ab700f > sudo docker run -it --network=new_nw ubuntu:latest /bin/bash > sudo docker network inspect new_nw [ { "Name": "new_nw", "Id": "d0f4e81ce4d629e904f c2.",... "Config": [ { "Subnet": " /16", "Gateway": " " }

79 Connect a container to a network ( network) --network Connect a container to a network You can also choose the IP addresses for the container with --ip and --ip6 flags when you start the container on a user-defined network $ docker run -itd --network=my-net busybox $ docker run -itd --network=my-net --ip= busybox If you do not specify a different network, new containers are automatically connected to the default bridge network

80 docker inspect Return low-level information on Docker objects $ docker ps -a CONTAINER ID NAMES 77c6e841638e mongo1 IMAGE COMMAND mongo "docker-entrypoint..." 10 days ago $ docker inspect mongo1 CREATED STATUS PORTS Exited (0) 10 days ago

81 Useful Docker Commands docker start starts a container so it is running docker stop stops a running container docker restart stops and starts a container docker kill sends a SIGKILL to a running container docker rm remove one or more containers docker rmi remove one or more images docker exec run a command in a running container docker save save one or more images to a tar archive (streamed to STDOUT by default) docker commit create a new image from a container s changes

82 $ docker ps -a CONTAINER ID NAMES 77c6e841638e mongo-docker $ $ $ $ $ IMAGE COMMAND mongo "docker-entrypoint..." 10 days ago docker start 77c6e841638e docker start 77 docker start 77c docker start 77c6 docker start mongo-docker CREATED STATUS PORTS Exited (0) 10 days ago

83

84

85 > docker run --name ubuntu_bash --rm -i -t ubuntu bash exit exit > > docker run --name ubuntu_bash -d --rm -i -t ubuntu bash 327f7e6b08d6bd1404ec88330df6af2638be3b85d055a66177b5a61b4ff04f75 > docker exec -it ubuntu_bash bash exit exit > docker exec -it ubuntu_bash ls bin dev home lib64 mnt proc run srv tmp var boot etc lib media opt root sbin sys usr > > docker save busybox > busybox.tar

86 Container Orchestration Docker Swarm is a native Docker container orchestrator that allows users to treat a group of Docker hosts as a single Docker Engine Kubernetes is an open-source platform for container deployment automation, scaling, and operations across clusters of hosts. The production ready orchestrator draws on Google s extensive experience of years of working with Linux containers

87 Node Package Manager (NPM) Install Node.js, npm, and stay up-to-date

88 What is npm? npm makes it easy for JavaScript developers to share and reuse code, and makes it easy to update the code that you re sharing, so you can build amazing things npm is installed with Node.js -- ubuntu > curl -sl sudo -E bash > sudo apt-get install -y nodejs > curl -sl sudo -E bash > sudo apt-get install -y nodejs

89 Installing NPM... which node /usr/local/bin/node node --version v8.2.1 which npm /usr/local/bin/npm npm --version 5.3.0

90 Other Nifty Commands $ npm init creates a package.json in root for you $ npm list lists all installed packages $ npm prune removes packages not depended on by your project according to your package.json $ npm outdated tells you which installed packages are outdated with respect to what is current in the npm registry but allowable by the version definition in your package.json

91

92 Installing Modules using NPM $ npm install <Module Name> example: $ npm install express installed express module, it created node_modules directory in the current directory where it installed the express module.

93

94 Installing Packages in Global Mode $ npm install express -g

95 Installing a Specific Version of a Package

96 Uninstalling Local Packages

97 package.json

98 package.json (2)

99 npm run... $ npm run test $ npm run start

100 The Dependencies property The dependencies property of a module's package.json is where dependencies - the other modules that this module uses - are defined. The dependencies property takes an object that has the name and version at which each dependency should be used

101 Version syntax ~version ^version version >version >=version <version <=version 1.2.x * latest Approximately equivalent to version Compatible with version matches the most recent Must match version exactly minor version (the middle Must be greater than version number). ~1.2.3 will match all 1.2.x versions but will miss , 1.2.1, etc., but not Matches any version Obtains latest release Matches the most recent major version (the first number). ^1.2.3 will match any 1.x.x release including 1.3.0, but will hold off on 2.0.0

102 The devdependencies property The devdependencies property of a package.json is almost identical to the dependencies property in terms of structure, with a key difference. The dependencies property is used to define the dependencies that a module needs to run in production. The devdependencies property is usually used to define the dependencies the module needs to run in development

103 --save vs --save-dev $ npm install [package_name] --save $ npm install [package_name] --save-dev --save-dev is used to save the package for development purpose. Example: unit tests, minification.. --save is used to save the package required for the application to run.

104 thanks for your attention any questions?