TensorFlow on vivo

Size: px

Start display at page:

Download "TensorFlow on vivo"

Clifford Ellis
5 years ago
Views:

1 TensorFlow on vivo xidianwangtao@gmail.com

2 Agenda Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

3 Outrageously large models Improving accuracy with up to 68 billion parameters

4 Distributed TensorFlow Derek TensorFlow DEV SUMMIT 2017 Distributed TensorFlow h"ps:// 3me_con3nue=703&v=la_M6bCV91M

7 Distributed TensorFlow Model

8 In-graph Replication

9 Between-graph Replication

10 Async/Sync Training

11 Between-graph + Async

12 Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

13 Motivation TensorFlow Task GPU GPU Task Task HDFS Read TensorFlow

14 Kubernetes is Suitable ResourceQuota, LimitRanger GPU (only limits) PLEG EFK Read Glusterfs, Ceph) TensorFlow

15 HDFS vs Glusterfs vs Ceph Glusterfs 12GB/s HDFS 3GB/s CephFS 2GB/s GlusterFS Read Performance is Best

16 Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

17 GlusterFS + K8S + TF

18 HDFS + K8S + TF GCEPersistentDisk AWSElasticBlockStor CephFS e Cinder AzureFile Glusterfs AzureDisk VsphereVolume FC (Fibre Channel) Quobyte Volumes FlexVolume HostPath Flocker VMware Photon NFS Portworx Volumes iscsi ScaleIO Volumes RBD StorageOS

19 Kube-scheduler Kube-apiserver etcd Job JobController NewJobController Kube-controller-manager Run.spec.completions.spec.parallelism.spec.activeDeadlineSeconds.spec.template.spec.backoffLimit RestartPolicy: Never or OnFailure syncjob func (jm *JobController) syncjob(key string) (bool, error) managejob func (jm *JobController) managejob(activepods []*v1.pod, succeeded int32, job *batch.job) (int32, error) Indexer SatisfiedExpectations

20 Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

21 Components

23 Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

24 Step 1- User Node $AlgorithmName copy User Node `/ var/www/html/$username/$algorithmname/` run.sh ( $AlgorithmName ) $AlgorithmName : User Node `/var/www/html/` httpd HTTP `

25 Step 2- User Node `/opt/tensorflow/` tfcluster_template.yaml.jinja HDFS GlusterFS

30 Tensorboard

31 Config tfcluster_template.yaml.jinja, name, worker_replicas, ps_replicas, script script Http

32 Step 3- TensorFlow Cluster `python render_template.py tfcluster_template.yaml.jinja > wangtao.yaml` k8s yaml `kubectl apply -f wangtao.yaml` Between-Graph TensorFlow Cluster

33 TensorFlow Cluster Kubernetes Dashboard namespace ps worker

34 PS log

35 Worker log

36 PS and Worker Service

37 Secret for Harbor

38 Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

39 Worker recreate pod kubernetes 1.7: ü kubelet: --maximum-dead-containers; ü Job Yaml:.spec.activeDeadlineSeconds; kubernetes 1.8: ü kubelet: --maximum-dead-containers; ü Job Yaml:.spec.activeDeadlineSeconds; ü Job Yaml:.spec.template.spec.backoffLimit (default 6)

40 Command or args? No such file or directory args shell command shell

For such Services, a cluster IP is not allocated, kube-proxy does not handle these services, and there is no load

41 Headless Service is Suitable Sometimes you dont need or want load-balancing and a single service IP. In this case, you can create headless services by specifying "None" for the cluster IP (spec.clusterip). For such Services, a cluster IP is not allocated, kube-proxy does not handle these services, and there is no load balancing or proxying done by the platform for them. How DNS is automatically configured depends on whether the service has selectors defined.

42 kubelet --cluster-dns= cluster-domain=tensorflow.pro.vivo

43 Take Care Dockerfile ENV In Dockerfile: ENV CLASSPATH.:/usr/lib/jvm/java-1.8.0/lib/tools.jar:$(/usr/lib/hadoop-2.6.1/bin/hadoop classpath glob) env CLASSPATH=.:/usr/lib/jvm/java-1.8.0/lib/tools.jar:$(/usr/lib/hadoop-2.6.1/bin/hadoop classpath glob)

44 Walkaround Pod.spec.containers.command: [ /bin/sh, -c, export CLASSPATH=.:/usr/lib/jvm/java-1.8.0/lib/tools.jar:$(/usr/lib/hadoop-2.6.1/bin/hadoop classpath --glob); ]

45 Ps Process Hang

46 Shared Queue? Maybe ü TensorFlow on Kubernetes namespace namespace worker job successed kill hang PS Deployment delete namespace ü DevOps/TaaS Events successed worker Kubernetes

47 Reuse PV? When you delete a PVC, corresponding PV becomes Released. Make the PV available to everybody - delete PV.Spec.ClaimRef, Such PV can bound to any PVC (assuming that capacity, access mode and selectors match) Make the PV available to a specific PVC - pre-fill PV.Spec.ClaimRef with a pointer to a PVC. Leave the PV.Spec.ClaimRef,UID empty, as the PVC does not to need exist at this point and you don't know PVC's UID. This PV can be bound only to the specified PVC.

50 global secret 1. secret: kubectl create secret docker-registry harborsecret docker-server=registry.vivo.xyz: 4443 docker-username=admin docker-password=harbor Kubectl get secret harborsecret -o yaml `.dockercfg`

51 3. tfcluster_template.yaml.jinja namespace harbor secret yaml, `.dockercfg`.dockercfg value image

52 Thinking worker PS? chief task restart.

53 Distributed TensorFlow Why TensorFlow on Kubernetes How TensorFlow on Kubernetes Deploy Architecture Step By Step The Major Problems I Have Encountered Todo List

54 Todo List NVIDIA GPU GPU IO? TensorFlow GPU TaaS TensorFlow ü ü TensorFlow Cluster ü ü TenserFlow Serving K8S Jupyter Notebook, Tensorboard

55 Q&A Thank you for your time. CSDN, OSCHINA

agenda PAE Docker Docker PAE

Docker 2016.03.26 agenda PAE Docker Docker PAE 2 3 PAE PlCloud APP Engine Docker Docker Caas APP 4 APP APP volume images 5 App 6 APP Show Time 7 8 Docker Public DockerHup Private registry push pull AUFS