Package cloudml. June 12, 2018

Transcription

1 Package cloudml June 12, 2018 Title Interface to the Google Cloud Machine Learning Platform Version Interface to the Google Cloud Machine Learning Platform < which provides cloud tools for training machine learning models. Depends R (>= 3.3.0), tfruns (>= 1.3) Imports jsonlite, packrat, processx, rprojroot, rstudioapi, tools, utils, withr, yaml Suggests tensorflow (>= 1.4.2), keras (>= 2.1.2), knitr, testthat License Apache License 2.0 SystemRequirements Python (>= 2.7.0) Encoding UTF-8 LazyData true RoxygenNote VignetteBuilder knitr NeedsCompilation no Author Javier Luraschi [aut, cre], JJ Allaire [aut], Kevin Ushey [aut], RStudio [cph] Maintainer Javier Luraschi <javier@rstudio.com> Repository CRAN Date/Publication :05:22 UTC R topics documented: cloudml_deploy cloudml_predict cloudml_train gcloud_init

2 2 cloudml_deploy gcloud_install gcloud_terminal gs_copy gs_local_dir gs_rsync job_cancel job_collect job_list job_status job_stream_logs job_trials Index 12 cloudml_deploy Deploy SavedModel to CloudML Deploys a SavedModel to CloudML model for online predictions. cloudml_deploy(export_dir_base, name, version = paste0(name, "_1"), region = NULL, config = NULL) export_dir_base A string containing a directory containing an exported SavedModels. Consider using tensorflow::export_savedmodel() to export this SavedModel. name version region config The name for this model (required) The version for this model. Versions start with a letter and contain only letters, numbers and underscores. Defaults to name_1 The region to be used to deploy this model. A list, YAML or JSON configuration file as described com/ml-engine/reference/rest/v1/projects.jobs. cloudml_predict() Other CloudML functions: cloudml_predict, cloudml_train

3 cloudml_predict 3 cloudml_predict Perform Prediction over a CloudML Model. Perform online prediction over a CloudML model, usually, created using cloudml_deploy() cloudml_predict(instances, name, version = paste0(name, "_1"), verbose = FALSE) instances name version verbose A list of instances to be predicted. While predicting a single instance, list wrapping this single instance is still expected. The name for this model (required) The version for this model. Versions start with a letter and contain only letters, numbers and underscores. Defaults to name_1 Should additional information be reported? cloudml_deploy() Other CloudML functions: cloudml_deploy, cloudml_train cloudml_train Train a model using Cloud ML Upload a TensorFlow application to Google Cloud, and use that application to train a model. cloudml_train(file = "train.r", master_type = NULL, flags = NULL, region = NULL, config = NULL, collect = "ask", dry_run = FALSE)

4 4 gcloud_init file master_type flags region config collect dry_run File to be used as entrypoint for training. Training master node machine type. "standard" provides a basic machine configuration suitable for training simple models with small to moderate datasets. See the documentation at machine-types#machine_type_table for details on available machine types. Named list with flag values (see flags()) or path to YAML file containing flag values. The region to be used for training. A list, YAML or JSON configuration file as described com/ml-engine/reference/rest/v1/projects.jobs. Logical. If TRUE, collect job when training is completed (blocks waiting for the job to complete). The default ("ask") will interactively prompt the user whether to collect the results or not. Triggers a local dry run over the deployment phase to validate packages and packing work as expected. job_status(), job_collect(), job_cancel() Other CloudML functions: cloudml_deploy, cloudml_predict Examples ## Not run: library(cloudml) gcloud_install() job <- cloudml_train("train.r") ## End(Not run) gcloud_init Initialize the Google Cloud SDK Initialize the Google Cloud SDK gcloud_init() Other Google Cloud SDK functions: gcloud_install, gcloud_terminal

5 gcloud_install 5 gcloud_install Install the Google Cloud SDK Installs the Google Cloud SDK which enables CloudML operations. gcloud_install(update = TRUE) update Attempt to update an existing installation. Other Google Cloud SDK functions: gcloud_init, gcloud_terminal Examples ## Not run: library(cloudml) gcloud_install() ## End(Not run) gcloud_terminal Create an RStudio terminal with access to the Google Cloud SDK Create an RStudio terminal with access to the Google Cloud SDK gcloud_terminal(command = NULL, clear = FALSE) command clear Command to send to terminal Clear terminal buffer Value Terminal id (invisibly)

6 6 gs_local_dir Other Google Cloud SDK functions: gcloud_init, gcloud_install gs_copy Copy files to / from Google Storage Use the gsutil cp command to copy data between your local file system and the cloud, copy data within the cloud, and copy data between cloud storage providers. gs_copy(source, destination, recursive = FALSE, echo = TRUE) source destination recursive echo The file to be copied. This can be either a path on the local filesystem, or a Google Storage URI (e.g. gs://[bucket_name]/[filename.csv]). The location where the source file should be copied to. This can be either a path on the local filesystem, or a Google Storage URI (e.g. gs://[bucket_name]/[filename.csv]). Boolean; perform a recursive copy? This must be specified if you intend on copying directories. Echo command output to console. gs_local_dir Get a local path to the contents of Google Storage bucket Provides a local filesystem interface to Google Storage buckets. Many package functions accept only local filesystem paths as input (rather than gs:// URLs). For these cases the gcloud_path() function will synchronize gs:// buckets to the local filesystem and provide a local path interface to their contents. gs_local_dir(url, local_dir = "gs", echo = FALSE) url local_dir echo Google Storage bucket URL (e.g. gs://<your-bucket>). Local directory to synchonize Google Storage bucket(s) to. Echo command output to console.

7 gs_rsync 7 Details Value If you pass a local path as the url it will be returned unmodified. This allows you to for example use a training flag for the location of data which points to a local directory during development and a Google Cloud bucket during cloud training. Local path to contents of bucket. Examples library(cloudml) gs_local_dir(getwd()) gs_rsync Synchronize content of two buckets/directories The gs_rsync function makes the contents under destination the same as the contents under source, by copying any missing files/objects (or those whose data has changed), and (if the delete option is specified) deleting any extra files/objects. source must specify a directory, bucket, or bucket subdirectory. gs_rsync(source, destination, delete = FALSE, recursive = FALSE, parallel = TRUE, dry_run = FALSE, options = NULL, echo = TRUE) source destination delete recursive parallel The file to be copied. This can be either a path on the local filesystem, or a Google Storage URI (e.g. gs://[bucket_name]/[filename.csv]). The location where the source file should be copied to. This can be either a path on the local filesystem, or a Google Storage URI (e.g. gs://[bucket_name]/[filename.csv]). Delete extra files under destination not found under source By default extra files are not deleted. Causes directories, buckets, and bucket subdirectories to be synchronized recursively. If you neglect to use this option gs_rsync() will make only the top-level directory in the source and destination URLs match, skipping any subdirectories. Causes synchronization to run in parallel. This can significantly improve performance if you are performing operations on a large number of files over a reasonably fast network connection.

8 8 job_collect dry_run options echo Causes rsync to run in "dry run" mode, i.e., just outputting what would be copied or deleted without actually doing any copying/deleting. Character vector of additional command line options to the gsutil rsync command (as specified at commands/rsync). Echo command output to console. job_cancel Cancel a job Cancel a job. job_cancel(job = "latest") job Job name or job object. Pass "latest" to indicate the most recently submitted job. Other job management functions: job_collect, job_list, job_status, job_stream_logs, job_trials job_collect Collect job output Collect the job outputs (e.g. fitted model) from a job. If the job has not yet finished running, job_collect() will block and wait until the job has finished. job_collect(job = "latest", trials = "best", destination = "runs", timeout = NULL, view = interactive())

9 job_list 9 job trials destination timeout view Job name or job object. Pass "latest" to indicate the most recently submitted job. Under hyperparameter tuning, specifies which trials to download. Use "best" to download best trial, "all" to download all, or a vector of trials c(1,2) or 1. The destination directory in which model outputs should be downloaded. Defaults to runs. Give up collecting job after the specified minutes. View the job results after collecting it. You can also pass "save" to save a copy of the run report at tfruns.d/view.html Other job management functions: job_cancel, job_list, job_status, job_stream_logs, job_trials job_list List all jobs List existing Google Cloud ML jobs. job_list(filter = NULL, limit = NULL, page_size = NULL, sort_by = NULL, uri = FALSE) filter limit page_size sort_by uri Filter the set of jobs to be returned. The maximum number of resources to list. By default, all jobs will be listed. Some services group resource list output into pages. This flag specifies the maximum number of resources per page. The default is determined by the service if it supports paging, otherwise it is unlimited (no paging). A comma-separated list of resource field key names to sort by. The default order is ascending. Prefix a field with ~ for descending order on that field. Print a list of resource URIs instead of the default output. Other job management functions: job_cancel, job_collect, job_status, job_stream_logs, job_trials

10 10 job_stream_logs job_status Current status of a job Get the status of a job, as an R list. job_status(job = "latest") job Job name or job object. Pass "latest" to indicate the most recently submitted job. Other job management functions: job_cancel, job_collect, job_list, job_stream_logs, job_trials job_stream_logs Show job log stream Show logs from a running Cloud ML Engine job. job_stream_logs(job = "latest", polling_interval = getoption("cloudml.stream_logs.polling", 5), task_name = NULL, allow_multiline_logs = FALSE) job Job name or job object. Pass "latest" to indicate the most recently submitted job. polling_interval Number of seconds to wait between efforts to fetch the latest log messages. task_name If set, display only the logs for this particular task. allow_multiline_logs Output multiline log messages as single records. Other job management functions: job_cancel, job_collect, job_list, job_status, job_trials

11 job_trials 11 job_trials Current trials of a job Get the hyperparameter trials for job, as an R data frame job_trials(x) x Job name or job object. Other job management functions: job_cancel, job_collect, job_list, job_status, job_stream_logs

12 Index cloudml_deploy, 2, 3, 4 cloudml_deploy(), 3 cloudml_predict, 2, 3, 4 cloudml_predict(), 2 cloudml_train, 2, 3, 3 flags(), 4 gcloud_init, 4, 5, 6 gcloud_install, 4, 5, 6 gcloud_terminal, 4, 5, 5 gs_copy, 6 gs_local_dir, 6 gs_rsync, 7 job_cancel, 8, 9 11 job_cancel(), 4 job_collect, 8, 8, 9 11 job_collect(), 4 job_list, 8, 9, 9, 10, 11 job_status, 8 10, 10, 11 job_status(), 4 job_stream_logs, 8 10, 10, 11 job_trials, 8 10, 11 tensorflow::export_savedmodel(), 2 12