GCLOUD_DATAPROC_JOBS_SUBMIT_HADOOP(1)

NAME

gcloud dataproc jobs submit hadoop - submit a Hadoop job to a cluster

SYNOPSIS

gcloud dataproc jobs submit hadoop --cluster=CLUSTER [--archives=[ARCHIVE,...]] [--async] [--bucket=BUCKET] [--driver-log-levels=[PACKAGE=LEVEL,...]] [--files=[FILE,...]] [--jars=[JAR,...]] [--labels=[KEY=VALUE,...]] [--properties=[PROPERTY=VALUE,...]] [--class=MAIN_CLASS] [--jar=MAIN_JAR] [GLOBAL-FLAG ...] [-- JOB_ARGS ...]

DESCRIPTION
Submit a Hadoop job to a cluster.

POSITIONAL ARGUMENTS

    [-- JOB_ARGS ...]
        The arguments to pass to the driver. The '--' argument must be specified between gcloud specific args on the left and JOB_ARGS on the right.

REQUIRED FLAGS

    --cluster=CLUSTER
        The Dataproc cluster to submit the job to.

OPTIONAL FLAGS

    --archives=[ARCHIVE,...]
        Comma separated list of archives to be provided to the job. must be one of the following file formats: .zip, .tar, .tar.gz, or .tgz.

    --async
        Does not wait for the job to run.

    --bucket=BUCKET
        The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

    --driver-log-levels=[PACKAGE=LEVEL,...]
        A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO

    --files=[FILE,...]
        Comma separated list of files to be provided to the job.

    --jars=[JAR,...]
        Comma separated list of jar files to be provided to the MR and driver classpaths.

    --labels=[KEY=VALUE,...]
        A list of label KEY=VALUE pairs to add.

    --properties=[PROPERTY=VALUE,...]
        A list of key value pairs to configure Hadoop.

    --class=MAIN_CLASS
        The class containing the main method of the driver. Must be in a provided jar or jar that is already on the classpath

    --jar=MAIN_JAR
        The HCFS URI of jar file containing the driver jar.

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a Hadoop job that runs the main class of a jar, run:
    $ gcloud dataproc jobs submit hadoop --cluster my_cluster \
         --jar my_jar.jar arg1 arg2
To submit a Hadoop job that runs a specific class of a jar, run:
    $ gcloud dataproc jobs submit hadoop --cluster my_cluster \
        --class org.my.main.Class --jars my_jar1.jar,my_jar2.jar arg1 \
        arg2
To submit a Hadoop job that runs a jar that is already on the cluster, run:
    $ gcloud dataproc jobs submit hadoop --cluster my_cluster \
        --jar file:///usr/lib/hadoop-op/hadoop-op-examples.jar \
        wordcount gs://my_bucket/my_file.txt gs://my_bucket/output