GCLOUD_DATAPROC_JOBS_SUBMIT_PYSPARK(1)

NAME

gcloud dataproc jobs submit pyspark - submit a PySpark job to a cluster

SYNOPSIS

gcloud dataproc jobs submit pyspark PY_FILE --cluster=CLUSTER [--archives=[ARCHIVE,...]] [--async] [--bucket=BUCKET] [--driver-log-levels=[PACKAGE=LEVEL,...]] [--files=[FILE,...]] [--jars=[JAR,...]] [--labels=[KEY=VALUE,...]] [--properties=[PROPERTY=VALUE,...]] [--py-files=[PY_FILE,...]] [GLOBAL-FLAG ...] [-- JOB_ARGS ...]

DESCRIPTION
Submit a PySpark job to a cluster.

POSITIONAL ARGUMENTS

    PY_FILE
        The main .py file to run as the driver.

    [-- JOB_ARGS ...]
        The arguments to pass to the driver. The '--' argument must be specified between gcloud specific args on the left and JOB_ARGS on the right.

REQUIRED FLAGS

    --cluster=CLUSTER
        The Dataproc cluster to submit the job to.

OPTIONAL FLAGS

    --archives=[ARCHIVE,...]
        Comma separated list of archives to be provided to the job. must be one of the following file formats: .zip, .tar, .tar.gz, or .tgz.

    --async
        Does not wait for the job to run.

    --bucket=BUCKET
        The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

    --driver-log-levels=[PACKAGE=LEVEL,...]
        A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO

    --files=[FILE,...]
        Comma separated list of files to be provided to the job.

    --jars=[JAR,...]
        Comma separated list of jar files to be provided to the executor and driver classpaths.

    --labels=[KEY=VALUE,...]
        A list of label KEY=VALUE pairs to add.

    --properties=[PROPERTY=VALUE,...]
        A list of key value pairs to configure PySpark.

    --py-files=[PY_FILE,...]
        Comma separated list of Python files to be provided to the job.Must be one of the following file formats" .py, ,.zip, or .egg

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a PySpark job with a local script, run:
    $ gcloud dataproc jobs submit pyspark --cluster my_cluster \
        my_script.py
To submit a Spark job that runs a script that is already on the cluster, run:
    $ gcloud dataproc jobs submit pyspark --cluster my_cluster \
        file:///usr/lib/spark/examples/src/main/python/pi.py 100