GCLOUD_DATAPROC_JOBS_SUBMIT_SPARK-SQL(1)

NAME

gcloud dataproc jobs submit spark-sql - submit a Spark SQL job to a cluster

SYNOPSIS

gcloud dataproc jobs submit spark-sql --cluster=CLUSTER (--execute=QUERY, -e QUERY | --file=FILE, -f FILE) [--async] [--bucket=BUCKET] [--driver-log-levels=[PACKAGE=LEVEL,...]] [--jars=[JAR,...]] [--labels=[KEY=VALUE,...]] [--params=[PARAM=VALUE,...]] [--properties=[PROPERTY=VALUE,...]] [GLOBAL-FLAG ...]

DESCRIPTION
Submit a Spark SQL job to a cluster.

REQUIRED FLAGS

    --cluster=CLUSTER
        The Dataproc cluster to submit the job to.

Exactly one of these must be specified:

    --execute=QUERY, -e QUERY
        A Spark SQL query to execute as part of the job.

    --file=FILE, -f FILE
        HCFS URI of file containing Spark SQL script to execute as the job.

OPTIONAL FLAGS

    --async
        Does not wait for the job to run.

    --bucket=BUCKET
        The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

    --driver-log-levels=[PACKAGE=LEVEL,...]
        A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO

    --jars=[JAR,...]
        Comma separated list of jar files to be provided to the executor and driver classpaths. May contain UDFs.

    --labels=[KEY=VALUE,...]
        A list of label KEY=VALUE pairs to add.

    --params=[PARAM=VALUE,...]
        A list of key value pairs to set variables in the Hive queries.

    --properties=[PROPERTY=VALUE,...]
        A list of key value pairs to configure Hive.

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a Spark SQL job with a local script, run:
    $ gcloud dataproc jobs submit spark-sql --cluster my_cluster \
        --file my_queries.ql
To submit a Spark SQL job with inline queries, run:
    $ gcloud dataproc jobs submit spark-sql --cluster my_cluster \
        -e \
        "CREATE EXTERNAL TABLE foo(bar int) LOCATION\
     'gs://my_bucket/'" -e "SELECT * FROM foo WHERE bar > 2"