    configuration information, the nodes to be managed, information about
    how those nodes are grouped into partitions, and various scheduling
    parameters associated with those partitions. This file should be
    consistent across all nodes in the cluster.
    .LP
    The file location can be modified at system build time using the
    DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF
    environment variable. The Slurm daemons also allow you to override
    both the built\-in and environment\-provided location using the "\-f"
    option on the command line.
    .LP
    The contents of the file are case insensitive except for the names of nodes
    and partitions. Any text following a "#" in the configuration file is treated
    as a comment through the end of that line.
    Changes to the configuration file take effect upon restart of
    Slurm daemons, daemon receipt of the SIGHUP signal, or execution
    of the command "scontrol reconfigure" unless otherwise noted.
    .LP
    If a line begins with the word "Include" followed by whitespace
    and then a file name, that file will be included inline with the current
    configuration file. For large or complex systems, multiple configuration files
    may prove easier to manage and enable reuse of some files (See INCLUDE
    MODIFIERS for more details).
    .LP
    Note on file permissions:
    .LP
    The \fIslurm.conf\fR file must be readable by all users of Slurm, since it
    is used by many of the Slurm commands.  Other files that are defined
    in the \fIslurm.conf\fR file, such as log files and job accounting files,
    may need to be created/owned by the user "SlurmUser" to be successfully
    accessed.  Use the "chown" and "chmod" commands to set the ownership
    and permissions appropriately.
    See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
    about the various files and directories used by Slurm.

    .SH "PARAMETERS"
    .LP
    The overall configuration parameters available include:

    .TP
    \fBAccountingStorageBackupHost\fR
    The name of the backup machine hosting the accounting storage database.
    If used with the accounting_storage/slurmdbd plugin, this is where the backup
    slurmdbd would be running.
    Only used with systems using SlurmDBD, ignored otherwise.

    .TP
    \fBAccountingStorageEnforce\fR
    This controls what level of association\-based enforcement to impose
    on job submissions.  Valid options are any combination of
    \fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR,
    \fIsafe\fR, and \fIwckeys\fR, or \fIall\fR for all things (except nojobs
    enforced, users can be limited by association to whatever job size or run
    time limits are defined.

    If \fInojobs\fR is set, Slurm will not account for any jobs or steps on the
    system. Likewise, if \fInosteps\fR is set, Slurm will not account for any
    steps that have run.

    If \fIsafe\fR is enforced, a job will only be launched against an association
    or qos that has a \fBGrpTRESMins\fR limit set, if the job will be able to
    run to completion. Without this option set, jobs will be launched as long as
    their usage hasn't reached the cpu-minutes limit. This can lead to jobs being
    launched but then killed when the limit is reached.

    With \fIqos\fR and/or \fIwckeys\fR enforced jobs will not be scheduled unless
    a valid qos and/or workload characterization key is specified.

    When \fBAccountingStorageEnforce\fR is changed, a restart of the slurmctld
    daemon is required (not just a "scontrol reconfig").

    .TP
    \fBAccountingStorageExternalHost\fR
    A comma-separated list of external slurmdbds (<host/ip>[:port][,...]) to
    register with. If no port is given, the \fBAccountingStoragePort\fR will be
    used.

    This allows clusters registered with the external slurmdbd to communicate with
    each other using the \fI--cluster/-M\fR client command options.

    The cluster will add itself to the external slurmdbd if it doesn't exist. If a
    non-external cluster already exists on the external slurmdbd, the slurmctld
    will ignore registering to the external slurmdbd.

    .TP
    \fBAccountingStorageHost\fR
    The name of the machine hosting the accounting storage database.
    Only used with systems using SlurmDBD, ignored otherwise.
    Also see \fBDefaultStorageHost\fR.

    .TP
    \fBAccountingStorageParameters\fR
    Comma-separated list of key-value pair parameters. Currently
    supported values include options to establish a secure connection to the
    database:
    .RS
    .TP 2
    \fBSSL_CERT\fR
    The path name of the client public key certificate file.
    .TP
    \fBSSL_CA\fR
    The path name of the Certificate Authority (CA) certificate file.
    .TP
    \fBSSL_CAPATH\fR
    authentication this can be configured to use a MUNGE daemon
    specifically configured to provide authentication between clusters
    while the default MUNGE daemon provides authentication within a
    cluster.  In that case, \fBAccountingStoragePass\fR should specify the
    named port to be used for communications with the alternate MUNGE
    daemon (e.g.  "/var/run/munge/global.socket.2"). The default value is
    NULL.  Also see \fBDefaultStoragePass\fR.

    .TP
    \fBAccountingStoragePort\fR
    The listening port of the accounting storage database server.
    Only used for database type storage plugins, ignored otherwise.
    The default value is SLURMDBD_PORT as established at system
    build time. If no value is explicitly specified, it will be set to 6819.
    This value must be equal to the \fBDbdPort\fR parameter in the
    slurmdbd.conf file.
    Also see \fBDefaultStoragePort\fR.

    .TP
    \fBAccountingStorageTRES\fR
    Comma-separated list of resources you wish to track on the cluster.
    These are the resources requested by the sbatch/srun job when it
    is submitted. Currently this consists of any GRES, BB (burst buffer) or
    license along with CPU, Memory, Node, Energy, FS/[Disk|Lustre], IC/OFED, Pages,
    and VMem. By default Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem
    are tracked. These default TRES cannot be disabled, but only appended to.
    AccountingStorageTRES=gres/craynetwork,license/iop1
    will track billing, cpu, energy, memory, nodes, fs/disk, pages and vmem along
    with a gres called craynetwork as well as a license called iop1. Whenever these
    resources are used on the cluster they are recorded. The TRES are automatically
    set up in the database on the start of the slurmctld.

    If multiple GRES of different types are tracked (e.g. GPUs of different types),
    then job requests with matching type specifications will be recorded.
    Given a configuration of
    "AccountingStorageTRES=gres/gpu,gres/gpu:tesla,gres/gpu:volta"
    Then "gres/gpu:tesla" and "gres/gpu:volta" will track only jobs that explicitly
    request those two GPU types, while "gres/gpu" will track allocated GPUs of any
    type ("tesla", "volta" or any other GPU type).

    Given a configuration of
    "AccountingStorageTRES=gres/gpu:tesla,gres/gpu:volta"
    Then "gres/gpu:tesla" and "gres/gpu:volta" will track jobs that explicitly
    request those GPU types.
    If a job requests GPUs, but does not explicitly specify the GPU type, then
    its resource allocation will be accounted for as either "gres/gpu:tesla" or
    "gres/gpu:volta", although the accounting may not match the actual GPU type
    allocated to the job and the GPUs allocated to the job could be heterogeneous.
    In an environment containing various GPU types, use of a job_submit plugin
    may be desired in order to force jobs to explicitly specify some GPU type.

    .TP
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStorageUser\fR.

    .TP
    \fBAccountingStoreJobComment\fR
    If set to "YES" then include the job's comment field in the job
    complete message sent to the Accounting Storage database.  The default
    is "YES".
    Note the AdminComment and SystemComment are always recorded in the database.

    .TP
    \fBAcctGatherNodeFreq\fR
    The AcctGather plugins sampling interval for node accounting.
    For AcctGather plugin values of none, this parameter is ignored.
    For all other values this parameter is the number
    of seconds between node accounting samples. For the
    acct_gather_energy/rapl plugin, set a value less
    than 300 because the counters may overflow beyond this rate.
    The default value is zero. This value disables accounting sampling
    for nodes. Note: The accounting sampling interval for jobs is
    determined by the value of \fBJobAcctGatherFrequency\fR.

    .TP
    \fBAcctGatherEnergyType\fR
    Identifies the plugin to be used for energy consumption accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    energy consumption data for jobs and nodes. The collection of energy
    consumption data takes place on the node level, hence only in case of exclusive
    job allocation the energy consumption measurements will reflect the job's
    real consumption. In case of node sharing between jobs the reported consumed
    energy per job (through sstat or sacct) will not reflect the real energy
    consumed by the jobs.

    Configurable values at present are:
    .RS
    .TP 20
    \fBacct_gather_energy/none\fR
    No energy consumption data is collected.
    .TP
    \fBacct_gather_energy/ipmi\fR
    Energy consumption data is collected from the Baseboard Management Controller
    (BMC) using the Intelligent Platform Management Interface (IPMI).
    .TP
    \fBacct_gather_energy/pm_counters\fR
    Energy consumption data is collected from the Baseboard Management
    Controller (BMC) for HPE Cray systems.
    .TP
    \fBacct_gather_energy/rapl\fR
    Energy consumption data is collected from hardware sensors using the Running
    Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the
    execution of the command "sudo modprobe msr".
    .TP
    network traffic by the jobs.

    Configurable values at present are:
    .RS
    .TP 20
    \fBacct_gather_interconnect/none\fR
    No infiniband network data are collected.
    .TP
    \fBacct_gather_interconnect/ofed\fR
    Infiniband network traffic data are collected from the hardware monitoring
    counters of Infiniband devices through the OFED library.
    In order to account for per job network traffic, add the "ic/ofed" TRES to
    \fIAccountingStorageTRES\fR.
    .RE

    .TP
    \fBAcctGatherFilesystemType\fR
    Identifies the plugin to be used for filesystem traffic accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    filesystem traffic data for jobs and nodes.
    The collection of filesystem traffic data takes place on the node level,
    hence only in case of exclusive job allocation the collected values will
    reflect the job's real traffic. In case of node sharing between jobs the reported
    filesystem traffic per job (through sstat or sacct) will not reflect the real
    filesystem traffic by the jobs.

    Configurable values at present are:
    .RS
    .TP 20
    \fBacct_gather_filesystem/none\fR
    No filesystem data are collected.
    .TP
    \fBacct_gather_filesystem/lustre\fR
    Lustre filesystem traffic data are collected from the counters found in
    /proc/fs/lustre/.
    In order to account for per job lustre traffic, add the "fs/lustre" TRES to
    \fIAccountingStorageTRES\fR.
    .RE

    .TP
    \fBAcctGatherProfileType\fR
    Identifies the plugin to be used for detailed job profiling.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    detailed data such as I/O counts, memory usage, or energy consumption for jobs
    and nodes. There are interfaces in this plugin to collect data as step start
    and completion, task start and completion, and at the account gather
    frequency. The data collected at the node level is related to jobs only in
    case of exclusive job allocation.

    Configurable values at present are:
    .RS

    .TP
    \fBAllowSpecResourcesUsage\fR
    If set to "YES", Slurm allows individual jobs to override node's configured
    CoreSpecCount value. For a job to take advantage of this feature,
    a command line option of \-\-core\-spec must be specified.  The default
    value for this option is "YES" for Cray systems and "NO" for other system types.

    .TP
    \fBAuthAltTypes\fR
    Comma-separated list of alternative authentication plugins that the slurmctld
    will permit for communication. Acceptable values at present include
    \fIauth/jwt\fR.

    \fBNOTE\fR: \fIauth/jwt\fR requires a jwt_hs256.key to be populated in the
    \fBStateSaveLocation\fR directory for \fBslurmctld\fR only. The jwt_hs256.key
    should only be visible to the SlurmUser and root. It is not suggested to place
    the jwt_hs256.key on any nodes but the controller running \fBslurmctld\fR.
    \fIauth/jwt\fR can be activated by the presence of the \fISLURM_JWT\fR
    environment variable.  When activated, it will override the default
    \fBAuthType\fR.

    .TP
    \fBAuthAltParameters\fR
    Used to define alternative authentication plugins options. Multiple options may
    be comma separated.
    .RS
    .TP 15
    \fBdisable_token_creation\fR
    Disable "scontrol token" use by non-SlurmUser accounts.
    .TP
    \fBjwt_key=\fR
    Absolute path to JWT key file. Key must be HS256, and should only be accessible
    by SlurmUser. If not set, the default key file is jwt_hs256.key in
    \fIStateSaveLocation\fR.
    .RE

    .TP
    \fBAuthInfo\fR
    Additional information to be used for authentication of communications
    between the Slurm daemons (slurmctld and slurmd) and the Slurm
    clients.  The interpretation of this option is specific to the
    configured \fBAuthType\fR.
    Multiple options may be specified in a comma delimited list.
    If not specified, the default authentication information will be used.
    .RS
    .TP 14
    \fBcred_expire\fR
    Default job step credential lifetime, in seconds (e.g. "cred_expire=1200").
    It must be sufficiently long enough to load user environment, run prolog,
    deal with the slurmd getting paged out of memory, etc.
    This also controls how long a requeued job must wait before starting again.
    .TP
    \fBAuthType\fR
    The authentication method for communications between Slurm
    components.
    Acceptable values at present include "auth/munge" and "auth/none".
    The default value is "auth/munge".
    "auth/none" includes the UID in each communication, but it is not verified.
    This may be fine for testing purposes, but
    \fBdo not use "auth/none" if you desire any security\fR.
    "auth/munge" indicates that MUNGE is to be used.
    (See "https://dun.github.io/munge/" for more information).
    All Slurm daemons and commands must be terminated prior to changing
    the value of \fBAuthType\fR and later restarted.

    .TP
    \fBBackupAddr\fR
    Deprecated option, see \fBSlurmctldHost\fR.

    .TP
    \fBBackupController\fR
    Deprecated option, see \fBSlurmctldHost\fR.

    The backup controller recovers state information from the
    \fBStateSaveLocation\fR directory, which must be readable and writable from both
    the primary and backup controllers.
    While not essential, it is recommended that you specify a backup controller.
    See  the \fBRELOCATING CONTROLLERS\fR section if you change this.

    .TP
    \fBBatchStartTimeout\fR
    The maximum time (in seconds) that a batch job is permitted for
    launching before being considered missing and releasing the
    allocation. The default value is 10 (seconds). Larger values may be
    required if more time is required to execute the \fBProlog\fR, load
    user environment variables, or if the slurmd daemon gets paged from memory.
    .br
    .br
    \fBNote\fR: The test for a job being successfully launched is only performed when
    the Slurm daemon on the compute node registers state with the slurmctld daemon
    on the head node, which happens fairly rarely.
    Therefore a job will not necessarily be terminated if its start time exceeds
    \fBBatchStartTimeout\fR.
    This configuration parameter is also applied to launch tasks and avoid aborting
    \fBsrun\fR commands due to long running \fBProlog\fR scripts.

    .TP
    \fBBurstBufferType\fR
    The plugin used to manage burst buffers. Acceptable values at present are:
    .RS
    .TP
    \fBburst_buffer/datawarp\fR
    Use Cray DataWarp API to provide burst buffer functionality.
    The name by which this Slurm managed cluster is known in the
    accounting database.  This is needed distinguish accounting records
    when multiple clusters report to the same database. Because of limitations
    in some databases, any upper case letters in the name will be silently mapped
    to lower case. In order to avoid confusion, it is recommended that the name
    be lower case.

    .TP
    \fBCommunicationParameters\fR
    Comma-separated options identifying communication options.
    .RS
    .TP 15
    \fBCheckGhalQuiesce\fR
    Used specifically on a Cray using an Aries Ghal interconnect.  This will check
    to see if the system is quiescing when sending a message, and if so, we wait
    until it is done before sending.
    .TP
    \fBDisableIPv4\fR
    Disable IPv4 only operation for all slurm daemons (except slurmdbd). This
    should also be set in your \fBslurmdbd.conf\fR file.
    .TP
    \fBEnableIPv6\fR
    Enable using IPv6 addresses for all slurm daemons (except slurmdbd). When
    using both IPv4 and IPv6, address family preferences will be based on your
    /etc/gai.conf file. This should also be set in your \fBslurmdbd.conf\fR file.
    .TP
    \fBNoAddrCache\fR
    By default, Slurm will cache a node's network address after successfully
    establishing the node's network address. This option disables the cache and
    Slurm will look up the node's network address each time a connection is made.
    This is useful, for example, in a cloud environment where the node addresses
    come and go out of DNS.
    .TP
    \fBNoCtldInAddrAny\fR
    Used to directly bind to the address of what the node resolves to running
    the slurmctld instead of binding messages to any address on the node,
    which is the default.
    .TP
    \fBNoInAddrAny\fR
    Used to directly bind to the address of what the node resolves to instead
    of binding messages to any address on the node which is the default.
    This option is for all daemons/clients except for the slurmctld.
    .RE

    .TP
    \fBCompleteWait\fR
    The time to wait, in seconds, when any job is in the COMPLETING state
    before any additional jobs are scheduled. This is to attempt to keep jobs on
    nodes that were recently in use, with the goal of preventing fragmentation.
    If set to zero, pending jobs will be started as soon as possible.
    Since a COMPLETING job's resources are released for use by other
    \fBControlAddr\fR
    Deprecated option, see \fBSlurmctldHost\fR.

    .TP
    \fBControlMachine\fR
    Deprecated option, see \fBSlurmctldHost\fR.

    .TP
    \fBCoreSpecPlugin\fR
    Identifies the plugins to be used for enforcement of core specialization.
    The slurmd daemon must be restarted for a change in CoreSpecPlugin
    to take effect.
    Acceptable values at present include:
    .RS
    .TP 20
    \fBcore_spec/cray_aries\fR
    used only for Cray systems
    .TP
    \fBcore_spec/none\fR
    used for all other system types
    .RE

    .TP
    \fBCpuFreqDef\fR
    Default CPU frequency value or frequency governor to use when running a
    job step if it has not been explicitly set with the \-\-cpu\-freq option.
    Acceptable values at present include a numeric value (frequency in kilohertz)
    or one of the following governors:
    .RS
    .TP 14
    \fBConservative\fR
    attempts to use the Conservative CPU governor
    .TP
    \fBOnDemand\fR
    attempts to use the OnDemand CPU governor
    .TP
    \fBPerformance\fR
    attempts to use the Performance CPU governor
    .TP
    \fBPowerSave\fR
    attempts to use the PowerSave CPU governor
    .RE
    There is no default value. If unset, no attempt to set the governor is
    made if the \-\-cpu\-freq option has not been set.

    .TP
    \fBCpuFreqGovernors\fR
    List of CPU frequency governors allowed to be set with the salloc, sbatch, or
    srun option  \-\-cpu\-freq.
    Acceptable values at present include:
    .RS
    .TP 14
    .RE
    The default is OnDemand, Performance and UserSpace.
    .TP
    \fBCredType\fR
    The cryptographic signature tool to be used in the creation of
    job step credentials.
    The slurmctld daemon must be restarted for a change in \fBCredType\fR
    to take effect.
    Acceptable values at present include "cred/munge" and "cred/none".
    The default value is "cred/munge" and is the recommended.

    .TP
    \fBDebugFlags\fR
    Defines specific subsystems which should provide more detailed event logging.
    Multiple subsystems can be specified with comma separators.
    Most DebugFlags will result in verbose-level logging for the identified
    subsystems, and could impact performance.
    Valid subsystems available include:
    .RS
    .TP 17
    \fBAccrue\fR
    Accrue counters accounting details
    .TP
    \fBAgent\fR
    RPC agents (outgoing RPCs from Slurm daemons)
    .TP
    \fBBackfill\fR
    Backfill scheduler details
    .TP
    \fBBackfillMap\fR
    Backfill scheduler to log a very verbose map of reserved resources through
    time. Combine with \fBBackfill\fR for a verbose and complete view of the
    backfill scheduler's work.
    .TP
    \fBBurstBuffer\fR
    Burst Buffer plugin
    .TP
    \fBCPU_Bind\fR
    CPU binding details for jobs and steps
    .TP
    \fBCpuFrequency\fR
    Cpu frequency details for jobs and steps using the \-\-cpu\-freq option.
    .TP
    \fBData\fR
    Generic data structure details.
    .TP
    \fBDependency\fR
    Job dependency debug info
    .TP
    \fBElasticsearch\fR
    Elasticsearch debug info
    .TP
    .TP
    \fBHetjob\fR
    Heterogeneous job details
    .TP
    \fBGang\fR
    Gang scheduling details
    .TP
    \fBJobContainer\fR
    Job container plugin details
    .TP
    \fBLicense\fR
    License management details
    .TP
    \fBNetwork\fR
    Network details
    .TP
    \fBNetworkRaw\fR
    Dump raw hex values of key Network communications. Warning: very verbose.
    .TP
    \fBNodeFeatures\fR
    Node Features plugin debug info
    .TP
    \fBNO_CONF_HASH\fR
    Do not log when the slurm.conf files differ between Slurm daemons
    .TP
    \fBPower\fR
    Power management plugin
    .TP
    \fBPowerSave\fR
    Power save (suspend/resume programs) details
    .TP
    \fBPriority\fR
    Job prioritization
    .TP
    \fBProfile\fR
    AcctGatherProfile plugins details
    .TP
    \fBProtocol\fR
    Communication protocol details
    .TP
    \fBReservation\fR
    Advanced reservations
    .TP
    \fBRoute\fR
    Message forwarding debug info
    .TP
    \fBSelectType\fR
    Resource selection plugin
    .TP
    \fBSteps\fR
    Slurmctld resource allocation for job steps
    .TP
    Slurmctld triggers
    .TP
    \fBWorkQueue\fR
    Work Queue details
    .RE

    .TP
    \fBDefCpuPerGPU\fR
    Default count of CPUs allocated per allocated GPU. This value is used only if
    the job didn't specify --cpus-per-task and --cpus-per-gpu.

    .TP
    \fBDefMemPerCPU\fR
    Default real memory size available per allocated CPU in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBDefMemPerCPU\fR would generally be used if individual processors
    are allocated to jobs (\fBSelectType=select/cons_res\fR or
    \fBSelectType=select/cons_tres\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.

    .TP
    \fBDefMemPerGPU\fR
    Default real memory size available per allocated GPU in megabytes.
    The default value is 0 (unlimited).
    Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.

    .TP
    \fBDefMemPerNode\fR
    Default real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBDefMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.

    .TP
    \fBDefaultStorageHost\fR
    The default name of the machine hosting the accounting storage and
    job completion databases.
    Only used for database type storage plugins and when the
    \fBAccountingStorageHost\fR and \fBJobCompHost\fR have not been
    defined.

    \fBDefaultStoragePort\fR
    The listening port of the accounting storage and/or job completion
    database server.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBAccountingStoragePort\fR and \fBJobCompPort\fR.

    .TP
    \fBDefaultStorageType\fR
    The accounting and job completion storage mechanism type.  Acceptable
    values at present include "filetxt", "mysql" and "none".
    The value "filetxt" indicates that records will be written to a file.
    The value "mysql" indicates that accounting records will be written to a MySQL
    or MariaDB database.
    The default value is "none", which means that records are not maintained.
    Also see \fBAccountingStorageType\fR and \fBJobCompType\fR.

    .TP
    \fBDefaultStorageUser\fR
    The user account for accessing the accounting storage and/or job
    completion database.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR.

    .TP
    \fBDependencyParameters\fR
    Multiple options may be comma separated.

    .RS
    .TP
    \fBdisable_remote_singleton\fR
    By default, when a federated job has a singleton dependency, each cluster in the
    federation must clear the singleton dependency before the job's singleton
    dependency is considered satisfied. Enabling this option means that only the
    origin cluster must clear the singleton dependency. This option must be set
    in every cluster in the federation.
    .TP
    \fBkill_invalid_depend\fR
    If a job has an invalid dependency and it can never run terminate it
    and set its state to be JOB_CANCELLED. By default the job stays pending
    with reason DependencyNeverSatisfied.
    \fBmax_depend_depth=#\fR
    Maximum number of jobs to test for a circular job dependency. Stop testing
    after this number of job dependencies have been tested. The default value is
    10 jobs.
    .RE

    .TP
    \fBDisableRootJobs\fR
    If set to "YES" then user root will be prevented from running any jobs.
    The default value is "NO", meaning user root will be able to execute jobs.
    \fBDisableRootJobs\fR may also be set by partition.

    If set to "ANY" a job must satisfy any of the requested partitions
    to be submitted. The default value is "NO".
    NOTE: If set, then a job's QOS can not be used to exceed partition limits.
    NOTE: The partition limits being considered are its configured MaxMemPerCPU,
    MaxMemPerNode, MinNodes, MaxNodes, MaxTime, AllocNodes, AllowAccounts,
    AllowGroups, AllowQOS, and QOS usage threshold.

    .TP
    \fBEpilog\fR
    Fully qualified pathname of a script to execute as user root on every
    node when a user's job completes (e.g. "/usr/local/slurm/epilog"). A
    glob pattern (See \fBglob\fR (7)) may also be used to run more than
    one epilog script (e.g. "/etc/slurm/epilog.d/*"). The Epilog script
    or scripts may be used to purge files, disable user login, etc.
    By default there is no epilog.
    See \fBProlog and Epilog Scripts\fR for more information.

    .TP
    \fBEpilogMsgTime\fR
    The number of microseconds that the slurmctld daemon requires to process
    an epilog completion message from the slurmd daemons. This parameter can
    be used to prevent a burst of epilog completion messages from being sent
    at the same time which should help prevent lost messages and improve
    throughput for large jobs.
    The default value is 2000 microseconds.
    For a 1000 node job, this spreads the epilog completion messages out over
    two seconds.

    .TP
    \fBEpilogSlurmctld\fR
    Fully qualified pathname of a program for the slurmctld to execute
    upon termination of a job allocation (e.g.
    "/usr/local/slurm/epilog_controller").
    The program executes as SlurmUser, which gives it permission to drain
    nodes and requeue the job if a failure occurs (See scontrol(1)).
    Exactly what the program does and how it accomplishes this is completely at
    the discretion of the system administrator.
    Information about the job being initiated, its allocated nodes, etc. are
    passed to the program using environment variables.
    See \fBProlog and Epilog Scripts\fR for more information.

    .TP
    \fBExtSensorsFreq\fR
    The external sensors plugin sampling interval.
    If \fBExtSensorsType=ext_sensors/none\fR, this parameter is ignored.
    For all other values of \fBExtSensorsType\fR, this parameter is the number
    of seconds between external sensors samples for hardware components (nodes,
    switches, etc.) The default value is zero. This value disables external
    sensors sampling. Note: This parameter does not affect external sensors
    data collection for jobs/steps.

    .TP
    External sensors data is collected from the RRD database.
    .RE

    .TP
    \fBFairShareDampeningFactor\fR
    Dampen the effect of exceeding a user or group's fair share of allocated
    resources. Higher values will provides greater ability to differentiate
    between exceeding the fair share at high levels (e.g. a value of 1 results
    in almost no difference between overconsumption by a factor of 10 and 100,
    while a value of 5 will result in a significant difference in priority).
    The default value is 1.

    .TP
    \fBFederationParameters\fR
    Used to define federation options. Multiple options may be comma separated.

    .RS
    .TP
    \fBfed_display\fR
    If set, then the client status commands (e.g. squeue, sinfo, sprio, etc.) will
    display information in a federated view by default. This option is functionally
    equivalent to using the \-\-federation options on each command. Use the client's
    \-\-local option to override the federated view and get a local view of the
    given cluster.
    .RE

    .TP
    \fBFirstJobId\fR
    The job id to be used for the first submitted to Slurm without a
    specific requested value. Job id values generated will incremented by 1
    for each subsequent job. This may be used to provide a meta\-scheduler
    with a job id space which is disjoint from the interactive jobs.
    The default value is 1.
    Also see \fBMaxJobId\fR

    .TP
    \fBGetEnvTimeout\fR
    Controls how long the job should wait (in seconds) to load the user's
    environment before attempting to load it from a cache file.
    Applies when the salloc or sbatch \fI\-\-get\-user\-env\fR option is used.
    If set to 0 then always load the user's environment from the cache file.
    The default value is 2 seconds.

    .TP
    \fBGresTypes\fR
    A comma delimited list of generic resources to be managed (e.g.
    \fIGresTypes=gpu,mps\fR).
    These resources may have an associated GRES plugin of the same name providing
    additional functionality.
    No generic resources are managed by default.
    Ensure this parameter is consistent across all nodes in the cluster for
    proper operation.
    \fBGroupUpdateTime\fR
    Controls how frequently information about which users are members of
    groups allowed to use a partition will be updated, and how long user
    group membership lists will be cached.
    The time interval is given in seconds with a default value of 600 seconds.
    A value of zero will prevent periodic updating of group membership information.
    Also see the \fBGroupUpdateForce\fR parameter.

    .TP
    \fBGpuFreqDef\fR=[<\fItype\fR]=\fIvalue\fR>[,<\fItype\fR=\fIvalue\fR>]
    Default GPU frequency to use when running a job step if it
    has not been explicitly set using the \-\-gpu\-freq option.
    This option can be used to independently configure the GPU and its memory
    frequencies. Defaults to "high,memory=high".
    After the job is completed, the frequencies of all affected GPUs will be reset
    to the highest possible values.
    In some cases, system power caps may override the requested values.
    The field \fItype\fR can be "memory".
    If \fItype\fR is not specified, the GPU frequency is implied.
    The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
    a numeric value in megahertz (MHz).
    If the specified numeric value is not possible, a value as close as
    possible will be used.
    See below for definition of the values.
    Examples of use include "GpuFreqDef=medium,memory=high and "GpuFreqDef=450".

    Supported \fIvalue\fR definitions:
    .RS
    .TP 10
    \fBlow\fR
    the lowest available frequency.
    .TP
    \fBmedium\fR
    attempts to set a frequency in the middle of the available range.
    .TP
    \fBhigh\fR
    the highest available frequency.
    .TP
    \fBhighm1\fR
    (high minus one) will select the next highest available frequency.
    .RE

    .TP
    \fBHealthCheckInterval\fR
    The interval in seconds between executions of \fBHealthCheckProgram\fR.
    The default value is zero, which disables execution.

    .TP
    \fBHealthCheckNodeState\fR
    Identify what node states should execute the \fBHealthCheckProgram\fR.
    Multiple state values may be specified with a comma separator.
    The default value is ANY to execute on nodes in any state.
    \fBIDLE\fR
    Run on nodes in the IDLE state.
    .TP
    \fBMIXED\fR
    Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).
    .RE

    .TP
    \fBHealthCheckProgram\fR
    Fully qualified pathname of a script to execute as user root periodically
    on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This
    program may be used to verify the node is fully operational and DRAIN the node
    or send email if a problem is detected.
    Any action to be taken must be explicitly performed by the program
    (e.g. execute
    "scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
    to drain a node).
    The execution interval is controlled using the \fBHealthCheckInterval\fR
    parameter.
    Note that the \fBHealthCheckProgram\fR will be executed at the same time
    on all nodes to minimize its impact upon parallel programs.
    This program is will be killed if it does not terminate normally within
    60 seconds.
    This program will also be executed when the slurmd daemon is first started and
    before it registers with the slurmctld daemon.
    By default, no program will be executed.

    .TP
    \fBInactiveLimit\fR
    The interval, in seconds, after which a non\-responsive job allocation
    command (e.g. \fBsrun\fR or \fBsalloc\fR) will result in the job being
    terminated. If the node on which the command is executed fails or the
    command abnormally terminates, this will terminate its job allocation.
    This option has no effect upon batch jobs.
    When setting a value, take into consideration that a debugger using \fBsrun\fR
    to launch an application may leave the \fBsrun\fR command in a stopped state
    for extended periods of time.
    This limit is ignored for jobs running in partitions with the
    \fBRootOnly\fR flag set (the scheduler running as root will be
    responsible for the job).
    The default value is unlimited (zero) and may not exceed 65533 seconds.

    .TP
    \fBInteractiveStepOptions\fR
    When LaunchParameters=use_interactive_step is enabled, launching salloc will
    automatically start an srun process with InteractiveStepOptions to launch
    a terminal on a node in the job allocation.
    The default value is "--interactive --preserve-env --pty $SHELL".

    .TP
    \fBJobAcctGatherType\fR
    The job accounting mechanism type.
    the messages between Slurm daemons. Any previously running job steps are
    managed by a slurmstepd daemon that will persist through the lifetime of
    that job step and not change its communication protocol. Only change this
    configuration parameter when there are no running job steps.

    .TP
    \fBJobAcctGatherFrequency\fR
    The job accounting and profiling sampling intervals.
    The supported format is follows:
    .RS
    .TP 12
    \fBJobAcctGatherFrequency=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
    where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
    interval for the jobacct_gather plugin or a
    sampling interval for a profiling type by the
    acct_gather_profile plugin. Multiple,
    comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals
    may be specified. Supported datatypes are as follows:
    .RS
    .TP
    \fBtask=\fI<interval>\fR
    where \fI<interval>\fR is the task sampling interval in seconds
    for the jobacct_gather plugins and for task
    profiling by the acct_gather_profile plugin.
    .TP
    \fBenergy=\fI<interval>\fR
    where \fI<interval>\fR is the sampling interval in seconds
    for energy profiling using the acct_gather_energy plugin
    .TP
    \fBnetwork=\fI<interval>\fR
    where \fI<interval>\fR is the sampling interval in seconds
    for infiniband profiling using the acct_gather_interconnect
    plugin.
    .TP
    \fBfilesystem=\fI<interval>\fR
    where \fI<interval>\fR is the sampling interval in seconds
    for filesystem profiling using the acct_gather_filesystem
    plugin.
    .TP
    .RE
    .RE
    The default value for task sampling interval
    is 30 seconds. The default value for all other intervals is 0.
    An interval of 0 disables sampling of the specified type.
    If the task sampling interval is 0, accounting
    information is collected only at job termination (reducing Slurm
    interference with the job).
    .br
    .br
    Smaller (non\-zero) values have a greater impact upon job performance,
    but a value of 30 seconds is not likely to be noticeable for
    applications having less than 10,000 tasks.
    .TP
    \fBUsePss\fR
    Use PSS value instead of RSS to calculate real usage of memory.
    The PSS value will be saved as RSS.
    .TP
    \fBOverMemoryKill\fR
    Kill processes that are being detected to use more memory than requested by
    steps every time accounting information is gathered by the JobAcctGather plugin.
    This parameter should be used with caution because a job exceeding its memory
    allocation may affect other processes and/or machine health.

    \fBNOTE\fR: If available, it is recommended to limit memory by enabling
    task/cgroup as a TaskPlugin and making use of ConstrainRAMSpace=yes in the
    cgroup.conf instead of using this JobAcctGather mechanism for memory
    enforcement. With OverMemoryKill, memory limit is applied against each process
    individually and is not applied to the step as a whole. This means that when
    jobs have a process that consumes too much memory, the process will be killed
    but the step will continue to run. When using cgroups with
    ConstrainRAMSpace=yes, a process that consumes too much memory will result in
    the job step being killed. Using JobAcctGather is polling based and there is a
    delay before a job is killed, which could lead to system Out of Memory events.
    .RE

    .TP
    \fBJobCompHost\fR
    The name of the machine hosting the job completion database.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStorageHost\fR.

    .TP
    \fBJobCompLoc\fR
    The fully qualified file name where job completion records are written
    when the \fBJobCompType\fR is "jobcomp/filetxt" or the database where
    job completion records are stored when the \fBJobCompType\fR is a
    database, or a complete URL endpoint with format <host>:<port>/<target>/_doc
    when \fBJobCompType\fR is "jobcomp/elasticsearch" like i.e.
    "localhost:9200/slurm/_doc".
    NOTE: More information is available at the Slurm web site
    <https://slurm.schedmd.com/elasticsearch.html>.
    Also see \fBDefaultStorageLoc\fR.

    .TP
    \fBJobCompParams\fR
    Pass arbitrary text string to job completion plugin.
    Also see \fBJobCompType\fR.

    .TP
    \fBJobCompPass\fR
    The password used to gain access to the database to store the job
    completion data.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStoragePass\fR.
    Upon job completion, a record of the job is purged from the system.
    If using the accounting infrastructure this plugin may not be of interest
    since some of the information is redundant.

    .TP
    \fBjobcomp/elasticsearch\fR
    Upon job completion, a record of the job should be written to an
    Elasticsearch server, specified by the \fBJobCompLoc\fR parameter.
    .br
    \fBNOTE\fR: More information is available at the Slurm web site
    ( https://slurm.schedmd.com/elasticsearch.html ).

    .TP
    \fBjobcomp/filetxt\fR
    Upon job completion, a record of the job should be written to a text file,
    specified by the \fBJobCompLoc\fR parameter.

    .TP
    \fBjobcomp/lua\fR
    Upon job completion, a record of the job should be processed by the
    \fIjobcomp.lua\fR script, located in the default script directory
    (typically the subdirectory \fIetc\fR of the installation directory.

    .TP
    \fBjobcomp/mysql\fR
    Upon job completion, a record of the job should be written to a MySQL
    or MariaDB database, specified by the \fBJobCompLoc\fR parameter.

    .TP
    \fBjobcomp/script\fR
    Upon job completion, a script specified by the \fBJobCompLoc\fR parameter is
    to be executed with environment variables providing the job information.
    .RE

    .TP
    \fBJobCompUser\fR
    The user account for accessing the job completion database.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStorageUser\fR.

    .TP
    \fBJobContainerType\fR
    Identifies the plugin to be used for job tracking.
    The slurmd daemon must be restarted for a change in JobContainerType
    to take effect.
    \fBNOTE\fR: The \fBJobContainerType\fR applies to a job allocation, while
    \fBProctrackType\fR applies to job steps.
    Acceptable values at present include:
    .RS
    .TP 20
    \fBjob_container/cncu\fR
    Used only for Cray systems (CNCU = Compute Node Clean Up)
    the existing file.
    By default, any existing file is truncated.

    .TP
    \fBJobRequeue\fR
    This option controls the default ability for batch jobs to be requeued.
    Jobs may be requeued explicitly by a system administrator, after node
    failure, or upon preemption by a higher priority job.
    If \fBJobRequeue\fR is set to a value of 1, then batch job may be requeued
    unless explicitly disabled by the user.
    If \fBJobRequeue\fR is set to a value of 0, then batch job will not be requeued
    unless explicitly enabled by the user.
    Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR
    option to change the default behavior for individual jobs.
    The default value is 1.

    .TP
    \fBJobSubmitPlugins\fR
    A comma delimited list of job submission plugins to be used.
    The specified plugins will be executed in the order listed.
    These are intended to be site\-specific plugins which can be used to set
    default job parameters and/or logging events.
    Sample plugins available in the distribution include "all_partitions",
    "defaults", "logging", "lua", and "partition".
    For examples of use, see the Slurm code in "src/plugins/job_submit" and
    "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs.
    Slurm can be configured to use multiple job_submit plugins if desired,
    however the lua plugin will only execute one lua script named "job_submit.lua"
    located in the default script directory (typically the subdirectory "etc" of
    the installation directory).
    No job submission plugins are used by default.

    .TP
    \fBKeepAliveTime\fR
    Specifies how long sockets communications used between the srun command and its
    slurmstepd process are kept alive after disconnect.
    Longer values can be used to improve reliability of communications in the
    event of network failures.
    The default value leaves the system default value.
    The value may not exceed 65533.

    .TP
    \fBKillOnBadExit\fR
    If set to 1, a step will be terminated immediately if any task is
    crashed or aborted, as indicated by a non-zero exit code.
    With the default value of 0, if one of the processes is crashed or aborted
    the other processes will continue to run while the crashed or aborted process
    waits. The user can override this configuration parameter by using srun's
    \fB\-K\fR, \fB\-\-kill\-on\-bad\-exit\fR.

    .TP
    \fBKillWait\fR
    .RS
    .TP 20
    \fBnode_features/knl_cray\fR
    used only for Intel Knights Landing processors (KNL) on Cray systems
    .TP 20
    \fBnode_features/knl_generic\fR
    used for Intel Knights Landing processors (KNL) on a generic Linux system
    .RE

    .TP
    \fBLaunchParameters\fR
    Identifies options to the job launch plugin.
    Acceptable values include:
    .RS
    .TP 24
    \fBbatch_step_set_cpu_freq\fR
    Set the cpu frequency for the batch step from given \-\-cpu\-freq, or
    slurm.conf CpuFreqDef, option.  By default only steps started with srun will
    utilize the cpu freq setting options.

    NOTE: If you are using srun to launch your steps inside a batch script
    (advised) this option will create a situation where you may have multiple
    agents setting the cpu_freq as the batch step usually runs on the same
    resources one or more steps the sruns in the script will create.
    .TP 24
    \fBcray_net_exclusive\fR
    Allow jobs on a Cray Native cluster exclusive access to network resources.
    This should only be set on clusters providing exclusive access to each
    node to a single job at once, and not using parallel steps within the job,
    otherwise resources on the node can be oversubscribed.
    .TP 24
    \fBenable_nss_slurm\fR
    Permits passwd and group resolution for a job to be serviced by slurmstepd rather
    than requiring a lookup from a network based service. See
    https://slurm.schedmd.com/nss_slurm.html for more information.
    .TP 24
    \fBlustre_no_flush\fR
    If set on a Cray Native cluster, then do not flush the Lustre cache on job step
    completion. This setting will only take effect after reconfiguring, and will
    only take effect for newly launched jobs.
    .TP 24
    \fBmem_sort\fR
    Sort NUMA memory at step start. User can override this default with
    SLURM_MEM_BIND environment variable or \-\-mem\-bind=nosort command line option.
    .TP
    \fBmpir_use_nodeaddr\fR
    When launching tasks Slurm creates entries in MPIR_proctable that are used by
    parallel debuggers, profilers, and related tools to attach to running process.
    By default the MPIR_proctable entries contain MPIR_procdesc structures where
    the host_name is set to NodeName by default. If this option is specified,
    NodeAddr will be used in this context instead.
    .TP
    Have srun verify existence of the executable program along with user
    execute permission on the node where srun was called before attempting to
    launch it on nodes in the step.
    .TP
    \fBuse_interactive_step\fR
    Have salloc use the Interactive Step to launch a shell on an allocated compute
    node rather than locally to wherever salloc was invoked. This is accomplished
    by launching the srun command with InteractiveStepOptions as options.

    This does not affect salloc called with a command as an argument. These jobs
    will continue to be executed as the calling user on the calling host.
    .RE

    .TP
    \fBLaunchType\fR
    Identifies the mechanism to be used to launch application tasks.
    Acceptable values include:
    .RS
    .TP
    \fBlaunch/slurm\fR
    The default value.
    .RE

    .TP
    \fBLicenses\fR
    Specification of licenses (or other resources available on all
    nodes of the cluster) which can be allocated to jobs.
    License names can optionally be followed by a colon
    and count with a default count of one.
    Multiple license names should be comma separated (e.g.
    "Licenses=foo:4,bar").
    Note that Slurm prevents jobs from being scheduled if their
    required license specification is not available.
    Slurm does not prevent jobs from using licenses that are
    not explicitly listed in the job submission specification.

    .TP
    \fBLogTimeFormat\fR
    Format of the timestamp in slurmctld and slurmd log files. Accepted
    values are "iso8601", "iso8601_ms", "rfc5424", "rfc5424_ms", "clock",
    "short" and "thread_id". The values ending in "_ms" differ from the ones without
    in that fractional seconds with millisecond precision are printed. The
    default value is "iso8601_ms". The "rfc5424" formats are the same as
    the "iso8601" formats except that the timezone value is also
    shown. The "clock" format shows a timestamp in microseconds retrieved
    with the C standard clock() function. The "short" format is a short
    date and time format. The "thread_id" format shows the timestamp
    in the C standard ctime() function form without the year but
    including the microseconds, the daemon's process ID and the current thread name
    and ID.

    .TP
    to allow for an index value of zero.
    Configure MaxArraySize to 0 in order to disable job array use.
    The value may not exceed 4000001.
    The value of \fBMaxJobCount\fR should be much larger than \fBMaxArraySize\fR.
    The default value is 1001.
    See also \fBmax_array_tasks\fR in SchedulerParameters.

    .TP
    \fBMaxDBDMsgs\fR
    When communication to the SlurmDBD is not possible the slurmctld will queue
    messages meant to processed when the SlurmDBD is available again.
    In order to avoid running out of memory the slurmctld will only queue so many
    messages. The default value is 10000, or \fBMaxJobCount\fR * 2 + Node Count
    * 4, whichever is greater.  The value can not be less than 10000.

    .TP
    \fBMaxJobCount\fR
    The maximum number of jobs Slurm can have in its active database
    at one time. Set the values of \fBMaxJobCount\fR and \fBMinJobAge\fR
    to ensure the slurmctld daemon does not exhaust its memory or other
    resources. Once this limit is reached, requests to submit additional
    jobs will fail. The default value is 10000 jobs.
    NOTE: Each task of a job array counts as one job even though they will not
    occupy separate job records until modified or initiated.
    Performance can suffer with more than a few hundred thousand jobs.
    Setting per MaxSubmitJobs per user is generally valuable to prevent a single
    user from filling the system with jobs.
    This is accomplished using Slurm's database and configuring enforcement of
    resource limits.
    This value may not be reset via "scontrol reconfig".
    It only takes effect upon restart of the slurmctld daemon.

    .TP
    \fBMaxJobId\fR
    The maximum job id to be used for jobs submitted to Slurm without a specific
    requested value. Job ids are unsigned 32bit integers with the first 26 bits
    reserved for local job ids and the remaining 6 bits reserved for a cluster id
    to identify a federated job's origin. The maximum allowed local job id is
    67,108,863 (0x3FFFFFF). The default value is 67,043,328 (0x03ff0000).
    \fBMaxJobId\fR only applies to the local job id and not the federated job id.
    Job id values generated will be incremented by 1 for each subsequent job. Once
    \fBMaxJobId\fR is reached, the next job will be assigned \fBFirstJobId\fR.
    Federated jobs will always have a job ID of 67,108,865 or higher.
    Also see \fBFirstJobId\fR.

    .TP
    \fBMaxMemPerCPU\fR
    Maximum real memory size available per allocated CPU in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBMaxMemPerCPU\fR would generally be used if individual processors
    are allocated to jobs (\fBSelectType=select/cons_res\fR or
    \fBSelectType=select/cons_tres\fR).
    Maximum real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBMaxMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

    .TP
    \fBMaxStepCount\fR
    The maximum number of steps that any job can initiate. This parameter
    is intended to limit the effect of bad batch scripts.
    The default value is 40000 steps.

    .TP
    \fBMaxTasksPerNode\fR
    Maximum number of tasks Slurm will allow a job step to spawn
    on a single node. The default \fBMaxTasksPerNode\fR is 512.
    May not exceed 65533.

    .TP
    \fBMCSParameters\fR
    MCS = Multi-Category Security
    MCS Plugin Parameters.
    The supported parameters are specific to the \fBMCSPlugin\fR.
    Changes to this value take effect when the Slurm daemons are reconfigured.
    More information about MCS is available here
    <https://slurm.schedmd.com/mcs.html>.

    .TP
    \fBMCSPlugin\fR
    MCS = Multi-Category Security : associate a security label to jobs and ensure
    that nodes can only be shared among jobs using the same security label.
    Acceptable values include:
    .RS
    .TP 12
    \fBmcs/none\fR
    is the default value.
    No security label associated with jobs,
    no particular security restriction when sharing nodes among jobs.
    .TP
    \fBmcs/account\fR
    only users with the same account can share the nodes (requires enabling of accounting).
    .TP
    \fBmcs/group\fR
    only users with the same group can share the nodes.
    .TP
    \fBmcs/user\fR
    a node cannot be shared with other users.
    .RE
    A value of zero prevents any job record purging.
    Jobs are not purged during a backfill cycle, so it can take longer than
    MinJobAge seconds to purge a job if using the backfill scheduling plugin.
    In order to eliminate some possible race conditions, the minimum non\-zero
    value for \fBMinJobAge\fR recommended is 2.

    .TP
    \fBMpiDefault\fR
    Identifies the default type of MPI to be used.
    Srun may override this configuration parameter in any case.
    Currently supported versions include:
    \fBpmi2\fR,
    \fBpmix\fR, and
    \fBnone\fR (default, which works for many other versions of MPI).
    More information about MPI use is available here
    <https://slurm.schedmd.com/mpi_guide.html>.

    .TP
    \fBMpiParams\fR
    MPI parameters.
    Used to identify ports used by older versions of OpenMPI and native
    Cray systems.  The input format is "ports=12000\-12999" to identify a range of
    communication ports to be used.
    NOTE: This is not needed for modern versions of OpenMPI, taking it out can cause
    a small boost in scheduling performance.
    NOTE: This is require for Cray's PMI.

    .TP
    \fBOverTimeLimit\fR
    Number of minutes by which a job can exceed its time limit before
    being canceled.
    Normally a job's time limit is treated as a \fIhard\fR limit and the job will be
    killed upon reaching that limit.
    Configuring \fBOverTimeLimit\fR will result in the job's time limit being
    treated like a \fIsoft\fR limit.
    Adding the \fBOverTimeLimit\fR value to the \fIsoft\fR time limit provides a
    \fIhard\fR time limit, at which point the job is canceled.
    This is particularly useful for backfill scheduling, which bases upon
    each job's soft time limit.
    The default value is zero.
    May not exceed 65533 minutes.
    A value of "UNLIMITED" is also supported.

    .TP
    \fBPluginDir\fR
    Identifies the places in which to look for Slurm plugins.
    This is a colon\-separated list of directories, like the PATH
    environment variable.
    The default value is the prefix given at configure time + "/lib/slurm".

    .TP
    \fBPlugStackConfig\fR
    <https://slurm.schedmd.com/power_mgmt.html>.
    Options current supported by any plugins are listed below.
    .RS
    .TP
    \fBbalance_interval=#\fR
    Specifies the time interval, in seconds, between attempts to rebalance power
    caps across the nodes.
    This also controls the frequency at which Slurm attempts to collect current
    power consumption data (old data may be used until new data is available from
    the underlying infrastructure and values below 10 seconds are not recommended
    for Cray systems).
    The default value is 30 seconds.
    Supported by the power/cray_aries plugin.
    .TP
    \fBcapmc_path=\fR
    Specifies the absolute path of the capmc command.
    The default value is "/opt/cray/capmc/default/bin/capmc".
    Supported by the power/cray_aries plugin.
    .TP
    \fBcap_watts=#\fR
    Specifies the total power limit to be established across all compute nodes
    managed by Slurm.
    A value of 0 sets every compute node to have an unlimited cap.
    The default value is 0.
    Supported by the power/cray_aries plugin.
    .TP
    \fBdecrease_rate=#\fR
    Specifies the maximum rate of change in the power cap for a node where the
    actual power usage is below the power cap by an amount greater than
    \fBlower_threshold\fR (see below).
    Value represents a percentage of the difference between a node's minimum and
    maximum power consumption.
    The default value is 50 percent.
    Supported by the power/cray_aries plugin.
    .TP
    \fBget_timeout=#\fR
    Amount of time allowed to get power state information in milliseconds.
    The default value is 5,000 milliseconds or 5 seconds.
    Supported by the power/cray_aries plugin and represents the time allowed for the
    capmc command to respond to various "get" options.
    .TP
    \fBincrease_rate=#\fR
    Specifies the maximum rate of change in the power cap for a node where the
    actual power usage is within \fBupper_threshold\fR (see below) of the power cap.
    Value represents a percentage of the difference between a node's minimum and
    maximum power consumption.
    The default value is 20 percent.
    Supported by the power/cray_aries plugin.
    .TP
    \fBjob_level\fR
    All nodes associated with every job will have the same power cap, to the extent
    possible.
    .TP
    \fBrecent_job=#\fR
    If a job has started or resumed execution (from suspend) on a compute node
    within this number of seconds from the current time, the node's power cap will
    be increased to the maximum.
    The default value is 300 seconds.
    Supported by the power/cray_aries plugin.

    .TP
    \fBset_timeout=#\fR
    Amount of time allowed to set power state information in milliseconds.
    The default value is 30,000 milliseconds or 30 seconds.
    Supported by the power/cray plugin and represents the time allowed for the
    capmc command to respond to various "set" options.
    .TP
    \fBset_watts=#\fR
    Specifies the power limit to be set on every compute nodes managed by Slurm.
    Every node gets this same power cap and there is no variation through time
    based upon actual power usage on the node.
    Supported by the power/cray_aries plugin.
    .TP
    \fBupper_threshold=#\fR
    Specify an upper power consumption threshold.
    If a node's current power consumption is above this percentage of its current
    cap, then its power cap will be increased to the extent possible.
    The default value is 95 percent.
    Supported by the power/cray_aries plugin.
    .RE

    .TP
    \fBPowerPlugin\fR
    Identifies the plugin used for system power management.
    Currently supported plugins include:
    \fBcray_aries\fR and
    \fBnone\fR.
    Changes to this value require restarting Slurm daemons to take effect.
    More information about system power management is available here
    <https://slurm.schedmd.com/power_mgmt.html>.
    By default, no power plugin is loaded.

    .TP
    \fBPreemptMode\fR
    Mechanism used to preempt jobs or enable gang scheduling. When the
    \fBPreemptType\fR parameter is set to enable preemption, the
    \fBPreemptMode\fR selects the default mechanism used to preempt the eligible
    jobs for the cluster.
    .br
    \fBPreemptMode\fR may be specified on a per partition basis to override this
    default value if \fBPreemptType=preempt/partition_prio\fR. Alternatively, it
    can be specified on a per QOS basis if \fBPreemptType=preempt/qos\fR. In either
    case, a valid default \fBPreemptMode\fR value must be specified for the
    cluster as a whole when preemption is enabled.
    These reserved nodes aren't available to other jobs during that backfill
    cycle, even if the other jobs could fit on the nodes. Therefore, jobs may
    preempt more resources during a single backfill iteration than they requested.

    \fBNOTE:\fR
    For heterogeneous job to be considered for preemption all components
    must be eligible for preemption. When a heterogeneous job is to be preempted
    the first identified component of the job with the highest order PreemptMode
    (\fBSUSPEND\fR (highest), \fBREQUEUE\fR, \fBCANCEL\fR (lowest)) will be
    used to set the PreemptMode for all components. The \fBGraceTime\fR and user
    warning signal for each component of the heterogeneous job remain unique.
    Heterogeneous jobs are excluded from GANG scheduling operations.
    .RS
    .TP 12
    \fBOFF\fR
    Is the default value and disables job preemption and gang scheduling.
    It is only compatible with \fBPreemptType=preempt/none\fR at a global level.
    A common use case for this parameter is to set it on a partition to disable
    preemption for that partition.
    .TP
    \fBCANCEL\fR
    The preempted job will be cancelled.
    .TP
    \fBGANG\fR
    Enables gang scheduling (time slicing) of jobs in the same partition, and
    allows the resuming of suspended jobs.

    \fBNOTE:\fR
    Gang scheduling is performed independently for each partition, so
    if you only want time-slicing by \fBOverSubscribe\fR, without any preemption,
    then configuring partitions with overlapping nodes is not recommended.
    On the other hand, if you want to use \fBPreemptType=preempt/partition_prio\fR
    to allow jobs from higher PriorityTier partitions to Suspend jobs from lower
    PriorityTier partitions you will need overlapping partitions, and
    \fBPreemptMode=SUSPEND,GANG\fR to use the Gang scheduler to resume the suspended
    jobs(s).
    In any case, time-slicing won't happen between jobs on different partitions.

    \fBNOTE:\fR
    Heterogeneous jobs are excluded from GANG scheduling operations.
    .TP
    \fBREQUEUE\fR
    Preempts jobs by requeuing them (if possible) or canceling them.
    For jobs to be requeued they must have the \-\-requeue sbatch option set
    or the cluster wide JobRequeue parameter in slurm.conf must be set to one.
    .TP
    \fBSUSPEND\fR
    The preempted jobs will be suspended, and later the Gang scheduler will resume
    them. Therefore the \fBSUSPEND\fR preemption mode always needs the \fBGANG\fR
    option to be specified at the cluster level. Also, because the suspended jobs
    will still use memory on the allocated nodes, Slurm needs to be able to track
    memory resources to be able to suspend jobs.
    .TP
    \fBPreemptType\fR
    Specifies the plugin used to identify which jobs can be
    preempted in order to start a pending job.
    .RS
    .TP
    \fBpreempt/none\fR
    Job preemption is disabled.
    This is the default.
    .TP
    \fBpreempt/partition_prio\fR
    Job preemption is based upon partition \fBPriorityTier\fR.
    Jobs in higher \fBPriorityTier\fR partitions may preempt jobs from lower
    \fBPriorityTier\fR partitions.
    This is not compatible with \fBPreemptMode=OFF\fR.
    .TP
    \fBpreempt/qos\fR
    Job preemption rules are specified by Quality Of Service (QOS) specifications
    in the Slurm database.
    This option is not compatible with \fBPreemptMode=OFF\fR.
    A configuration of \fBPreemptMode=SUSPEND\fR is only supported by the
    \fBSelectType=select/cons_res\fR and \fBSelectType=select/cons_tres\fR plugins.
    See the \fBsacctmgr\fR man page to configure the options for \fBpreempt/qos\fR.
    .RE

    .TP
    \fBPreemptExemptTime\fR
    Global option for minimum run time for all jobs before they can be considered
    for preemption. Any QOS PreemptExemptTime takes precedence over the global
    option.
    A time of -1 disables the option, equivalent to 0. Acceptable time formats
    include "minutes", "minutes:seconds", "hours:minutes:seconds", "days\-hours",
    "days\-hours:minutes", and "days\-hours:minutes:seconds".

    .TP
    \fBPrEpParameters\fR
    Parameters to be passed to the \fBPrEpPlugins\fR.

    .TP
    \fBPrEpPlugins\fR
    A resource for programmers wishing to write their own plugins for the Prolog and
    Epilog (PrEp) scripts. The default, and currently the only implemented plugin is
    \fIprep/script\fR. Additional plugins can be specified in a comma-separated
    list. For more information please see the PrEp Plugin API documentation page:
    <https://slurm.schedmd.com/prep_plugins.html>

    .TP
    \fBPriorityCalcPeriod\fR
    The period of time in minutes in which the half-life decay will be
    re-calculated.
    Applicable only if PriorityType=priority/multifactor.
    The default value is 5 (minutes).

    .TP
    \fBPriorityFavorSmall\fR
    Specifies that small jobs should be given preferential scheduling priority.
    Applicable only if PriorityType=priority/multifactor.
    Supported values are "YES" and "NO".  The default value is "NO".

    .TP
    \fBPriorityFlags\fR
    Flags to modify priority behavior.
    Applicable only if PriorityType=priority/multifactor.
    The keywords below have no associated value
    (e.g. "PriorityFlags=ACCRUE_ALWAYS,SMALL_RELATIVE_TO_TIME").
    .RS
    .TP 17
    \fBACCRUE_ALWAYS\fR
    If set, priority age factor will be increased despite job dependencies
    or holds.
    .TP
    \fBCALCULATE_RUNNING\fR
    If set, priorities will be recalculated not only for pending jobs, but also
    running and suspended jobs.
    .TP
    \fBDEPTH_OBLIVIOUS\fR
    If set, priority will be calculated based similar to the normal multifactor
    calculation, but depth of the associations in the tree do not adversely effect
    their priority. This option automatically enables NO_FAIR_TREE.
    .TP
    \fBNO_FAIR_TREE\fR
    Disables the "fair tree" algorithm, and reverts to "classic" fair share
    priority scheduling.
    .TP
    \fBINCR_ONLY\fR
    If set, priority values will only increase in value. Job priority will never
    decrease in value.
    .TP
    \fBMAX_TRES\fR
    If set, the weighted TRES value (e.g. TRESBillingWeights) is calculated as the
    MAX of individual TRES' on a node (e.g. cpus, mem, gres) plus the sum of all
    global TRES' (e.g. licenses).
    .TP
    \fBNO_NORMAL_ALL\fR
    If set, all NO_NORMAL_* flags are set.
    .TP
    \fBNO_NORMAL_ASSOC\fR
    If set, the association factor is not normalized against the highest association
    priority.
    .TP
    \fBNO_NORMAL_PART\fR
    If set, the partition factor is not normalized against the highest partition
    \fBPriorityJobFactor\fR.
    .TP
    Specifies the job age which will be given the maximum age factor in computing
    priority. For example, a value of 30 minutes would result in all jobs over
    30 minutes old would get the same age\-based priority.
    Applicable only if PriorityType=priority/multifactor.
    The unit is a time string (i.e. min, hr:min:00, days\-hr:min:00,
    or days\-hr).  The default value is 7\-0 (7 days).

    .TP
    \fBPriorityParameters\fR
    Arbitrary string used by the PriorityType plugin.

    .TP
    \fBPrioritySiteFactorParameters\fR
    Arbitrary string used by the PrioritySiteFactorPlugin plugin.

    .TP
    \fBPrioritySiteFactorPlugin\fR
    The specifies an optional plugin to be used alongside "priority/multifactor",
    which is meant to initially set and continuously update the SiteFactor
    priority factor.
    The default value is "site_factor/none".

    .TP
    \fBPriorityType\fR
    This specifies the plugin to be used in establishing a job's scheduling
    priority.
    Also see \fBPriorityFlags\fR for configuration options.
    The default value is "priority/basic".
    .RS
    .TP
    \fBpriority/basic\fR
    Jobs are evaluated in a First In, First Out (FIFO) manner.
    .TP
    \fBpriority/multifactor\fR
    Jobs are assigned a priority based upon a variety of factors
    that include size, age, Fairshare, etc.
    .RE
    .RS
    .nr step 1 1
    When not FIFO scheduling, jobs are prioritized in the following order:
    .br

    1. Jobs that can preempt
    .br
    2. Jobs with an advanced reservation
    .br
    3. Partition PriorityTier
    .br
    4. Job priority
    .br
    5. Job submit time
    .br
    .TP 12
    \fBNONE\fR
    Never clear historic usage. The default value.
    .TP
    \fBNOW\fR
    Clear the historic usage now.
    Executed at startup and reconfiguration time.
    .TP
    \fBDAILY\fR
    Cleared every day at midnight.
    .TP
    \fBWEEKLY\fR
    Cleared every week on Sunday at time 00:00.
    .TP
    \fBMONTHLY\fR
    Cleared on the first day of each month at time 00:00.
    .TP
    \fBQUARTERLY\fR
    Cleared on the first day of each quarter at time 00:00.
    .TP
    \fBYEARLY\fR
    Cleared on the first day of each year at time 00:00.
    .RE

    .TP
    \fBPriorityWeightAge\fR
    An integer value that sets the degree to which the queue wait time
    component contributes to the job's priority.
    Applicable only if PriorityType=priority/multifactor.
    Requires AccountingStorageType=accounting_storage/slurmdbd.
    The default value is 0.

    .TP
    \fBPriorityWeightAssoc\fR
    An integer value that sets the degree to which the association
    component contributes to the job's priority.
    Applicable only if PriorityType=priority/multifactor.
    The default value is 0.

    .TP
    \fBPriorityWeightFairshare\fR
    An integer value that sets the degree to which the fair-share
    component contributes to the job's priority.
    Applicable only if PriorityType=priority/multifactor.
    Requires AccountingStorageType=accounting_storage/slurmdbd.
    The default value is 0.

    .TP
    \fBPriorityWeightJobSize\fR
    An integer value that sets the degree to which the job size
    component contributes to the job's priority.
    Applicable only if PriorityType=priority/multifactor.

    .TP
    \fBPriorityWeightTRES\fR
    A comma-separated list of TRES Types and weights that sets the degree that each
    TRES Type contributes to the job's priority.

    .nf
    e.g.
    PriorityWeightTRES=CPU=1000,Mem=2000,GRES/gpu=3000
    .fi

    Applicable only if PriorityType=priority/multifactor and if
    AccountingStorageTRES is configured with each TRES Type.
    Negative values are allowed.
    The default values are 0.

    .TP
    \fBPrivateData\fR
    This controls what type of information is hidden from regular users.
    By default, all information is visible to all users.
    User \fBSlurmUser\fR and \fBroot\fR can always view all information.
    Multiple values may be specified with a comma separator.
    Acceptable values include:
    .RS
    .TP
    \fBaccounts\fR
    (NON-SlurmDBD ACCOUNTING ONLY) Prevents users from viewing any account
    definitions unless they are coordinators of them.
    .TP
    \fBcloud\fR
    Powered down nodes in the cloud are visible.
    .TP
    \fBevents\fR
    prevents users from viewing event information unless they have operator status
    or above.
    .TP
    \fBjobs\fR
    Prevents users from viewing jobs or job steps belonging
    to other users. (NON-SlurmDBD ACCOUNTING ONLY) Prevents users from viewing
    job records belonging to other users unless they are coordinators of
    the association running the job when using sacct.
    .TP
    \fBnodes\fR
    Prevents users from viewing node state information.
    .TP
    \fBpartitions\fR
    Prevents users from viewing partition state information.
    .TP
    \fBreservations\fR
    Prevents regular users from viewing reservations which they can not use.
    .TP
    \fBusage\fR
    Identifies the plugin to be used for process tracking on a job step basis.
    The slurmd daemon uses this mechanism to identify all processes
    which are children of processes it spawns for a user job step.
    The slurmd daemon must be restarted for a change in ProctrackType
    to take effect.
    NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to
    identify all processes associated with a job since processes
    can become a child of the init process (when the parent process
    terminates) or change their process group.
    To reliably track all processes, "proctrack/cgroup" is highly recommended.
    NOTE: The \fBJobContainerType\fR applies to a job allocation, while
    \fBProctrackType\fR applies to job steps.
    Acceptable values at present include:
    .RS
    .TP
    \fBproctrack/cgroup\fR
    Uses linux cgroups to constrain and track processes, and is the default
    for systems with cgroup support.
    .br
    \fBNOTE\fR: see "man cgroup.conf" for configuration details.
    .TP
    \fBproctrack/cray_aries\fR
    Uses Cray proprietary process tracking.
    .TP
    \fBproctrack/linuxproc\fR
    Uses linux process tree using parent process IDs.
    .TP
    \fBproctrack/pgid\fR
    Uses Process Group IDs.
    .br
    \fBNOTE\fR: This is the default for the BSD family.
    .RE

    .TP
    \fBProlog\fR
    Fully qualified pathname of a program for the slurmd to execute
    whenever it is asked to run a job step from a new job allocation (e.g.
    "/usr/local/slurm/prolog"). A glob pattern (See \fBglob\fR (7)) may
    also be used to specify more than one program to run (e.g.
    "/etc/slurm/prolog.d/*"). The slurmd executes the prolog before starting
    the first job step.  The prolog script or scripts may be used to purge files,
    enable user login, etc.  By default there is no prolog. Any configured script
    is expected to complete execution quickly (in less time than
    \fBMessageTimeout\fR).
    If the prolog fails (returns a non\-zero exit code), this will result in the
    node being set to a DRAIN state and the job being requeued in a held state,
    unless \fBnohold_on_prolog_fail\fR is configured in
    \fBSchedulerParameters\fR.
    See \fBProlog and Epilog Scripts\fR for more information.

    .TP
    \fBPrologEpilogTimeout\fR
    If set, the Prolog script will be executed at job allocation. By default,
    Prolog is executed just before the task is launched. Therefore, when salloc
    is started, no Prolog is executed. Alloc is useful for preparing things
    before a user starts to use any allocated resources.
    In particular, this flag is needed on a Cray system when cluster compatibility
    mode is enabled.

    \fBNOTE: Use of the Alloc flag will increase the time required to start jobs.\fR
    .TP
    \fBContain\fR
    At job allocation time, use the ProcTrack plugin to create a job container
    on all allocated compute nodes.
    This container may be used for user processes not launched under Slurm control,
    for example pam_slurm_adopt may place processes launched through a direct user
    login into this container. If using pam_slurm_adopt, then ProcTrackType must be
    set to either \fBproctrack/cgroup\fR or \fBproctrack/cray_aries\fR.
    Setting the Contain implicitly sets the Alloc flag.
    .TP
    \fBNoHold\fR
    If set, the Alloc flag should also be set.  This will allow for salloc to not
    block until the prolog is finished on each node.  The blocking will happen when
    steps reach the slurmd and before any execution has happened in the step.
    This is a much faster way to work and if using srun to launch your tasks you
    should use this flag. This flag cannot be combined with the Contain or X11
    flags.
    .TP
    \fBSerial\fR
    By default, the Prolog and Epilog scripts run concurrently on each node.
    This flag forces those scripts to run serially within each node, but with
    a significant penalty to job throughput on each node.
    .TP
    \fBX11\fR
    Enable Slurm's built-in X11 forwarding capabilities.
    This is incompatible with \fBProctrackType=proctrack/linuxproc\fR.
    Setting the X11 flag implicitly enables both Contain and Alloc flags as well.
    .RE

    .TP
    \fBPrologSlurmctld\fR
    Fully qualified pathname of a program for the slurmctld daemon to execute
    before granting a new job allocation (e.g.
    "/usr/local/slurm/prolog_controller").
    The program executes as SlurmUser on the same node where the slurmctld daemon
    executes, giving it permission to drain
    nodes and requeue the job if a failure occurs or cancel the job if appropriate.
    The program can be used to reboot nodes or perform other work to prepare
    resources for use.
    Exactly what the program does and how it accomplishes this is completely at
    the discretion of the system administrator.
    Information about the job being initiated, its allocated nodes, etc. are
    passed to the program using environment variables.
    While this program is running, the nodes associated with the job will be
    Controls the scheduling priority (nice value) of user spawned tasks.
    .RS
    .TP 5
    \fB0\fR
    The tasks will inherit the scheduling priority from the slurm daemon.
    This is the default value.
    .TP
    \fB1\fR
    The tasks will inherit the scheduling priority of the command used to
    submit them (e.g. \fBsrun\fR or \fBsbatch\fR).
    Unless the job is submitted by user root, the tasks will have a scheduling
    priority no higher than the slurm daemon spawning them.
    .TP
    \fB2\fR
    The tasks will inherit the scheduling priority of the command used to
    submit them (e.g. \fBsrun\fR or \fBsbatch\fR) with the restriction that
    their nice value will always be one higher than the slurm daemon (i.e.
    the tasks scheduling priority will be lower than the slurm daemon).
    .RE

    .TP
    \fBPropagateResourceLimits\fR
    A comma-separated list of resource limit names.
    The slurmd daemon uses these names to obtain the associated (soft) limit
    values from the user's process environment on the submit node.
    These limits are then propagated and applied to the jobs that
    will run on the compute nodes.
    This parameter can be useful when system limits vary among nodes.
    Any resource limits that do not appear in the list are not propagated.
    However, the user can override this by specifying which resource limits
    to propagate with the sbatch or srun "\-\-propagate" option. If neither
    \fBPropagateResourceLimits\fR or \fBPropagateResourceLimitsExcept\fR are
    configured and the "\-\-propagate" option is not specified, then the default
    action is to propagate all limits. Only one of the parameters, either
    PropagateResourceLimits or PropagateResourceLimitsExcept, may be specified.
    The user limits can not exceed hard limits under which the slurmd daemon
    operates. If the user limits are not propagated, the limits from the slurmd
    daemon will be propagated to the user's job. The limits used for the Slurm
    daemons can be set in the /etc/sysconf/slurm file. For more information, see:
    https://slurm.schedmd.com/faq.html#memlock
    The following limit names are supported by Slurm (although some
    options may not be supported on some systems):
    .RS
    .TP 10
    \fBALL\fR
    All limits listed below (default)
    .TP
    \fBNONE\fR
    No limits listed below
    .TP
    \fBAS\fR
    The maximum address space for a process
    .TP
    \fBMEMLOCK\fR
    The maximum size that may be locked into memory
    .TP
    \fBNOFILE\fR
    The maximum number of open files
    .TP
    \fBNPROC\fR
    The maximum number of processes available
    .TP
    \fBRSS\fR
    The maximum resident set size
    .TP
    \fBSTACK\fR
    The maximum stack size
    .RE

    .TP
    \fBPropagateResourceLimitsExcept\fR
    A comma-separated list of resource limit names.
    By default, all resource limits will be propagated, (as described by
    the \fBPropagateResourceLimits\fR parameter), except for the limits
    appearing in this list.   The user can override this by specifying which
    resource limits to propagate with the sbatch or srun "\-\-propagate" option.
    See \fBPropagateResourceLimits\fR above for a list of valid limit names.

    .TP
    \fBRebootProgram\fR
    Program to be executed on each compute node to reboot it. Invoked on each node
    once it becomes idle after the command "scontrol reboot" is executed by
    an authorized user or a job is submitted with the "\-\-reboot" option.
    After rebooting, the node is returned to normal use.
    See \fBResumeTimeout\fR to configure the time you expect a reboot to finish in.
    A node will be marked DOWN if it doesn't reboot within \fBResumeTimeout\fR.

    .TP
    \fBReconfigFlags\fR
    Flags to control various actions that may be taken when an "scontrol
    reconfig" command is issued. Currently the options are:
    .RS
    .TP 17
    \fBKeepPartInfo\fR
    If set, an "scontrol reconfig" command will maintain the in\-memory
    value of partition "state" and other parameters that may have been
    dynamically updated by "scontrol update".  Partition information in
    the slurm.conf file will be merged with in\-memory data.  This flag
    supersedes the KeepPartState flag.
    .TP
    \fBKeepPartState\fR
    If set, an "scontrol reconfig" command will preserve only the current
    "state" value of in\-memory partitions and will reset all other
    parameters of the partitions that may have been dynamically updated by
    Separate multiple exit code by a comma and/or specify numeric ranges using a
    "\-" separator (e.g. "RequeueExit=1\-9,18")
    Jobs will be put back in to pending state and later scheduled again.
    Restarted jobs will have the environment variable \fBSLURM_RESTART_COUNT\fP
    set to the number of times the job has been restarted.

    .TP
    \fBRequeueExitHold\fR
    Enables automatic requeue for batch jobs which exit with the specified
    values, with these jobs being held until released manually by the user.
    Separate multiple exit code by a comma and/or specify numeric ranges using a
    "\-" separator (e.g. "RequeueExitHold=10\-12,16")
    These jobs are put in the \fBJOB_SPECIAL_EXIT\fP exit state.
    Restarted jobs will have the environment variable \fBSLURM_RESTART_COUNT\fP
    set to the number of times the job has been restarted.

    .TP
    \fBResumeFailProgram\fR
    The program that will be executed when nodes fail to resume to by
    \fBResumeTimeout\fR. The argument to the program will be the names of the failed
    nodes (using Slurm's hostlist expression format).

    .TP
    \fBResumeProgram\fR
    Slurm supports a mechanism to reduce power consumption on nodes that
    remain idle for an extended period of time.
    This is typically accomplished by reducing voltage and frequency or powering
    the node down.
    \fBResumeProgram\fR is the program that will be executed when a node
    in power save mode is assigned work to perform.
    For reasons of reliability, \fBResumeProgram\fR may execute more than once
    for a node when the \fBslurmctld\fR daemon crashes and is restarted.
    If \fBResumeProgram\fR is unable to restore a node to service with a responding
    slurmd and an updated BootTime, it should requeue any job associated with the
    node and set the node state to DOWN. If the node isn't actually rebooted
    (i.e. when multiple-slurmd is configured) starting slurmd with "-b" option
    might be useful.
    The program executes as \fBSlurmUser\fR.
    The argument to the program will be the names of nodes to
    be removed from power savings mode (using Slurm's hostlist
    expression format).
    By default no program is run.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeRate\fR,
    \fBSuspendRate\fR, \fBSuspendTime\fR, \fBSuspendTimeout\fR, \fBSuspendProgram\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.
    More information is available at the Slurm web site
    ( https://slurm.schedmd.com/power_save.html ).

    .TP
    \fBResumeRate\fR
    The rate at which nodes in power save mode are returned to normal
    operation by \fBResumeProgram\fR.
    the jobs scheduled on the node requeued.
    Nodes which reboot after this time frame will be marked DOWN with a reason of
    "Node unexpectedly rebooted."
    The default value is 60 seconds.
    Related configuration options include \fBResumeProgram\fR, \fBResumeRate\fR,
    \fBSuspendRate\fR, \fBSuspendTime\fR, \fBSuspendTimeout\fR, \fBSuspendProgram\fR,
    \fBSuspendExcNodes\fR and \fBSuspendExcParts\fR.
    More information is available at the Slurm web site
    ( https://slurm.schedmd.com/power_save.html ).

    .TP
    \fBResvEpilog\fR
    Fully qualified pathname of a program for the slurmctld to execute
    when a reservation ends. The program can be used to cancel jobs, modify
    partition configuration, etc.
    The reservation named will be passed as an argument to the program.
    By default there is no epilog.

    .TP
    \fBResvOverRun\fR
    Describes how long a job already running in a reservation should be
    permitted to execute after the end time of the reservation has been
    reached.
    The time period is specified in minutes and the default value is 0
    (kill the job immediately).
    The value may not exceed 65533 minutes, although a value of "UNLIMITED"
    is supported to permit a job to run indefinitely after its reservation
    is terminated.

    .TP
    \fBResvProlog\fR
    Fully qualified pathname of a program for the slurmctld to execute
    when a reservation begins. The program can be used to cancel jobs, modify
    partition configuration, etc.
    The reservation named will be passed as an argument to the program.
    By default there is no prolog.

    .TP
    \fBReturnToService\fR
    Controls when a DOWN node will be returned to service.
    The default value is 0.
    Supported values include
    .RS
    .TP 4
    \fB0\fR
    A node will remain in the DOWN state until a system administrator
    explicitly changes its state (even if the slurmd daemon registers
    and resumes communications).
    .TP
    \fB1\fR
    A DOWN node will become available for use upon registration with a
    valid configuration only if it was set DOWN due to being non\-responsive.
    .TP
    \fBRoutePlugin\fR
    Identifies the plugin to be used for defining which nodes will be used
    for message forwarding.
    .RS
    .TP
    \fBroute/default\fR
    default, use TreeWidth.
    .TP
    \fBroute/topology\fR
    use the switch hierarchy defined in a \fItopology.conf\fR file.
    TopologyPlugin=topology/tree is required.
    .RE

    .TP
    \fBSbcastParameters\fR
    Controls sbcast command behavior. Multiple options can be specified in a comma
    separated list.
    Supported values include:
    .RS
    .TP 15
    \fBDestDir=\fR
    Destination directory for file being broadcast to allocated compute nodes.
    Default value is current working directory.
    .TP
    \fBCompression=\fR
    Specify default file compression library to be used.
    Supported values are "lz4", "none" and "zlib".
    The default value with the sbcast \-\-compress option is "lz4" and "none" otherwise.
    Some compression libraries may be unavailable on some systems.
    .RE

    .TP
    \fBSchedulerParameters\fR
    The interpretation of this parameter varies by \fBSchedulerType\fR.
    Multiple options may be comma separated.
    .RS
    .TP
    \fBallow_zero_lic\fR
    If set, then job submissions requesting more than configured licenses won't be
    rejected.
    .TP
    \fBassoc_limit_stop\fR
    If set and a job cannot start due to association limits, then do not attempt
    to initiate any lower priority jobs in that partition. Setting this can
    decrease system throughput and utilization, but avoid potentially starving larger
    jobs by preventing them from launching indefinitely.
    .TP
    \fBbatch_sched_delay=#\fR
    How long, in seconds, the scheduling of batch jobs can be delayed.
    This can be useful in a high\-throughput environment in which batch jobs are
    submitted at a very high rate (i.e. using the sbatch command) and one wishes
    This will tend to leave currently idle resources available for backfilling
    longer running jobs, but may result in allocations having less than optimal
    network topology.
    This option is currently only supported by the select/cons_res and
    select/cons_tres plugins (or select/cray_aries with SelectTypeParameters set to
    "OTHER_CONS_RES" or "OTHER_CONS_TRES", which layers the select/cray_aries plugin
    over the select/cons_res or select/cons_tres plugin respectively).
    .TP
    \fBbf_continue\fR
    The backfill scheduler periodically releases locks in order to permit other
    operations to proceed rather than blocking all activity for what could be an
    extended period of time.
    Setting this option will cause the backfill scheduler to continue processing
    pending jobs from its original job list after releasing locks even if job
    or node state changes.
    .TP
    \fBbf_hetjob_immediate\fR
    Instruct the backfill scheduler to attempt to start a heterogeneous job as
    soon as all of its components are determined able to do so. Otherwise, the
    backfill scheduler will delay heterogeneous jobs initiation attempts until
    after the rest of the queue has been processed. This delay may result in lower
    priority jobs being allocated resources, which could delay the initiation of
    the heterogeneous job due to account and/or QOS limits being reached. This
    option is disabled by default. If enabled and \fBbf_hetjob_prio=min\fR is not
    set, then it would be automatically set.
    .TP
    \fBbf_hetjob_prio=[min|avg|max]\fR
    At the beginning of each backfill scheduling cycle, a list of pending to be
    scheduled jobs is sorted according to the precedence order configured in
    \fBPriorityType\fR. This option instructs the scheduler to alter the sorting
    algorithm to ensure that all components belonging to the same heterogeneous job
    will be attempted to be scheduled consecutively (thus not fragmented in the
    resulting list). More specifically, all components from the same heterogeneous
    job will be treated as if they all have the same priority (minimum, average or
    maximum depending upon this option's parameter) when compared with other jobs
    (or other heterogeneous job components). The original order will be preserved
    within the same heterogeneous job. Note that the operation is calculated for
    the \fBPriorityTier\fR layer and for the \fBPriority\fR resulting from the
    priority/multifactor plugin calculations. When enabled, if any heterogeneous job
    requested an advanced reservation, then all of that job's components will be
    treated as if they had requested an advanced reservation (and get
    preferential treatment in scheduling).

    Note that this operation does not update the \fBPriority\fR values of the
    heterogeneous job components, only their order within the list, so the output of
    the sprio command will not be effected.

    Heterogeneous jobs have special scheduling properties: they are only scheduled
    by the backfill scheduling plugin, each of their components is considered
    separately when reserving resources (and might have different \fBPriorityTier\fR
    or different \fBPriority\fR values), and no heterogeneous job component is
    actually allocated resources until all if its components can be initiated.
    The backfill scheduling logic will reserve resources for the specified count
    of highest priority jobs in each partition.
    For example, bf_job_part_count_reserve=10 will cause the backfill scheduler to
    reserve resources for the ten highest priority jobs in each partition.
    Any lower priority job that can be started using currently available resources
    and not adversely impact the expected start time of these higher priority jobs
    will be started by the backfill scheduler
    The default value is zero, which will reserve resources for any pending job
    and delay initiation of lower priority jobs.
    Also see bf_min_age_reserve and bf_min_prio_reserve.
    Default: 0, Min: 0, Max: 100000.

    .TP
    \fBbf_max_job_array_resv=#\fR
    The maximum number of tasks from a job array for which the backfill scheduler
    will reserve resources in the future.
    Since job arrays can potentially have millions of tasks, the overhead in
    reserving resources for all tasks can be prohibitive.
    In addition various limits may prevent all the jobs from starting at the
    expected times.
    This has no impact upon the number of tasks from a job array that can be
    started immediately, only those tasks expected to start at some future time.
    Default: 20, Min: 0, Max: 1000.
    NOTE:
    Jobs submitted to multiple partitions appear in the job queue once per
    partition. If different copies of a single job array record aren't consecutive
    in the job queue and another job array record is in between, then
    bf_max_job_array_resv tasks are considered per partition that the job is
    submitted to.
    .TP
    \fBbf_max_job_assoc=#\fR
    The maximum number of jobs per user association to attempt starting with the
    backfill scheduler.
    This setting is similar to \fBbf_max_job_user\fR but is handy if a user
    has multiple associations equating to basically different users.
    One can set this limit to prevent users from flooding the backfill
    queue with jobs that cannot start and that prevent jobs from other users
    to start.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBbf_max_job_user\fR \fBbf_max_job_part\fR, \fBbf_max_job_test\fR
    and \fBbf_max_job_user_part=#\fR options.
    Set \fBbf_max_job_test\fR to a value much higher than \fBbf_max_job_assoc\fR.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    .TP
    \fBbf_max_job_part=#\fR
    The maximum number of jobs per partition to attempt starting with the backfill
    scheduler. This can be especially helpful for systems with large numbers of
    partitions and jobs.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBpartition_job_depth\fR and \fBbf_max_job_test\fR options.
    Set \fBbf_max_job_test\fR to a value much higher than \fBbf_max_job_part\fR.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    desirable.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 100, Min: 1, Max: 1,000,000.
    .TP
    \fBbf_max_job_user=#\fR
    The maximum number of jobs per user to attempt starting with the backfill
    scheduler for ALL partitions.
    One can set this limit to prevent users from flooding the backfill
    queue with jobs that cannot start and that prevent jobs from other users
    to start.  This is similar to the MAXIJOB limit in Maui.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBbf_max_job_part\fR, \fBbf_max_job_test\fR and
    \fBbf_max_job_user_part=#\fR options.
    Set \fBbf_max_job_test\fR to a value much higher than \fBbf_max_job_user\fR.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    .TP
    \fBbf_max_job_user_part=#\fR
    The maximum number of jobs per user per partition to attempt starting with the
    backfill scheduler for any single partition.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBbf_max_job_part\fR, \fBbf_max_job_test\fR and
    \fBbf_max_job_user=#\fR options.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    .TP
    \fBbf_max_time=#\fR
    The maximum time in seconds the backfill scheduler can spend (including time
    spent sleeping when locks are released) before discontinuing, even if maximum
    job counts have not been reached.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    The default value is the value of bf_interval (which defaults to 30 seconds).
    Default: bf_interval value (def. 30 sec), Min: 1, Max: 3600 (1h).
    NOTE: If bf_interval is short and bf_max_time is large, this may cause locks
    to be acquired too frequently and starve out other serviced RPCs. It's
    advisable if using this parameter to set max_rpc_cnt high enough that
    scheduling isn't always disabled, and low enough that the interactive
    workload can get through in a reasonable period of time. max_rpc_cnt needs to
    be below 256 (the default RPC thread limit). Running around the middle (150)
    may give you good results.
    NOTE: When increasing the amount of time spent in the backfill scheduling cycle,
    Slurm can be prevented from responding to client requests in a timely manner.
    To address this you can use \fBmax_rpc_cnt\fR to specify a number of queued RPCs
    before the scheduler stops to respond to these requests.
    .TP
    \fBbf_min_age_reserve=#\fR
    The backfill and main scheduling logic will not reserve resources for pending
    jobs until they have been pending and runnable for at least the specified
    number of seconds.
    In addition, jobs waiting for less than the specified number of seconds will
    not prevent a newly submitted job from starting immediately, even if the newly
    submitted job has a lower priority.
    This can be valuable if jobs lack time limits or all time limits have the same
    value.
    Also see bf_job_part_count_reserve and bf_min_age_reserve.
    Default: 0, Min: 0, Max: 2^63.
    .TP
    \fBbf_one_resv_per_job\fR
    Disallow adding more than one backfill reservation per job.
    The scheduling logic builds a sorted list of (job, partition) pairs. Jobs
    submitted to multiple partitions have as many entries in the list as requested
    partitions. By default, the backfill scheduler may evaluate all the
    (job, partition) entries for a single job, potentially reserving resources for
    each pair, but only starting the job in the reservation offering the earliest
    start time.
    Having a single job reserving resources for multiple partitions could impede
    other jobs (or hetjob components) from reserving resources already reserved for
    the reservations related to the partitions that don't offer the earliest start
    time.
    This option makes it so that a job submitted to multiple partitions will stop
    reserving resources once the first (job, partition) pair has booked a backfill
    reservation. Subsequent pairs from the same job will only be tested to start
    now. This allows for other jobs to be able to book the other pairs resources at
    the cost of not guaranteeing that the multi partition job will start in the
    partition offering the earliest start time (except if it can start now).
    This option is disabled by default.

    .TP
    \fBbf_resolution=#\fR
    The number of seconds in the resolution of data maintained about when jobs
    begin and end. Higher values result in better responsiveness and quicker
    backfill cycles by using larger blocks of time to determine node eligibility.
    However, higher values lead to less efficient system planning, and may miss
    opportunities to improve system utilization.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 60, Min: 1, Max: 3600 (1 hour).
    .TP
    \fBbf_running_job_reserve\fR
    Add an extra step to backfill logic, which creates backfill reservations
    for jobs running on whole nodes.
    This option is disabled by default.
    .TP
    \fBbf_window=#\fR
    The number of minutes into the future to look when considering jobs to schedule.
    Higher values result in more overhead and less responsiveness.
    A value at least as long as the highest allowed time limit is generally
    advisable to prevent job starvation.
    In order to limit the amount of data managed by the backfill scheduler,
    if the value of \fBbf_window\fR is increased, then it is generally advisable
    to also increase \fBbf_resolution\fR.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 1440 (1 day), Min: 1, Max: 43200 (30 days).
    .TP
    \fBbf_window_linear=#\fR
    For performance reasons, the backfill scheduler will decrease precision in
    calculation of job expected termination times. By default, the precision starts
    Without this option, the time windows will double on each iteration and thus
    be 30, 60, 120, 240 seconds, etc. The use of bf_window_linear is not recommended
    with more than a few hundred simultaneously executing jobs.
    .TP
    \fBbf_yield_interval=#\fR
    The backfill scheduler will periodically relinquish locks in order for other
    pending operations to take place.
    This specifies the times when the locks are relinquished in microseconds.
    Smaller values may be helpful for high throughput computing when used in
    conjunction with the \fBbf_continue\fR option.
    Also see the \fBbf_yield_sleep\fR option.
    Default: 2,000,000 (2 sec), Min: 1, Max: 10,000,000 (10 sec).
    .TP
    \fBbf_yield_sleep=#\fR
    The backfill scheduler will periodically relinquish locks in order for other
    pending operations to take place.
    This specifies the length of time for which the locks are relinquished in
    microseconds.
    Also see the \fBbf_yield_interval\fR option.
    Default: 500,000 (0.5 sec), Min: 1, Max: 10,000,000 (10 sec).
    .TP
    \fBbuild_queue_timeout=#\fR
    Defines the maximum time that can be devoted to building a queue of jobs to
    be tested for scheduling.
    If the system has a huge number of jobs with dependencies, just building the
    job queue can take so much time as to adversely impact overall system
    performance and this parameter can be adjusted as needed.
    The default value is 2,000,000 microseconds (2 seconds).
    .TP
    \fBcorrespond_after_task_cnt=#\fR
    Defines the number of array tasks that get split for potential aftercorr
    dependency check. Low number may result in dependent task check failures when
    the job one depends on gets purged before the split.
    Default: 10.
    .TP
    \fBdefault_queue_depth=#\fR
    The default number of jobs to attempt scheduling (i.e. the queue depth) when a
    running job completes or other routine actions occur, however the frequency
    with which the scheduler is run may be limited by using the \fBdefer\fR or
    \fBsched_min_interval\fR parameters described below.
    The full queue will be tested on a less frequent basis as defined by the
    \fBsched_interval\fR option described below. The default value is 100.
    See the \fBpartition_job_depth\fR option to limit depth by partition.
    .TP
    \fBdefer\fR
    Setting this option will avoid attempting to schedule each job
    individually at job submit time, but defer it until a later time when
    scheduling multiple jobs simultaneously may be possible.
    This option may improve system responsiveness when large numbers of jobs
    (many hundreds) are submitted at the same time, but it will delay the
    initiation time of individual jobs. Also see \fBdefault_queue_depth\fR above.
    .TP
    Disable job steps that span heterogeneous job allocations.
    The default value on Cray systems.
    .TP
    \fBenable_hetjob_steps\fR
    Enable job steps that span heterogeneous job allocations.
    The default value except for Cray systems.
    .TP
    \fBenable_user_top\fR
    Enable use of the "scontrol top" command by non-privileged users.
    .TP
    \fBIgnore_NUMA\fR
    Some processors (e.g. AMD Opteron 6000 series) contain multiple NUMA nodes per
    socket. This is a configuration which does not map into the hardware entities
    that Slurm optimizes resource allocation for (PU/thread, core, socket,
    baseboard, node and network switch). In order to optimize resource allocations
    on such hardware, Slurm will consider each NUMA node within the socket as a
    separate socket by default. Use the Ignore_NUMA option to report the correct
    socket count, but \fBnot\fR optimize resource allocations on the NUMA nodes.
    .TP
    \fBinventory_interval=#\fR
    On a Cray system using Slurm on top of ALPS this limits the number of times
    a Basil Inventory call is made.  Normally this call happens every scheduling
    consideration to attempt to close a node state change window with respects to
    what ALPS has.  This call is rather slow, so making it less frequently improves
    performance dramatically, but in the situation where a node changes state the
    window is as large as this setting.  In an HTC environment this setting is a
    must and we advise around 10 seconds.
    .TP
    \fBmax_array_tasks\fR
    Specify the maximum number of tasks that can be included in a job array.
    The default limit is MaxArraySize, but this option can be used to set a lower
    limit. For example, max_array_tasks=1000 and MaxArraySize=100001 would permit
    a maximum task ID of 100000, but limit the number of tasks in any single job
    array to 1000.
    .TP
    \fBmax_rpc_cnt=#\fR
    If the number of active threads in the slurmctld daemon is equal to or
    larger than this value, defer scheduling of jobs. The scheduler will check
    this condition at certain points in code and yield locks if necessary.
    This can improve Slurm's ability to process requests at a cost of initiating
    new jobs less frequently. Default: 0 (option disabled), Min: 0, Max: 1000.

    .RS
    NOTE: The maximum number of threads (MAX_SERVER_THREADS) is internally set to
    256 and defines the number of served RPCs at a given time. Setting max_rpc_cnt
    to more than 256 will be only useful to let backfill continue scheduling work
    after locks have been yielded (i.e. each 2 seconds) if there are a maximum of
    MAX(max_rpc_cnt/10, 20) RPCs in the queue. i.e. max_rpc_cnt=1000, the scheduler
    will be allowed to continue after yielding locks only when there are less than
    or equal to 100 pending RPCs.
    If a value is set, then a value of 10 or higher is recommended. It may require
    some tuning for each system, but needs to be high enough that scheduling isn't
    (i.e. MIN(10/2, 2) = 2).
    .TP
    \fBmax_script_size=#\fR
    Specify the maximum size of a batch script, in bytes.
    The default value is 4 megabytes.
    Larger values may adversely impact system performance.
    .TP
    \fBmax_switch_wait=#\fR
    Maximum number of seconds that a job can delay execution waiting for the
    specified desired switch count. The default value is 300 seconds.
    .TP
    \fBno_backup_scheduling\fR
    If used, the backup controller will not schedule jobs when it takes over. The
    backup controller will allow jobs to be submitted, modified and cancelled but
    won't schedule new jobs. This is useful in Cray environments when the backup
    controller resides on an external Cray node.  A restart is required to alter
    this option. This is explicitly set on a Cray/ALPS system.
    .TP
    \fBno_env_cache\fR
    If used, any job started on node that fails to load the env from a node will
    fail instead of using the cached env.  This will also implicitly imply the
    requeue_setup_env_fail option as well.
    .TP
    \fBnohold_on_prolog_fail\fR
    By default, if the Prolog exits with a non-zero value the job is requeued in
    a held state. By specifying this parameter the job will be requeued but not
    held so that the scheduler can dispatch it to another host.
    .TP
    \fBpack_serial_at_end\fR
    If used with the select/cons_res or select/cons_tres plugin,
    then put serial jobs at the end of
    the available nodes rather than using a best fit algorithm.
    This may reduce resource fragmentation for some workloads.
    .TP
    \fBpartition_job_depth=#\fR
    The default number of jobs to attempt scheduling (i.e. the queue depth)
    from each partition/queue in Slurm's main scheduling logic.
    The functionality is similar to that provided by the \fBbf_max_job_part\fR
    option for the backfill scheduling logic.
    The default value is 0 (no limit).
    Job's excluded from attempted scheduling based upon partition will not be
    counted against the \fBdefault_queue_depth\fR limit.
    Also see the \fBbf_max_job_part\fR option.
    .TP
    \fBpermit_job_expansion\fR
    Allow running jobs to request additional nodes be merged in with the current
    job allocation.
    .TP
    \fBpreempt_reorder_count=#\fR
    Specify how many attempts should be made in reordering preemptable jobs to
    minimize the count of jobs preempted.
    The default value is 1. High values may adversely impact performance.
    preempt/partition_prio or preempt/qos plugins.)
    .TP
    \fBreduce_completing_frag\fR
    This option is used to control how scheduling of resources is performed when
    jobs are in the COMPLETING state, which influences potential fragmentation.
    If this option is not set then no jobs will be started in any partition when
    any job is in the COMPLETING state for less than \fBCompleteWait\fR seconds.
    If this option is set then no jobs will be started in any individual partition
    that has a job in COMPLETING state for less than \fBCompleteWait\fR seconds.
    In addition, no jobs will be started in any partition with nodes that overlap
    with any nodes in the partition of the completing job.
    This option is to be used in conjunction with \fBCompleteWait\fR.

    \fBNOTE\fR: \fBCompleteWait\fR must be set in order for this to work. If
    \fBCompleteWait=0\fR then this option does nothing.

    \fBNOTE\fR: \fBreduce_completing_frag\fR only affects the main scheduler, not
    the backfill scheduler.
    .TP
    \fBrequeue_setup_env_fail\fR
    By default if a job environment setup fails the job keeps running with
    a limited environment. By specifying this parameter the job will be
    requeued in held state and the execution node drained.
    .TP
    \fBsalloc_wait_nodes\fR
    If defined, the salloc command will wait until all allocated nodes are ready for
    use (i.e. booted) before the command returns. By default, salloc will return as
    soon as the resource allocation has been made.
    .TP
    \fBsbatch_wait_nodes\fR
    If defined, the sbatch script will wait until all allocated nodes are ready for
    use (i.e. booted) before the initiation. By default, the sbatch script will be
    initiated as soon as the first node in the job allocation is ready. The sbatch
    command can use the \-\-wait\-all\-nodes option to override this configuration
    parameter.
    .TP
    \fBsched_interval=#\fR
    How frequently, in seconds, the main scheduling loop will execute and test all
    pending jobs.
    The default value is 60 seconds.
    .TP
    \fBsched_max_job_start=#\fR
    The maximum number of jobs that the main scheduling logic will start in any
    single execution.
    The default value is zero, which imposes no limit.
    .TP
    \fBsched_min_interval=#\fR
    How frequently, in microseconds, the main scheduling loop will execute and test
    any pending jobs.
    The scheduler runs in a limited fashion every time that any event happens which
    could enable a job to start (e.g. job submit, job terminate, etc.).
    If these events happen at a high frequency, the scheduler can run very
    When a step completes and there are steps ending resource allocation, then
    retry step allocations for at least this number of pending steps.
    Also see \fBstep_retry_time\fR.
    The default value is 8 steps.
    .TP
    \fBstep_retry_time=#\fR
    When a step completes and there are steps ending resource allocation, then
    retry step allocations for all steps which have been pending for at least this
    number of seconds.
    Also see \fBstep_retry_count\fR.
    The default value is 60 seconds.
    .TP
    \fBwhole_hetjob\fR
    Requests to cancel, hold or release any component of a heterogeneous job will
    be applied to all components of the job.

    NOTE: this option was previously named whole_pack and this is still supported
    for retrocompatibility.
    .RE

    .TP
    \fBSchedulerTimeSlice\fR
    Number of seconds in each time slice when gang scheduling is enabled
    (\fBPreemptMode=SUSPEND,GANG\fR).
    The value must be between 5 seconds and 65533 seconds.
    The default value is 30 seconds.

    .TP
    \fBSchedulerType\fR
    Identifies the type of scheduler to be used.
    Note the \fBslurmctld\fR daemon must be restarted for a change in
    scheduler type to become effective (reconfiguring a running daemon has
    no effect for this parameter).
    The \fBscontrol\fR command can be used to manually change job priorities
    if desired.
    Acceptable values include:
    .RS
    .TP
    \fBsched/backfill\fR
    For a backfill scheduling module to augment the default FIFO scheduling.
    Backfill scheduling will initiate lower\-priority jobs if doing
    so does not delay the expected initiation time of any higher
    priority job.
    Effectiveness of backfill scheduling is dependent upon users specifying
    job time limits, otherwise all jobs will have the same time limit and
    backfilling is impossible.
    Note documentation for the \fBSchedulerParameters\fR option above.
    This is the default configuration.
    .TP
    \fBsched/builtin\fR
    This is the FIFO scheduler which initiates jobs in priority order.
    If any job in the partition can not be scheduled, no lower priority job in that
    .RS
    .TP
    \fBenable\fR
    Enable the use of scrontab to submit and manage periodic repeating jobs.
    .RE

    .TP
    \fBSelectType\fR
    Identifies the type of resource selection algorithm to be used.
    Changing this value can only be done by restarting the slurmctld daemon.
    When changed, all job information (running and pending) will be lost,
    since the job state save format used by each plugin is different.
    The only exception to this is when changing from cons_res to cons_tres or from
    cons_tres to cons_res. However, if a job contains cons_tres-specific features
    and then SelectType is changed to cons_res, the job will be canceled, since
    there is no way for cons_res to satisfy requirements specific to cons_tres.

    Acceptable values include
    .RS
    .TP
    \fBselect/cons_res\fR
    The resources (cores and memory) within a node are individually allocated as
    consumable resources.
    Note that whole nodes can be allocated to jobs for selected
    partitions by using the \fIOverSubscribe=Exclusive\fR option.
    See the partition \fBOverSubscribe\fR parameter for more information.
    .TP
    \fBselect/cons_tres\fR
    The resources (cores, memory, GPUs and all other trackable resources) within
    a node are individually allocated as consumable resources.
    Note that whole nodes can be allocated to jobs for selected
    partitions by using the \fIOverSubscribe=Exclusive\fR option.
    See the partition \fBOverSubscribe\fR parameter for more information.
    .TP
    \fBselect/cray_aries\fR
    for a Cray system.
    The default value is "select/cray_aries" for all Cray systems.
    .TP
    \fBselect/linear\fR
    for allocation of entire nodes assuming a one\-dimensional array of nodes in
    which sequentially ordered nodes are preferable.
    For a heterogeneous cluster (e.g. different CPU counts on the various nodes),
    resource allocations will favor nodes with high CPU counts as needed based upon
    the job's node and CPU specification if \fBTopologyPlugin=topology/none\fR is
    configured. Use of other topology plugins with select/linear and heterogeneous
    nodes is not recommended and may result in valid job allocation requests being
    rejected.
    This is the default value.
    .RE

    .TP
    \fBSelectTypeParameters\fR
    The following options are supported for \fBSelectType=select/cray_aries\fR:
    .RS
    .TP
    \fBOTHER_CONS_RES\fR
    Layer the select/cons_res plugin under the select/cray_aries plugin, the default is
    to layer on select/linear.  This also allows all the options available for
    \fBSelectType=select/cons_res\fR.
    .TP
    \fBOTHER_CONS_TRES\fR
    Layer the select/cons_tres plugin under the select/cray_aries plugin, the default is
    to layer on select/linear.  This also allows all the options available for
    \fBSelectType=select/cons_tres\fR.
    .RE

    The following options are supported by the \fBSelectType=select/cons_res\fR
    and \fBSelectType=select/cons_tres\fR plugins:
    .RS
    .TP
    \fBCR_CPU\fR
    CPUs are consumable resources.
    Configure the number of \fBCPUs\fR on each node, which may be equal to the
    count of cores or hyper\-threads on the node depending upon the desired minimum
    resource allocation.
    The node's \fBBoards\fR, \fBSockets\fR, \fBCoresPerSocket\fR and
    \fBThreadsPerCore\fR may optionally be configured and result in job
    allocations which have improved locality; however doing so will prevent
    more than one job from being allocated on each core.
    .TP
    \fBCR_CPU_Memory\fR
    CPUs and memory are consumable resources.
    Configure the number of \fBCPUs\fR on each node, which may be equal to the
    count of cores or hyper\-threads on the node depending upon the desired minimum
    resource allocation.
    The node's \fBBoards\fR, \fBSockets\fR, \fBCoresPerSocket\fR and
    \fBThreadsPerCore\fR may optionally be configured and result in job
    allocations which have improved locality; however doing so will prevent
    more than one job from being allocated on each core.
    Setting a value for \fBDefMemPerCPU\fR is strongly recommended.
    .TP
    \fBCR_Core\fR
    Cores are consumable resources.
    On nodes with hyper\-threads, each thread is counted as a CPU to
    satisfy a job's resource requirement, but multiple jobs are not
    allocated threads on the same core.
    The count of CPUs allocated to a job is rounded up to account for every
    CPU on an allocated core. This will also impact total allocated memory when
    --mem-per-cpu is used to be multiply of total number of CPUs on allocated cores.
    .TP
    \fBCR_Core_Memory\fR
    Cores and memory are consumable resources.
    On nodes with hyper\-threads, each thread is counted as a CPU to
    satisfy a job's resource requirement, but multiple jobs are not
    boards and minimizes the number of sockets (within minimum boards)
    used for the allocation.
    This default behavior can be overridden specifying a particular
    "\-m" parameter with srun/salloc/sbatch.
    Without this option, cores will be allocated cyclically across the sockets.
    .TP
    \fBCR_LLN\fR
    Schedule resources to jobs on the least loaded nodes (based upon the number
    of idle CPUs). This is generally only recommended for an environment with
    serial jobs as idle resources will tend to be highly fragmented, resulting
    in parallel jobs being distributed across many nodes.
    Note that node \fBWeight\fR takes precedence over how many idle resources are
    on each node.
    Also see the partition configuration parameter \fBLLN\fR
    use the least loaded nodes in selected partitions.
    .TP
    \fBCR_Pack_Nodes\fR
    If a job allocation contains more resources than will be used for launching
    tasks (e.g. if whole nodes are allocated to a job), then rather than
    distributing a job's tasks evenly across its allocated nodes, pack them as
    tightly as possible on these nodes.
    For example, consider a job allocation containing two \fBentire\fR nodes with
    eight CPUs each.
    If the job starts ten tasks across those two nodes without this option, it will
    start five tasks on each of the two nodes.
    With this option, eight tasks will be started on the first node and two tasks
    on the second node.
    This can be superseded by "NoPack" in srun's "--distribution" option.
    CR_Pack_Nodes only applies when the "block" task distribution method is used.
    .TP
    \fBCR_Socket\fR
    Sockets are consumable resources.
    On nodes with multiple cores, each core or thread is counted as a CPU
    to satisfy a job's resource requirement, but multiple jobs are not
    allocated resources on the same socket.
    .TP
    \fBCR_Socket_Memory\fR
    Memory and sockets are consumable resources.
    On nodes with multiple cores, each core or thread is counted as a CPU
    to satisfy a job's resource requirement, but multiple jobs are not
    allocated resources on the same socket.
    Setting a value for \fBDefMemPerCPU\fR is strongly recommended.
    .TP
    \fBCR_Memory\fR
    Memory is a consumable resource.
    NOTE: This implies \fIOverSubscribe=YES\fR or \fIOverSubscribe=FORCE\fR for
    all partitions.
    Setting a value for \fBDefMemPerCPU\fR is strongly recommended.
    .RE
    .RE

    .TP

    .TP
    \fBSlurmctldDebug\fR
    The level of detail to provide \fBslurmctld\fR daemon's logs.
    The default value is \fBinfo\fR.
    If the \fBslurmctld\fR daemon is initiated with \-v or \-\-verbose options,
    that debug level will be preserve or restored upon reconfiguration.

    .RS
    .TP 10
    \fBquiet\fR
    Log nothing
    .TP
    \fBfatal\fR
    Log only fatal errors
    .TP
    \fBerror\fR
    Log only errors
    .TP
    \fBinfo\fR
    Log errors and general informational messages
    .TP
    \fBverbose\fR
    Log errors and verbose informational messages
    .TP
    \fBdebug\fR
    Log errors and verbose informational messages and debugging messages
    .TP
    \fBdebug2\fR
    Log errors and verbose informational messages and more debugging messages
    .TP
    \fBdebug3\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug5\fR
    Log errors and verbose informational messages and even more debugging messages
    .RE

    .TP
    \fBSlurmctldHost\fR
    The short, or long, hostname of the machine where Slurm control daemon is
    executed (i.e. the name returned by the command "hostname \-s").
    This hostname is optionally followed by the address, either the IP address or
    a name by which the address can be identified, enclosed in parentheses (e.g.
    SlurmctldHost=slurmctl-primary(12.34.56.78)). This value must be specified at
    least once. If specified more than once, the first hostname named will be
    where the daemon runs.
    If the first specified host fails, the daemon will execute on the second host.
    If both the first and second specified host fails, the daemon will execute on
    .RS
    .TP
    \fBallow_user_triggers\fR
    Permit setting triggers from non-root/slurm_user users. SlurmUser must also
    be set to root to permit these triggers to work. See the \fBstrigger\fR man
    page for additional details.
    .TP
    \fBcloud_dns\fR
    By default, Slurm expects that the network address for a cloud node won't
    be known until the creation of the node and that Slurm will be notified of the
    node's address (e.g. \fBscontrol update nodename=<name> nodeaddr=<addr>\fR).
    Since Slurm communications rely on the node configuration found in the
    slurm.conf, Slurm will tell the client command, after waiting for all nodes to
    boot, each node's ip address. However, in environments where the nodes are in
    DNS, this step can be avoided by configuring this option.
    .TP
    \fBcloud_reg_addrs\fR
    When a cloud node registers, the node's NodeAddr and NodeHostName will
    automatically be set. They will be reset back to the nodename after powering
    off.
    .TP
    \fBenable_configless\fR
    Permit "configless" operation by the slurmd, slurmstepd, and user commands.
    When enabled the slurmd will be permitted to retrieve config files from the
    slurmctld, and on any 'scontrol reconfigure' command new configs will
    be automatically pushed out and applied to nodes that are running in this
    "configless" mode.
    NOTE: a restart of the slurmctld is required for this to take effect.
    .TP
    \fBidle_on_node_suspend\fR
    Mark nodes as idle, regardless of current state, when suspending nodes with
    \fBSuspendProgram\fR so that nodes will be eligible to be resumed at a later
    time.
    .TP
    \fBpower_save_interval\fR
    How often the power_save thread looks to resume and suspend nodes. The
    power_save thread will do work sooner if there are node state changes. Default
    is 10 seconds.
    .TP
    \fBpower_save_min_interval\fR
    How often the power_save thread, at a minimum, looks to resume and suspend
    nodes. Default is 0.
    .TP
    \fBmax_dbd_msg_action\fR
    Action used once MaxDBDMsgs is reached, options are 'discard' (default) and 'exit'.

    When 'discard' is specified and MaxDBDMsgs is reached we start by purging
    pending messages of types Step start and complete, and it reaches MaxDBDMsgs
    again Job start messages are purged. Job completes and node state changes
    continue to consume the empty space created from the purgings until MaxDBDMsgs
    is reached again at which no new message is tracked creating data loss and
    potentially runaway jobs.
    Run the \fBRebootProgram\fR from the controller instead of on the slurmds. The
    RebootProgram will be passed a comma-separated list of nodes to reboot.
    .TP
    \fBuser_resv_delete\fR
    Allow any user able to run in a reservation to delete it.
    .RE

    .TP
    \fBSlurmctldPidFile\fR
    Fully qualified pathname of a file into which the  \fBslurmctld\fR daemon
    may write its process id. This may be used for automated signal processing.
    The default value is "/var/run/slurmctld.pid".

    .TP
    \fBSlurmctldPlugstack\fR
    A comma delimited list of Slurm controller plugins to be started when the
    daemon begins and terminated when it ends.
    Only the plugin's init and fini functions are called.

    .TP
    \fBSlurmctldPort\fR
    The port number that the Slurm controller, \fBslurmctld\fR, listens
    to for work. The default value is SLURMCTLD_PORT as established at system
    build time. If none is explicitly specified, it will be set to 6817.
    \fBSlurmctldPort\fR may also be configured to support a range of port
    numbers in order to accept larger bursts of incoming messages by specifying
    two numbers separated by a dash (e.g. \fBSlurmctldPort=6817\-6818\fR).
    NOTE: Either \fBslurmctld\fR and \fBslurmd\fR daemons must not
    execute on the same nodes or the values of \fBSlurmctldPort\fR and
    \fBSlurmdPort\fR must be different.

    \fBNote\fR: On Cray systems, Realm-Specific IP Addressing (RSIP) will
    automatically try to interact with anything opened on ports 8192\-60000.
    Configure SlurmctldPort to use a port outside of the configured SrunPortRange
    and RSIP's port range.

    .TP
    \fBSlurmctldPrimaryOffProg\fR
    This program is executed when a slurmctld daemon running as the primary server
    becomes a backup server. By default no program is executed.
    See also the related "SlurmctldPrimaryOnProg" parameter.

    .TP
    \fBSlurmctldPrimaryOnProg\fR
    This program is executed when a slurmctld daemon running as a backup server
    becomes the primary server. By default no program is executed.
    When using virtual IP addresses to manage High Available Slurm services,
    this program can be used to add the IP address to an interface (and optionally
    try to kill the unresponsive  slurmctld daemon and flush the ARP caches on
    nodes on the local Ethernet fabric).
    See also the related "SlurmctldPrimaryOffProg" parameter.
    .TP
    \fBfatal\fR
    Log only fatal errors
    .TP
    \fBerror\fR
    Log only errors
    .TP
    \fBinfo\fR
    Log errors and general informational messages
    .TP
    \fBverbose\fR
    Log errors and verbose informational messages
    .TP
    \fBdebug\fR
    Log errors and verbose informational messages and debugging messages
    .TP
    \fBdebug2\fR
    Log errors and verbose informational messages and more debugging messages
    .TP
    \fBdebug3\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug5\fR
    Log errors and verbose informational messages and even more debugging messages
    .RE

    .TP
    \fBSlurmctldTimeout\fR
    The interval, in seconds, that the backup controller waits for the
    primary controller to respond before assuming control.
    The default value is 120 seconds.
    May not exceed 65533.

    .TP
    \fBSlurmdDebug\fR
    The level of detail to provide \fBslurmd\fR daemon's logs.
    The default value is \fBinfo\fR.
    .RS
    .TP 10
    \fBquiet\fR
    Log nothing
    .TP
    \fBfatal\fR
    Log only fatal errors
    .TP
    \fBerror\fR
    Log only errors
    .TP
    \fBinfo\fR
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug5\fR
    Log errors and verbose informational messages and even more debugging messages
    .RE

    .TP
    \fBSlurmdLogFile\fR
    Fully qualified pathname of a file into which the  \fBslurmd\fR daemon's
    logs are written.
    The default value is none (performs logging via syslog).
    Any "%h" within the name is replaced with the hostname on which the
    \fBslurmd\fR is running.
    Any "%n" within the name is replaced with the Slurm node name on which the
    \fBslurmd\fR is running.
    .br
    See the section \fBLOGGING\fR if a pathname is specified.

    .TP
    \fBSlurmdParameters\fR
    Parameters specific to the Slurmd.
    Multiple options may be comma separated.
    .RS
    .TP
    \fBconfig_overrides\fR
    If set, consider the configuration of each node to be that specified in the
    slurm.conf configuration file and any node with less than the
    configured resources will \fBnot\fR be set DRAIN.
    This option is generally only useful for testing purposes.
    Equivalent to the now deprecated FastSchedule=2 option.
    .TP
    \fBshutdown_on_reboot\fR
    If set, the Slurmd will shut itself down when a reboot request is received.
    .RE

    .TP
    \fBSlurmdPidFile\fR
    Fully qualified pathname of a file into which the  \fBslurmd\fR daemon may write
    its process id. This may be used for automated signal processing.
    Any "%h" within the name is replaced with the hostname on which the
    \fBslurmd\fR is running.
    Any "%n" within the name is replaced with the Slurm node name on which the
    \fBslurmd\fR is running.
    The default value is "/var/run/slurmd.pid".

    .TP
    \fBSlurmdPort\fR
    The port number that the Slurm compute node daemon, \fBslurmd\fR, listens
    to for work. The default value is SLURMD_PORT as established at system
    build time. If none is explicitly specified, its value will be 6818.
    NOTE: Either slurmctld and slurmd daemons must not execute
    is "/var/spool/slurmd".
    Any "%h" within the name is replaced with the hostname on which the
    \fBslurmd\fR is running.
    Any "%n" within the name is replaced with the Slurm node name on which the
    \fBslurmd\fR is running.

    .TP
    \fBSlurmdSyslogDebug\fR
    The slurmd daemon will log events to the syslog file at the specified
    level of detail. If not set, the slurmd daemon will log to syslog at
    level \fBfatal\fR, unless there is no \fBSlurmdLogFile\fR and it is running
    in the background, in which case it will log to syslog at the level specified
    by \fBSlurmdDebug\fR  (at \fBfatal\fR in the case that \fBSlurmdDebug\fR
    is set to \fBquiet\fR) or it is run in the foreground, when it will be set to
    quiet.

    .RS
    .TP 10
    \fBquiet\fR
    Log nothing
    .TP
    \fBfatal\fR
    Log only fatal errors
    .TP
    \fBerror\fR
    Log only errors
    .TP
    \fBinfo\fR
    Log errors and general informational messages
    .TP
    \fBverbose\fR
    Log errors and verbose informational messages
    .TP
    \fBdebug\fR
    Log errors and verbose informational messages and debugging messages
    .TP
    \fBdebug2\fR
    Log errors and verbose informational messages and more debugging messages
    .TP
    \fBdebug3\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug5\fR
    Log errors and verbose informational messages and even more debugging messages
    .RE

    .TP
    \fBSlurmdTimeout\fR
    The interval, in seconds, that the Slurm controller waits for \fBslurmd\fR
    This user must exist on all nodes of the cluster for authentication
    of communications between Slurm components.
    The default value is "root".

    .TP
    \fBSlurmSchedLogFile\fR
    Fully qualified pathname of the scheduling event logging file.
    The syntax of this parameter is the same as for \fBSlurmctldLogFile\fR.
    In order to configure scheduler logging, set both the \fBSlurmSchedLogFile\fR
    and \fBSlurmSchedLogLevel\fR parameters.

    .TP
    \fBSlurmSchedLogLevel\fR
    The initial level of scheduling event logging, similar to the
    \fBSlurmctldDebug\fR parameter used to control the initial level of
    \fBslurmctld\fR logging.
    Valid values for \fBSlurmSchedLogLevel\fR are "0" (scheduler logging
    disabled) and "1" (scheduler logging enabled).
    If this parameter is omitted, the value defaults to "0" (disabled).
    In order to configure scheduler logging, set both the \fBSlurmSchedLogFile\fR
    and \fBSlurmSchedLogLevel\fR parameters.
    The scheduler logging level can be changed dynamically using \fBscontrol\fR.

    .TP
    \fBSlurmUser\fR
    The name of the user that the \fBslurmctld\fR daemon executes as.
    For security purposes, a user other than "root" is recommended.
    This user must exist on all nodes of the cluster for authentication
    of communications between Slurm components.
    The default value is "root".

    .TP
    \fBSrunEpilog\fR
    Fully qualified pathname of an executable to be run by srun following
    the completion of a job step.  The command line arguments for the
    executable will be the command and arguments of the job step.  This
    configuration parameter may be overridden by srun's \fB\-\-epilog\fR
    parameter. Note that while the other "Epilog" executables (e.g.,
    TaskEpilog) are run by slurmd on the compute nodes where the tasks are
    executed, the \fBSrunEpilog\fR runs on the node where the "srun" is
    executing.

    .TP
    \fBSrunPortRange\fR
    The \fBsrun\fR creates a set of listening ports to communicate with the
    controller, the slurmstepd and to handle the application I/O.
    By default these ports are ephemeral meaning the port numbers are selected
    by the kernel. Using this parameter allow sites to configure a range of ports
    from which srun ports will be selected. This is useful if sites want to
    allow only certain port range on their network.

    \fBNote\fR: On Cray systems, Realm-Specific IP Addressing (RSIP) will

    \fBsrun \-N 200\fR will use 13 listening ports.
    .RE

    .TP
    \fBSrunProlog\fR
    Fully qualified pathname of an executable to be run by srun prior to
    the launch of a job step.  The command line arguments for the
    executable will be the command and arguments of the job step.  This
    configuration parameter may be overridden by srun's \fB\-\-prolog\fR
    parameter. Note that while the other "Prolog" executables (e.g.,
    TaskProlog) are run by slurmd on the compute nodes where the tasks are
    executed, the \fBSrunProlog\fR runs on the node where the "srun" is
    executing.

    .TP
    \fBStateSaveLocation\fR
    Fully qualified pathname of a directory into which the Slurm controller,
    \fBslurmctld\fR, saves its state (e.g. "/usr/local/slurm/checkpoint").
    Slurm state will saved here to recover from system failures.
    \fBSlurmUser\fR must be able to create files in this directory.
    If you have a secondary \fBSlurmctldHost\fR configured, this location should be
    readable and writable by both systems.
    Since all running and pending job information is stored here, the use of
    a reliable file system (e.g. RAID) is recommended.
    The default value is "/var/spool".
    If any slurm daemons terminate abnormally, their core files will also be written
    into this directory.

    .TP
    \fBSuspendExcNodes\fR
    Specifies the nodes which are to not be placed in power save mode, even
    if the node remains idle for an extended period of time.
    Use Slurm's hostlist expression to identify nodes with an optional ":" separator
    and count of nodes to exclude from the preceding range.
    For example "nid[10-20]:4" will prevent 4 usable nodes (i.e IDLE and not
    DOWN, DRAINING or already powered down) in the set "nid[10-20]" from being
    powered down.
    Multiple sets of nodes can be specified with or without counts in a comma
    separated list (e.g "nid[10-20]:4,nid[80-90]:2").
    If a node count specification is given, any list of nodes to NOT have a node
    count must be after the last specification with a count.
    For example "nid[10-20]:4,nid[60-70]" will exclude 4 nodes in the set
    "nid[10-20]:4" plus all nodes in the set "nid[60-70]" while
    "nid[1-3],nid[10-20]:4" will exclude 4 nodes from the set "nid[1-3],nid[10-20]".
    By default no nodes are excluded.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendProgram\fR, \fBSuspendRate\fR, \fBSuspendTime\fR,
    \fBSuspendTimeout\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendExcParts\fR
    completely power the node off.
    The program executes as \fBSlurmUser\fR.
    The argument to the program will be the names of nodes to
    be placed into power savings mode (using Slurm's hostlist
    expression format).
    By default, no program is run.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendRate\fR, \fBSuspendTime\fR, \fBSuspendTimeout\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendRate\fR
    The rate at which nodes are placed into power save mode by \fBSuspendProgram\fR.
    The value is number of nodes per minute and it can be used to prevent
    a large drop in power consumption (e.g. after a large job completes).
    A value of zero results in no limits being imposed.
    The default value is 60 nodes per minute.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendProgram\fR, \fBSuspendTime\fR, \fBSuspendTimeout\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendTime\fR
    Nodes which remain idle or down for this number of seconds will be placed into
    power save mode by \fBSuspendProgram\fR.
    For efficient system utilization, it is recommended that the value of
    \fBSuspendTime\fR be at least as large as the sum of \fBSuspendTimeout\fR
    plus \fBResumeTimeout\fR.
    A value of \-1 disables power save mode and is the default.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendProgram\fR, \fBSuspendRate\fR, \fBSuspendTimeout\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendTimeout\fR
    Maximum time permitted (in seconds) between when a node suspend request
    is issued and when the node is shutdown.
    At that time the node must be ready for a resume request to be issued
    as needed for new work.
    The default value is 30 seconds.
    Related configuration options include \fBResumeProgram\fR, \fBResumeRate\fR,
    \fBResumeTimeout\fR, \fBSuspendRate\fR, \fBSuspendTime\fR, \fBSuspendProgram\fR,
    \fBSuspendExcNodes\fR and \fBSuspendExcParts\fR.
    More information is available at the Slurm web site
    ( https://slurm.schedmd.com/power_save.html ).

    .TP
    \fBSwitchType\fR
    Identifies the type of switch or interconnect used for application
    communications.
    Acceptable values include
    "switch/cray_aries" for Cray systems,
    .TP
    \fBTaskPlugin\fR
    Identifies the type of task launch plugin, typically used to provide
    resource management within a node (e.g. pinning tasks to specific
    processors). More than one task plugin can be specified in a comma-separated
    list. The prefix of "task/" is optional. Acceptable values include:
    .RS
    .TP 15
    \fBtask/affinity\fR
    enables resource containment using sched_setaffinity().
    This enables the \-\-cpu\-bind and/or \-\-mem\-bind srun options.
    .TP
    \fBtask/cgroup\fR
    enables resource containment using Linux control cgroups.
    This enables the \-\-cpu\-bind and/or \-\-mem\-bind srun options.
    NOTE: see "man cgroup.conf" for configuration details.
    .TP
    \fBtask/none\fR
    for systems requiring no special handling of user tasks.
    Lacks support for the \-\-cpu\-bind and/or \-\-mem\-bind srun options.
    The default value is "task/none".
    .RE

    .RS
    \fBNOTE:\fR It is recommended to stack \fBtask/affinity,task/cgroup\fR together
    when configuring TaskPlugin, and setting \fBTaskAffinity=no\fR and
    \fBConstrainCores=yes\fR in \fBcgroup.conf\fR. This setup uses the task/affinity
    plugin for setting the affinity of the tasks (which is better and different than
    task/cgroup) and uses the task/cgroup plugin to fence tasks into the specified
    resources, thus combining the best of both pieces.

    \fBNOTE:\fR For CRAY systems only: task/cgroup must be used with, and listed
    \fBafter\fR task/cray_aries in TaskPlugin. The task/affinity plugin can be listed
    anywhere, but the previous constraint must be satisfied. For CRAY systems,
    a configuration like this is recommended:
    .nf
    TaskPlugin=task/affinity,task/cray_aries,task/cgroup
    .fi
    .RE

    .TP
    \fBTaskPluginParam\fR
    Optional parameters for the task plugin.
    Multiple options should be comma separated.
    If \fBNone\fR, \fBBoards\fR, \fBSockets\fR, \fBCores\fR, \fBThreads\fR,
    and/or \fBVerbose\fR are specified, they will override
    the \fB\-\-cpu\-bind\fR option specified by the user
    in the \fBsrun\fR command.
    \fBNone\fR, \fBBoards\fR, \fBSockets\fR, \fBCores\fR and \fBThreads\fR are mutually
    exclusive and since they decrease scheduling flexibility are not generally
    recommended (select no more than one of them).

    \fBSockets\fR
    Bind to sockets by default.
    Overrides automatic binding.
    .TP
    \fBThreads\fR
    Bind to threads by default.
    Overrides automatic binding.
    .TP
    \fBSlurmdOffSpec\fR
    If specialized cores or CPUs are identified for the node (i.e. the
    \fBCoreSpecCount\fR or \fBCpuSpecList\fR are configured for the node),
    then Slurm daemons running on the compute node (i.e. slurmd and slurmstepd)
    should run outside of those resources (i.e. specialized resources are
    completely unavailable to Slurm daemons and jobs spawned by Slurm).
    This option may not be used with the \fBtask/cray_aries\fR plugin.
    .TP
    \fBVerbose\fR
    Verbosely report binding before tasks run.
    Overrides user options.
    .TP
    \fBAutobind\fR
    Set a default binding in the event that "auto binding" doesn't find a match.
    Set to Threads, Cores or Sockets (E.g. TaskPluginParam=autobind=threads).
    .RE

    .TP
    \fBTaskProlog\fR
    Fully qualified pathname of a program to be execute as the slurm job's
    owner prior to initiation of each task.
    Besides the normal environment variables, this has SLURM_TASK_PID
    available to identify the process ID of the task being started.
    Standard output from this program can be used to control the environment
    variables and output for the user program.
    .RS
    .TP 20
    \fBexport NAME=value\fR
    Will set environment variables for the task being spawned.
    Everything after the equal sign to the end of the
    line will be used as the value for the environment variable.
    Exporting of functions is not currently supported.
    .TP
    \fBprint ...\fR
    Will cause that line (without the leading "print ")
    to be printed to the job's standard output.
    .TP
    \fBunset NAME\fR
    Will clear environment variables for the task being spawned.
    .TP
    The order of task prolog/epilog execution is as follows:
    .TP
    \fB1. pre_launch_priv()\fR
    Function in TaskPlugin
    .TP
    \fB5. User epilog\fR
    Job-step-specific task program defined using
    \fBsrun\fR's \fB\-\-task\-epilog\fR option or \fBSLURM_TASK_EPILOG\fR
    environment variable
    .TP
    \fB6. TaskEpilog\fR
    System\-wide per task program defined in slurm.conf
    .TP
    \fB7. post_term()\fR
    Function in TaskPlugin
    .RE

    .TP
    \fBTCPTimeout\fR
    Time permitted for TCP connection to be established. Default value is 2 seconds.

    .TP
    \fBTmpFS\fR
    Fully qualified pathname of the file system available to user jobs for
    temporary storage. This parameter is used in establishing a node's \fBTmpDisk\fR
    space.
    The default value is "/tmp".

    .TP
    \fBTopologyParam\fR
    Comma-separated options identifying network topology options.
    .RS
    .TP 15
    \fBDragonfly\fR
    Optimize allocation for Dragonfly network.
    Valid when TopologyPlugin=topology/tree.
    .TP
    \fBTopoOptional\fR
    Only optimize allocation for network topology if the job includes a switch
    option. Since optimizing resource allocation for topology involves much higher
    system overhead, this option can be used to impose the extra overhead only on
    jobs which can take advantage of it. If most job allocations are not optimized
    for network topology, they may fragment resources to the point that topology
    optimization for other jobs will be difficult to achieve.
    \fBNOTE\fR: Jobs may span across nodes without common parent switches with
    this enabled.
    .RE

    .TP
    \fBTopologyPlugin\fR
    Identifies the plugin to be used for determining the network topology
    and optimizing job allocations to minimize network contention.
    See \fBNETWORK TOPOLOGY\fR below for details.
    Additional plugins may be provided in the future which gather topology
    information directly from the network.
    Acceptable values include:
    Boolean yes or no.  Used to set display and track of the Workload
    Characterization Key.  Must be set to track correct wckey usage.
    NOTE: You must also set TrackWCKey in your slurmdbd.conf file to create
    historical usage reports.

    .TP
    \fBTreeWidth\fR
    \fBSlurmd\fR daemons use a virtual tree network for communications.
    \fBTreeWidth\fR specifies the width of the tree (i.e. the fanout).
    On architectures with a front end node running the slurmd daemon, the value
    must always be equal to or greater than the number of front end nodes which
    eliminates the need for message forwarding between the slurmd daemons.
    On other architectures the default value is 50, meaning each slurmd daemon can
    communicate with up to 50 other slurmd daemons and over 2500 nodes can be
    contacted with two message hops.
    The default value will work well for most clusters.
    Optimal system performance can typically be achieved if \fBTreeWidth\fR
    is set to the square root of the number of nodes in the cluster for
    systems having no more than 2500 nodes or the cube root for larger
    systems. The value may not exceed 65533.

    .TP
    \fBUnkillableStepProgram\fR
    If the processes in a job step are determined to be unkillable for a period
    of time specified by the \fBUnkillableStepTimeout\fR variable, the program
    specified by \fBUnkillableStepProgram\fR will be executed.
    By default no program is run.

    See section \fBUNKILLABLE STEP PROGRAM SCRIPT\fR for more information.

    .TP
    \fBUnkillableStepTimeout\fR
    The length of time, in seconds, that Slurm will wait before deciding that
    processes in a job step are unkillable (after they have been signaled with
    SIGKILL) and execute \fBUnkillableStepProgram\fR.
    The default timeout value is 60 seconds.
    If exceeded, the compute node will be drained to prevent future jobs from being
    scheduled on the node.

    .TP
    \fBUsePAM\fR
    If set to 1, PAM (Pluggable Authentication Modules for Linux) will be enabled.
    PAM is used to establish the upper bounds for resource limits. With PAM support
    enabled, local system administrators can dynamically configure system resource
    limits. Changing the upper bound of a resource limit will not alter the limits
    of running jobs, only jobs started after a change has been made will pick up
    the new limits.
    The default value is 0 (not to enable PAM support).
    Remember that PAM also needs to be configured to support Slurm as a service.
    For sites using PAM's directory based configuration option, a configuration
    file named \fBslurm\fR should be created. The module\-type, control\-flags, and
    module\-path names that should be included in the file are:
    \fBNOTE\fR: UsePAM option has nothing to do with the
    \fBcontribs/pam/pam_slurm\fR and/or \fBcontribs/pam_slurm_adopt\fR modules. So
    these two modules can work independently of the value set for UsePAM.

    .TP
    \fBVSizeFactor\fR
    Memory specifications in job requests apply to real memory size (also known
    as resident set size). It is possible to enforce virtual memory limits for
    both jobs and job steps by limiting their virtual memory to some percentage
    of their real memory allocation. The \fBVSizeFactor\fR parameter specifies
    the job's or job step's virtual memory limit as a percentage of its real
    memory limit. For example, if a job's real memory limit is 500MB and
    VSizeFactor is set to 101 then the job will be killed if its real memory
    exceeds 500MB or its virtual memory exceeds 505MB (101 percent of the
    real memory limit).
    The default value is 0, which disables enforcement of virtual memory limits.
    The value may not exceed 65533 percent.

    \fBNOTE\fR: This parameter is dependent on \fBOverMemoryKill\fR being
    configured in \fBJobAcctGatherParams\fR. It is also possible
    to configure the \fBTaskPlugin\fR to use \fBtask/cgroup\fR for memory
    enforcement. \fBVSizeFactor\fR will not have an effect on memory enforcement
    done through cgroups.

    .TP
    \fBWaitTime\fR
    Specifies how many seconds the srun command should by default wait after
    the first task terminates before terminating all remaining tasks. The
    "\-\-wait" option on the srun command line overrides this value.
    The default value is 0, which disables this feature.
    May not exceed 65533 seconds.

    .TP
    \fBX11Parameters\fR
    For use with Slurm's built-in X11 forwarding implementation.
    .RS
    .TP 8
    \fBhome_xauthority\fR
    If set, xauth data on the compute node will be placed in \fB~/.Xauthority\fR
    rather than in a temporary file under \fBTmpFS\fR.
    .RE

    .SH "NODE CONFIGURATION"
    The configuration of nodes (or machines) to be managed by Slurm is
    also specified in \fB/etc/slurm.conf\fR.
    Changes in node configuration (e.g. adding nodes, changing their
    processor count, etc.) require restarting both the slurmctld daemon
    and the slurmd daemons.
    All slurmd daemons must know each node in the system to forward
    messages in support of hierarchical communications.
    Only the NodeName must be supplied in the configuration file.
    All other node configuration information is optional.
    \fBNodeName\fR is "DEFAULT".
    The default entry values will apply only to lines following it in the
    configuration file and the default values can be reset multiple times
    in the configuration file with multiple entries where "NodeName=DEFAULT".
    Each line where \fBNodeName\fR is "DEFAULT" will replace or add to previous
    default values and not a reinitialize the default values.
    The "NodeName=" specification must be placed on every line
    describing the configuration of nodes.
    A single node name can not appear as a NodeName value in more than one line
    (duplicate node name records will be ignored).
    In fact, it is generally possible and desirable to define the
    configurations of all nodes in only a few lines.
    This convention permits significant optimization in the scheduling
    of larger clusters.
    In order to support the concept of jobs requiring consecutive nodes
    on some architectures,
    node specifications should be place in this file in consecutive order.
    No single node name may be listed more than once in the configuration
    file.
    Use "DownNodes=" to record the state of nodes which are temporarily
    in a DOWN, DRAIN or FAILING state without altering permanent
    configuration information.
    A job step's tasks are allocated to nodes in order the nodes appear
    in the configuration file. There is presently no capability within
    Slurm to arbitrarily order a job step's tasks.
    .LP
    Multiple node names may be comma separated (e.g. "alpha,beta,gamma")
    and/or a simple node range expression may optionally be used to
    specify numeric ranges of nodes to avoid building a configuration
    file with large numbers of entries.
    The node range expression can contain one  pair of square brackets
    with a sequence of comma-separated numbers and/or ranges of numbers
    separated by a "\-" (e.g. "linux[0\-64,128]", or "lx[15,18,32\-33]").
    Note that the numeric ranges can include one or more leading
    zeros to indicate the numeric portion has a fixed number of digits
    (e.g. "linux[0000\-1023]").
    Multiple numeric ranges can be included in the expression
    (e.g. "rack[0\-63]_blade[0\-41]").
    If one or more numeric expressions are included, one of them
    must be at the end of the name (e.g. "unit[0\-31]rack" is invalid),
    but arbitrary names can always be used in a comma-separated list.
    .LP
    The node configuration specified the following information:

    .TP
    \fBNodeName\fR
    Name that Slurm uses to refer to a node.
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    through the host database (/etc/hosts) or DNS, depending on the resolver
    settings.  Note that if the short form of the hostname is not used, it

    .TP
    \fBNodeHostname\fR
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    through the host database (/etc/hosts) or DNS, depending on the resolver
    settings.  Note that if the short form of the hostname is not used, it
    may prevent use of hostlist expressions (the numeric portion in brackets
    must be at the end of the string).
    A node range expression can be used to specify a set of nodes.
    If an expression is used, the number of nodes identified by
    \fBNodeHostname\fR on a line in the configuration file must
    be identical to the number of nodes identified by \fBNodeName\fR.
    By default, the \fBNodeHostname\fR will be identical in value to
    \fBNodeName\fR.

    .TP
    \fBNodeAddr\fR
    Name that a node should be referred to in establishing
    a communications path.
    This name will be used as an
    argument to the getaddrinfo() function for identification.
    If a node range expression is used to designate multiple nodes,
    they must exactly match the entries in the \fBNodeName\fR
    (e.g. "NodeName=lx[0\-7] NodeAddr=elx[0\-7]").
    \fBNodeAddr\fR may also contain IP addresses.
    By default, the \fBNodeAddr\fR will be identical in value to
    \fBNodeHostname\fR.

    .TP
    \fBBcastAddr\fR
    Alternate network path to be used for sbcast network traffic to a given node.
    This name will be used as an argument to the getaddrinfo() function.
    If a node range expression is used to designate multiple nodes,
    they must exactly match the entries in the \fBNodeName\fR
    (e.g. "NodeName=lx[0\-7] BcastAddr=elx[0\-7]").
    \fBBcastAddr\fR may also contain IP addresses.
    By default, the \fBBcastAddr\fR is unset, and sbcast traffic will be routed
    to the \fBNodeAddr\fR for a given node.
    Note: cannot be used with CommunicationParameters=NoInAddrAny.

    .TP
    \fBBoards\fR
    Number of Baseboards in nodes with a baseboard controller.
    Note that when Boards is specified, SocketsPerBoard,
    CoresPerSocket, and ThreadsPerCore should be specified.
    Boards and CPUs are mutually exclusive.
    The default value is 1.

    .TP
    \fBCoreSpecCount\fR
    The CoresPerSocket value describes physical cores, not the
    logical number of processors per socket.
    \fBNOTE\fR: If you have multi\-core processors, you will likely
    need to specify this parameter in order to optimize scheduling.
    The default value is 1.

    .TP
    \fBCpuBind\fR
    If a job step request does not specify an option to control how tasks are bound
    to allocated CPUs (\-\-cpu-bind) and all nodes allocated to the job have the same
    \fBCpuBind\fR option the node \fBCpuBind\fR option will control how tasks are
    bound to allocated resources. Supported values for \fBCpuBind\fR are "none",
    "board", "socket", "ldom" (NUMA), "core" and "thread".

    .TP
    \fBCPUs\fR
    Number of logical processors on the node (e.g. "2").
    CPUs and Boards are mutually exclusive. It can be set to the total
    number of sockets(supported only by select/linear), cores or threads.
    This can be useful when you want to schedule only the cores on a hyper-threaded
    node. If \fBCPUs\fR is omitted, its default will be set equal to the product of
    \fBBoards\fR, \fBSockets\fR, \fBCoresPerSocket\fR, and \fBThreadsPerCore\fR.

    .TP
    \fBCpuSpecList\fR
    A comma delimited list of Slurm abstract CPU IDs reserved for system use.
    The list will be expanded to include all other CPUs, if any, on the same cores.
    These cores will not be available for allocation to user jobs.
    Depending upon the \fBTaskPluginParam\fR option of \fBSlurmdOffSpec\fR,
    Slurm daemons (i.e. slurmd and slurmstepd) may either be confined to these
    resources (the default) or prevented from using these resources.
    Isolation of the Slurm daemons from user jobs may improve application performance.
    If this option and \fBCoreSpecCount\fR are both designated for a node,
    an error is generated.
    This option has no effect unless cgroup job confinement is also configured
    (\fBTaskPlugin=task/cgroup\fR with \fBConstrainCores=yes\fR in cgroup.conf).

    .TP
    \fBFeatures\fR
    A comma delimited list of arbitrary strings indicative of some
    characteristic associated with the node.
    There is no value or count associated with a feature at this time, a node
    either has a feature or it does not.
    A desired feature may contain a numeric component indicating,
    for example, processor speed but this numeric component will be considered to
    be part of the feature string. Features are intended to be used to filter nodes
    eligible to run jobs via the \fB--constraint\fR argument.
    By default a node has no features.
    Also see \fBGres\fR for being able to have more control such as types and
    count. Using features is faster than scheduling against GRES but is limited to
    Boolean operations.

    The final field must specify a generic resources count.
    A suffix of "K", "M", "G", "T" or "P" may be used to multiply the number by
    1024, 1048576, 1073741824, etc. respectively.
    (e.g."Gres=gpu:tesla:1,gpu:kepler:1,bandwidth:lustre:no_consume:4G").
    By default a node has no generic resources and its maximum count is
    that of an unsigned 64bit integer.
    Also see \fBFeatures\fR for Boolean flags to filter nodes using job constraints.

    .TP
    \fBMemSpecLimit\fR
    Amount of memory, in megabytes, reserved for system use and not available
    for user allocations.
    If the task/cgroup plugin is configured and that plugin constrains memory
    allocations (i.e. \fBTaskPlugin=task/cgroup\fR in slurm.conf, plus
    \fBConstrainRAMSpace=yes\fR in cgroup.conf), then Slurm compute node daemons
    (slurmd plus slurmstepd) will be allocated the specified memory limit. Note that
    having the Memory set in \fBSelectTypeParameters\fR as any of the options that
    has it as a consumable resource is needed for this option to work.
    The daemons will not be killed if they exhaust the memory allocation
    (ie. the Out-Of-Memory Killer is disabled for the daemon's memory cgroup).
    If the task/cgroup plugin is not configured, the specified memory will only be
    unavailable for user allocations.

    .TP
    \fBPort\fR
    The port number that the Slurm compute node daemon, \fBslurmd\fR, listens
    to for work on this particular node. By default there is a single port number
    for all \fBslurmd\fR daemons on all compute nodes as defined by the
    \fBSlurmdPort\fR configuration parameter. Use of this option is not generally
    recommended except for development or testing purposes. If multiple
    \fBslurmd\fR daemons execute on a node this can specify a range of ports.

    \fBNote\fR: On Cray systems, Realm-Specific IP Addressing (RSIP) will
    automatically try to interact with anything opened on ports 8192\-60000.
    Configure Port to use a port outside of the configured SrunPortRange and
    RSIP's port range.

    .TP
    \fBProcs\fR
    See \fBCPUs\fR.

    .TP
    \fBRealMemory\fR
    Size of real memory on the node in megabytes (e.g. "2048").
    The default value is 1. Lowering RealMemory with the goal of setting
    aside some amount for the OS and not available for job allocations
    will not work as intended if Memory is not set as a consumable
    resource in \fBSelectTypeParameters\fR. So one of the *_Memory
    options need to be enabled for that goal to be accomplished.
    Also see \fBMemSpecLimit\fR.

    .TP
    Sockets is interpreted as SocketsPerBoard rather than total sockets.
    The default value is 1.

    .TP
    \fBSocketsPerBoard\fR
    Number of physical processor sockets/chips on a baseboard.
    Sockets and SocketsPerBoard are mutually exclusive.
    The default value is 1.

    .TP
    \fBState\fR
    State of the node with respect to the initiation of user jobs.
    Acceptable values are \fICLOUD\fR, \fIDOWN\fR, \fIDRAIN\fR, \fIFAIL\fR,
    \fIFAILING\fR, \fIFUTURE\fR and \fIUNKNOWN\fR.
    Node states of \fIBUSY\fR and \fIIDLE\fR should not be specified in the node
    configuration, but set the node state to \fIUNKNOWN\fR instead.
    Setting the node state to \fIUNKNOWN\fR will result in the node state being
    set to \fIBUSY\fR, \fIIDLE\fR or other appropriate state based upon recovered
    system state information.
    The default value is \fIUNKNOWN\fR.
    Also see the \fBDownNodes\fR parameter below.
    .RS
    .TP 10
    \fBCLOUD\fP
    Indicates the node exists in the cloud.
    Its initial state will be treated as powered down.
    The node will be available for use after its state is recovered from Slurm's
    state save file or the slurmd daemon starts on the compute node.
    .TP
    \fBDOWN\fP
    Indicates the node failed and is unavailable to be allocated work.
    .TP
    \fBDRAIN\fP
    Indicates the node is unavailable to be allocated work.
    .TP
    \fBFAIL\fP
    Indicates the node is expected to fail soon, has
    no jobs allocated to it, and will not be allocated
    to any new jobs.
    .TP
    \fBFAILING\fP
    Indicates the node is expected to fail soon, has
    one or more jobs allocated to it, but will not be allocated
    to any new jobs.
    .TP
    \fBFUTURE\fP
    Indicates the node is defined for future use and need not
    exist when the Slurm daemons are started. These nodes can be made available
    for use simply by updating the node state using the scontrol command rather
    than restarting the slurmctld daemon. After these nodes are made available,
    change their \fRState\fR in the slurm.conf file. Until these nodes are made
    available, they will not be seen using any Slurm commands or nor will
    because NodeAddr and NodeHostName are not defined in the slurm.conf -- and the
    fanout communications need to be disabled by setting TreeWidth to a high number
    (e.g. 65533). If the DNS mapping is made, then the \fBcloud_dns\fR
    \fBSlurmctldParameter\fR can be used.
    .RE

    .TP
    \fBUNKNOWN\fP
    Indicates the node's state is undefined but will be established
    (set to \fIBUSY\fR or \fIIDLE\fR) when the \fBslurmd\fR daemon on that
    node registers. \fIUNKNOWN\fR is the default state.
    .RE

    .TP
    \fBThreadsPerCore\fR
    Number of logical threads in a single physical core (e.g. "2").
    Note that the Slurm can allocate resources to jobs down to the
    resolution of a core. If your system is configured with more than
    one thread per core, execution of a different job on each thread
    is not supported unless you configure \fBSelectTypeParameters=CR_CPU\fR
    plus \fBCPUs\fR; do not configure \fBSockets\fR, \fBCoresPerSocket\fR or
    \fBThreadsPerCore\fR.
    A job can execute a one task per thread from within one job step or
    execute a distinct job step on each of the threads.
    Note also if you are running with more than 1 thread per core and running
    the select/cons_res or select/cons_tres plugin then you will want to set
    the SelectTypeParameters
    variable to something other than CR_CPU to avoid unexpected results.
    The default value is 1.

    .TP
    \fBTmpDisk\fR
    Total size of temporary disk storage in \fBTmpFS\fR in megabytes
    (e.g. "16384"). \fBTmpFS\fR (for "Temporary File System")
    identifies the location which jobs should use for temporary storage.
    Note this does not indicate the amount of free
    space available to the user on the node, only the total file
    system size. The system administration should ensure this file
    system is purged as needed so that user jobs have access to
    most of this space.
    The Prolog and/or Epilog programs (specified in the configuration file)
    might be used to ensure the file system is kept clean.
    The default value is 0.

    .TP
    \fBTRESWeights\fR
    TRESWeights are used to calculate a value that represents how
    busy a node is. Currently only used in federation configurations. TRESWeights
    are different from TRESBillingWeights \-\- which is used for fairshare
    calculations.

    TRES weights are specified as a comma\-separated list of
    The priority of the node for scheduling purposes.
    All things being equal, jobs will be allocated the nodes with
    the lowest weight which satisfies their requirements.
    For example, a heterogeneous collection of nodes might
    be placed into a single partition for greater system
    utilization, responsiveness and capability. It would be
    preferable to allocate smaller memory nodes rather than larger
    memory nodes if either will satisfy a job's requirements.
    The units of weight are arbitrary, but larger weights
    should be assigned to nodes with more processors, memory,
    disk space, higher processor speed, etc.
    Note that if a job allocation request can not be satisfied
    using the nodes with the lowest weight, the set of nodes
    with the next lowest weight is added to the set of nodes
    under consideration for use (repeat as needed for higher
    weight values). If you absolutely want to minimize the number
    of higher weight nodes allocated to a job (at a cost of higher
    scheduling overhead), give each node a distinct \fBWeight\fR
    value and they will be added to the pool of nodes being
    considered for scheduling individually.
    The default value is 1.

    .SH "DOWN NODE CONFIGURATION"
    The \fBDownNodes=\fR parameter permits you to mark certain nodes as in a
    \fIDOWN\fR, \fIDRAIN\fR, \fIFAIL\fR, \fIFAILING\fR or \fIFUTURE\fR state
    without altering the permanent configuration information listed under a
    \fBNodeName=\fR specification.

    .TP
    \fBDownNodes\fR
    Any node name, or list of node names, from the \fBNodeName=\fR specifications.

    .TP
    \fBReason\fR
    Identifies the reason for a node being in state \fIDOWN\fR, \fIDRAIN\fR,
    \fIFAIL\fR, \fIFAILING\fR or \fIFUTURE\fR.
    \Use quotes to enclose a reason having more than one word.

    .TP
    \fBState\fR
    State of the node with respect to the initiation of user jobs.
    Acceptable values are \fIDOWN\fR, \fIDRAIN\fR, \fIFAIL\fR, \fIFAILING\fR
    and \fIFUTURE\fR.
    For more information about these states see the descriptions under \fBState\fR
    in the \fBNodeName=\fR section above.
    The default value is \fIDOWN\fR.

    .SH "FRONTEND NODE CONFIGURATION"
    On computers where frontend nodes are used to execute batch scripts
    rather than compute nodes (Cray ALPS systems), one may configure one or more
    frontend nodes using the configuration parameters defined below. These options
    are very similar to those used in configuring compute nodes. These options may
    \fBAllowUsers\fR
    Comma-separated list of user names which may execute jobs on this front end
    node. By default, all users may use this front end node.
    May not be used with the \fBDenyUsers\fR option.

    .TP
    \fBDenyGroups\fR
    Comma-separated list of group names which are prevented from executing jobs on
    this front end node.
    May not be used with the \fBAllowGroups\fR option.

    .TP
    \fBDenyUsers\fR
    Comma-separated list of user names which are prevented from executing jobs on
    this front end node.
    May not be used with the \fBAllowUsers\fR option.

    .TP
    \fBFrontendName\fR
    Name that Slurm uses to refer to a frontend node.
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    through the host database (/etc/hosts) or DNS, depending on the resolver
    settings.  Note that if the short form of the hostname is not used, it
    may prevent use of hostlist expressions (the numeric portion in brackets
    must be at the end of the string).
    If the \fBFrontendName\fR is "DEFAULT", the values specified
    with that record will apply to subsequent node specifications
    unless explicitly set to other values in that frontend node record or
    replaced with a different set of default values.
    Each line where \fBFrontendName\fR is "DEFAULT" will replace or add to previous
    default values and not a reinitialize the default values.

    .TP
    \fBFrontendAddr\fR
    Name that a frontend node should be referred to in establishing
    a communications path. This name will be used as an
    argument to the getaddrinfo() function for identification.
    As with \fBFrontendName\fR, list the individual node addresses rather than
    using a hostlist expression.
    The number of \fBFrontendAddr\fR records per line must equal the number of
    \fBFrontendName\fR records per line (i.e. you can't map to node names to
    one address).
    \fBFrontendAddr\fR may also contain IP addresses.
    By default, the \fBFrontendAddr\fR will be identical in value to
    \fBFrontendName\fR.

    .TP
    \fBPort\fR
    The port number that the Slurm compute node daemon, \fBslurmd\fR, listens
    to for work on this particular frontend node. By default there is a single port

    .TP
    \fBState\fR
    State of the frontend node with respect to the initiation of user jobs.
    Acceptable values are \fIDOWN\fR, \fIDRAIN\fR, \fIFAIL\fR, \fIFAILING\fR
    and \fIUNKNOWN\fR.
    Node states of \fIBUSY\fR and \fIIDLE\fR should not be specified in the node
    configuration, but set the node state to \fIUNKNOWN\fR instead.
    Setting the node state to \fIUNKNOWN\fR will result in the node state being
    set to \fIBUSY\fR, \fIIDLE\fR or other appropriate state based upon recovered
    system state information.
    For more information about these states see the descriptions under \fBState\fR
    in the \fBNodeName=\fR section above.
    The default value is \fIUNKNOWN\fR.

    .LP
    As an example, you can do something similar to the following to define four
    front end nodes for running slurmd daemons.
    .nf
    FrontendName=frontend[00\-03] FrontendAddr=efrontend[00\-03] State=UNKNOWN
    .fi

    .SH "NODESET CONFIGURATION"
    The nodeset configuration allows you to define a name for a specific set of
    nodes which can be used to simplify the partition configuration section,
    especially for heterogenous or condo-style systems. Each nodeset may be defined
    by an explicit list of nodes, and/or by filtering the nodes by a particular
    configured feature. If both \fBFeature=\fR and \fBNodes=\fR are used the
    nodeset shall be the union of the two subsets.
    Note that the nodesets are only used to simplify the partition definitions
    at present, and are not usable outside of the partition configuration.
    .TP
    \fBFeature\fR
    All nodes with this single feature will be included as part of this nodeset.
    .TP
    \fBNodes\fR
    List of nodes in this set.
    .TP
    \fBNodeSet\fR
    Unique name for a set of nodes. Must not overlap with any NodeName definitions.

    .SH "PARTITION CONFIGURATION"
    The partition configuration permits you to establish different job
    limits or access controls for various groups (or partitions) of nodes.
    Nodes may be in more than one partition, making partitions serve
    as general purpose queues.
    For example one may put the same set of nodes into two different
    partitions, each with different constraints (time limit, job sizes,
    groups allowed to use the partition, etc.).
    Jobs are allocated resources within a single partition.
    Default values can be specified with a record in which
    \fBPartitionName\fR is "DEFAULT".
    represent a different partition.
    The partition configuration file contains the following information:

    .TP
    \fBAllocNodes\fR
    Comma-separated list of nodes from which users can submit jobs in the
    partition.
    Node names may be specified using the node range expression syntax
    described above.
    The default value is "ALL".

    .TP
    \fBAllowAccounts\fR
    Comma-separated list of accounts which may execute jobs in the partition.
    The default value is "ALL".
    \fBNOTE:\fR If AllowAccounts is used then DenyAccounts will not be enforced.
    Also refer to DenyAccounts.

    .TP
    \fBAllowGroups\fR
    Comma-separated list of group names which may view and execute jobs in this
    partition.
    A user will be permitted to view and submit a job to this partition if
    AllowGroups has \fBat least one\fR group associated with the user.
    Jobs executed as user root or as user SlurmUser will be allowed to view and
    use any partition, regardless of the value of AllowGroups. In addition, a Slurm
    Admin or Operator will be able to view any partition, regardless of the value
    of AllowGroups.
    If user root attempts to execute a job as another user (e.g. using
    srun's \-\-uid option), then the job will be subject to AllowGroups as if it
    were submitted by that user.
    By default, AllowGroups is unset, meaning all groups are allowed to use this
    partition. The special value 'ALL' is equivalent to this.
    Even when PrivateData does not hide partition information, AllowGroups will
    still hide partition information accordingly.
    \fBNOTE:\fR For performance reasons, Slurm maintains a list of user IDs
    allowed to use each partition and this is checked at job submission time.
    This list of user IDs is updated when the \fBslurmctld\fR daemon is restarted,
    reconfigured (e.g. "scontrol reconfig") or the partition's \fBAllowGroups\fR
    value is reset, even if is value is unchanged
    (e.g. "scontrol update PartitionName=name AllowGroups=group").
    For a user's access to a partition to change, both his group membership must
    change and Slurm's internal user ID list must change using one of the methods
    described above.

    .TP
    \fBAllowQos\fR
    Comma-separated list of Qos which may execute jobs in the partition.
    Jobs executed as user root can use any partition without regard to
    the value of AllowQos.
    The default value is "ALL".
    \fBNOTE:\fR If AllowQos is used then DenyQos will not be enforced.
    "core" and "thread".

    .TP
    \fBDefault\fR
    If this keyword is set, jobs submitted without a partition
    specification will utilize this partition.
    Possible values are "YES" and "NO".
    The default value is "NO".

    .TP
    \fBDefCpuPerGPU\fR
    Default count of CPUs allocated per allocated GPU. This value is used only if
    the job didn't specify --cpus-per-task and --cpus-per-gpu.

    .TP
    \fBDefMemPerCPU\fR
    Default real memory size available per allocated CPU in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBDefMemPerCPU\fR would generally be used if individual processors
    are allocated to jobs (\fBSelectType=select/cons_res\fR or
    \fBSelectType=select/cons_tres\fR).
    If not set, the \fBDefMemPerCPU\fR value for the entire cluster will be used.
    Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
    exclusive.

    .TP
    \fBDefMemPerGPU\fR
    Default real memory size available per allocated GPU in megabytes.
    Also see \fBDefMemPerCPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
    exclusive.

    .TP
    \fBDefMemPerNode\fR
    Default real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBDefMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    If not set, the \fBDefMemPerNode\fR value for the entire cluster will be used.
    Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
    exclusive.

    .TP
    \fBDenyAccounts\fR
    Comma-separated list of accounts which may not execute jobs in the partition.
    By default, no accounts are denied access
    \fBNOTE:\fR If AllowAccounts is used then DenyAccounts will not be enforced.
    Also refer to AllowAccounts.
    .TP
    \fBDisableRootJobs\fR
    If set to "YES" then user root will be prevented from running any jobs
    on this partition.
    The default value will be the value of \fBDisableRootJobs\fR set
    outside of a partition specification (which is "NO", allowing user
    root to execute jobs).

    .TP
    \fBExclusiveUser\fR
    If set to "YES" then nodes will be exclusively allocated to users.
    Multiple jobs may be run for the same user, but only one user can be active
    at a time.
    This capability is also available on a per-job basis by using the
    \fB\-\-exclusive=user\fR option.

    .TP
    \fBGraceTime\fR
    Specifies, in units of seconds, the preemption grace time
    to be extended to a job which has been selected for preemption.
    The default value is zero, no preemption grace time is allowed on
    this partition.
    Once a job has been selected for preemption, its end time is set to the current
    time plus GraceTime. The job's tasks are immediately sent SIGCONT and SIGTERM
    signals in order to provide notification of its imminent termination.
    This is followed by the SIGCONT, SIGTERM and SIGKILL signal sequence upon
    reaching its new end time. This second set of signals is sent to both the
    tasks \fBand\fR the containing batch script, if applicable.
    See also the global \fBKillWait\fR configuration parameter.

    .TP
    \fBHidden\fR
    Specifies if the partition and its jobs are to be hidden by default.
    Hidden partitions will by default not be reported by the Slurm APIs or commands.
    Possible values are "YES" and "NO".
    The default value is "NO".
    Note that partitions that a user lacks access to by virtue of the
    \fBAllowGroups\fR parameter will also be hidden by default.

    .TP
    \fBLLN\fR
    Schedule resources to jobs on the least loaded nodes (based upon the number
    of idle CPUs). This is generally only recommended for an environment with
    serial jobs as idle resources will tend to be highly fragmented, resulting
    in parallel jobs being distributed across many nodes.
    Note that node \fBWeight\fR takes precedence over how many idle resources are
    on each node.
    Also see the \fBSelectParameters\fR configuration parameter \fBCR_LLN\fR to
    use the least loaded nodes in every partition.

    .TP
    \fBMaxCPUsPerNode\fR
    If not set, the \fBMaxMemPerCPU\fR value for the entire cluster will be used.
    Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
    \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

    .TP
    \fBMaxMemPerNode\fR
    Maximum real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBMaxMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    If not set, the \fBMaxMemPerNode\fR value for the entire cluster will be used.
    Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

    .TP
    \fBMaxNodes\fR
    Maximum count of nodes which may be allocated to any single job.
    The default value is "UNLIMITED", which is represented internally as \-1.
    This limit does not apply to jobs executed by SlurmUser or user root.

    .TP
    \fBMaxTime\fR
    Maximum run time limit for jobs.
    Format is minutes, minutes:seconds, hours:minutes:seconds,
    days\-hours, days\-hours:minutes, days\-hours:minutes:seconds or
    "UNLIMITED".
    Time resolution is one minute and second values are rounded up to
    the next minute.
    This limit does not apply to jobs executed by SlurmUser or user root.

    .TP
    \fBMinNodes\fR
    Minimum count of nodes which may be allocated to any single job.
    The default value is 0.
    This limit does not apply to jobs executed by SlurmUser or user root.

    .TP
    \fBNodes\fR
    Comma-separated list of nodes or nodesets which are associated with this
    partition.
    Node names may be specified using the node range expression syntax
    described above. A blank list of nodes
    (i.e. "Nodes= ") can be used if one wants a partition to exist,
    but have no resources (possibly on a temporary basis).
    A value of "ALL" is mapped to all nodes configured in the cluster.

    .TP
    \fBOverSubscribe\fR
    Controls the ability of the partition to execute more than one job at a
    time on each resource (node, socket or core depending upon the value
    .na
    \fIhttps://slurm.schedmd.com/cons_res.html\fR
    .br
    \fIhttps://slurm.schedmd.com/cons_res_share.html\fR
    .br
    \fIhttps://slurm.schedmd.com/gang_scheduling.html\fR
    .br
    \fIhttps://slurm.schedmd.com/preempt.html\fR
    .ad

    .RS
    .TP 12
    \fBEXCLUSIVE\fR
    Allocates entire nodes to jobs even with \fBSelectType=select/cons_res\fR or
    \fBSelectType=select/cons_tres\fR configured.
    Jobs that run in partitions with \fBOverSubscribe=EXCLUSIVE\fR will have
    exclusive access to all allocated nodes.
    .TP
    \fBFORCE\fR
    Makes all resources in the partition available for oversubscription without
    any means for users to disable it.
    May be followed with a colon and maximum number of jobs in
    running or suspended state.
    For example \fBOverSubscribe=FORCE:4\fR enables each node, socket or
    core to oversubscribe each resource four ways.
    Recommended only for systems using \fBPreemptMode=suspend,gang\fR.

    \fBNOTE\fR: \fBOverSubscribe=FORCE:1\fR is a special case that is not exactly
    equivalent to \fBOverSubscribe=NO\fR. \fBOverSubscribe=FORCE:1\fR disables
    the regular oversubscription of resources in the same partition but it will
    still allow oversubscription due to preemption. Setting \fBOverSubscribe=NO\fR
    will prevent oversubscription from happening due to preemption as well.

    \fBNOTE\fR: If using \fBPreemptType=preempt/qos\fR you can specify a value for
    \fBFORCE\fR that is greater than 1. For example, \fBOverSubscribe=FORCE:2\fR
    will permit two jobs per resource normally, but a third job can be started
    only if done so through preemption based upon QOS.

    \fBNOTE\fR: If \fBOverSubscribe\fR is configured to \fBFORCE\fR or \fBYES\fR
    in your slurm.conf and the system is not configured to use preemption
    (\fBPreemptMode=OFF\fR) accounting can easily grow to values greater than
    the actual utilization. It may be common on such systems to get error messages
    in the slurmdbd log stating: "We have more allocated time than is possible."

    .TP
    \fBYES\fR
    Makes all resources in the partition available for sharing upon request by
    the job.
    Resources will only be over\-subscribed when explicitly requested
    by the user using the "\-\-oversubscribe" option on job submission.
    May be followed with a colon and maximum number of jobs in
    running or suspended state.

    .TP
    \fBPartitionName\fR
    Name by which the partition may be referenced (e.g. "Interactive").
    This name can be specified by users when submitting jobs.
    If the \fBPartitionName\fR is "DEFAULT", the values specified
    with that record will apply to subsequent partition specifications
    unless explicitly set to other values in that partition record or
    replaced with a different set of default values.
    Each line where \fBPartitionName\fR is "DEFAULT" will replace or add to previous
    default values and not a reinitialize the default values.

    .TP
    \fBPreemptMode\fR
    Mechanism used to preempt jobs or enable gang scheduling for this
    partition when \fBPreemptType=preempt/partition_prio\fR is configured.
    This partition\-specific \fBPreemptMode\fR configuration parameter will
    override the cluster\-wide \fBPreemptMode\fR for this partition.
    It can be set to OFF to disable preemption and gang scheduling for this
    partition.
    See also \fBPriorityTier\fR and the above description of the cluster\-wide
    \fBPreemptMode\fR parameter for further details.

    .TP
    \fBPriorityJobFactor\fR
    Partition factor used by priority/multifactor plugin in calculating job priority.
    The value may not exceed 65533.
    Also see PriorityTier.

    .TP
    \fBPriorityTier\fR
    Jobs submitted to a partition with a higher \fBPriorityTier\fR value will be
    evaluated by the scheduler before pending jobs in a partition with a lower
    \fBPriorityTier\fR value. They will also be considered for preemption of running
    jobs in partition(s) with lower \fBPriorityTier\fR values if
    \fIPreemptType=preempt/partition_prio\fR.
    The value may not exceed 65533.
    Also see PriorityJobFactor.

    .TP
    \fBQOS\fR
    Used to extend the limits available to a QOS on a partition.  Jobs will not be
    associated to this QOS outside of being associated to the partition.  They
    will still be associated to their requested QOS.
    By default, no QOS is used.
    \fBNOTE:\fR If a limit is set in both the Partition's QOS and the Job's QOS
    the Partition QOS will be honored unless the Job's QOS has the
    \fBOverPartQOS\fR flag set in which the Job's QOS will have priority.

    .TP
    \fBReqResv\fR
    Specifies users of this partition are required to designate a reservation
    Possible values are "YES" and "NO".
    The default value is "NO".

    .TP
    \fBSelectTypeParameters\fR
    Partition\-specific resource allocation type.
    This option replaces the global \fBSelectTypeParameters\fR value.
    Supported values are \fBCR_Core\fR, \fBCR_Core_Memory\fR, \fBCR_Socket\fR and
    \fBCR_Socket_Memory\fR.
    Use requires the system\-wide \fBSelectTypeParameters\fR value be set to
    any of the four supported values previously listed; otherwise, the
    partition-specific value will be ignored.

    .TP
    \fBShared\fR
    The \fBShared\fR configuration parameter has been replaced by the
    \fBOverSubscribe\fR parameter described above.

    .TP
    \fBState\fR
    State of partition or availability for use.  Possible values
    are "UP", "DOWN", "DRAIN" and "INACTIVE". The default value is "UP".
    See also the related "Alternate" keyword.
    .RS
    .TP 10
    \fBUP\fP
    Designates that new jobs may be queued on the partition, and that
    jobs may be allocated nodes and run from the partition.
    .TP
    \fBDOWN\fP
    Designates that new jobs may be queued on the partition, but
    queued jobs may not be allocated nodes and run from the partition. Jobs
    already running on the partition continue to run. The jobs
    must be explicitly canceled to force their termination.
    .TP
    \fBDRAIN\fP
    Designates that no new jobs may be queued on the partition (job
    submission requests will be denied with an error message), but jobs
    already queued on the partition may be allocated nodes and run.
    See also the "Alternate" partition specification.
    .TP
    \fBINACTIVE\fP
    Designates that no new jobs may be queued on the partition,
    and jobs already queued may not be allocated nodes and run.
    See also the "Alternate" partition specification.
    .RE

    .TP
    \fBTRESBillingWeights\fR
    TRESBillingWeights is used to define the billing weights of each TRES type that
    will be used in calculating the usage of a job. The calculated usage is used
    when calculating fairshare and when enforcing the TRES billing limit on jobs.

    Negative values are allowed.

    When a job is allocated 1 CPU and 8 GB of memory on a partition configured with
    TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0", the billable TRES will be:
    (1*1.0) + (8*0.25) + (0*2.0) = 3.0.

    If PriorityFlags=MAX_TRES is configured, the billable TRES is calculated as the
    MAX of individual TRES' on a node (e.g. cpus, mem, gres) plus the sum of all
    global TRES' (e.g. licenses). Using the same example above the billable TRES
    will be MAX(1*1.0, 8*0.25) + (0*2.0) = 2.0.

    If TRESBillingWeights is not defined then the job is billed against the total
    number of allocated CPUs.

    \fBNOTE:\fR TRESBillingWeights doesn't affect job priority directly as it is
    currently not used for the size of the job. If you want TRES' to play a role in
    the job's priority then refer to the PriorityWeightTRES option.

    .RE

    .SH "PROLOG AND EPILOG SCRIPTS"
    There are a variety of prolog and epilog program options that
    execute with various permissions and at various times.
    The four options most likely to be used are:
    \fBProlog\fR and \fBEpilog\fR (executed once on each compute node
    for each job) plus \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR
    (executed once on the \fBControlMachine\fR for each job).

    \fBNOTE\fR: Standard output and error messages are normally not preserved.
    Explicitly write output and error messages to an appropriate location
    if you wish to preserve that information.

    \fBNOTE\fR:  By default the Prolog script is ONLY run on any individual
    node when it first sees a job step from a new allocation. It does not
    run the Prolog immediately when an allocation is granted.  If no job steps
    from an allocation are run on a node, it will never run the Prolog for that
    allocation.  This Prolog behaviour can be changed by the
    \fBPrologFlags\fR parameter.  The Epilog, on the other hand, always
    runs on every node of an allocation when the allocation is released.

    If the Epilog fails (returns a non\-zero exit code), this will result in the
    node being set to a DRAIN state.
    If the EpilogSlurmctld fails (returns a non\-zero exit code), this will only
    be logged.
    If the Prolog fails (returns a non\-zero exit code), this will result in the
    node being set to a DRAIN state and the job being requeued in a held state
    unless \fBnohold_on_prolog_fail\fR is configured in
    \fBSchedulerParameters\fR.
    If the PrologSlurmctld fails (returns a non\-zero exit code), this will result
    in the job being requeued to be executed on another node if possible. Only
    batch jobs can be requeued.
    If this job is part of a job array, this will be set to the job ID.
    Otherwise it will not be set.
    To reference this specific task of a job array, combine
    SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID
    (e.g. "scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ...");
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_ARRAY_TASK_ID\fR
    If this job is part of a job array, this will be set to the task ID.
    Otherwise it will not be set.
    To reference this specific task of a job array, combine
    SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID
    (e.g. "scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ...");
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_ARRAY_TASK_MAX\fR
    If this job is part of a job array, this will be set to the maximum
    task ID.
    Otherwise it will not be set.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_ARRAY_TASK_MIN\fR
    If this job is part of a job array, this will be set to the minimum
    task ID.
    Otherwise it will not be set.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_ARRAY_TASK_STEP\fR
    If this job is part of a job array, this will be set to the step
    size of task IDs.
    Otherwise it will not be set.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_CLUSTER_NAME\fR
    Name of the cluster executing the job.
    .TP
    \fBSLURM_CONF\fR
    Location of the slurm.conf file. Available in \fBProlog\fR and \fBEpilog\fR.
    .TP
    \fBSLURMD_NODENAME\fR
    Name of the node running the task. In the case of a parallel job executing
    on multiple compute nodes, the various tasks will have this environment
    variable set to different values on each compute node. Available in
    \fBProlog\fR and \fBEpilog\fR.
    .TP
    \fBSLURM_JOB_ACCOUNT\fR
    Account name used for the job.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_JOB_CONSTRAINTS\fR
    Features required to run the job.
    Available in \fBProlog\fR, \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    terminate if it was terminated by a signal.
    Available in \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_JOB_GID\fR
    Group ID of the job's owner.
    .TP
    \fBSLURM_JOB_GPUS\fR
    GPU IDs allocated to the job (if any).
    Available in the \fBProlog\fR.
    .TP
    \fBSLURM_JOB_GROUP\fR
    Group name of the job's owner.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_JOB_ID\fR
    Job ID.
    .TP
    \fBSLURM_JOBID\fR
    Job ID.
    .TP
    \fBSLURM_JOB_NAME\fR
    Name of the job.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_JOB_NODELIST\fR
    Nodes assigned to job. A Slurm hostlist expression.
    "scontrol show hostnames" can be used to convert this to a
    list of individual host names.
    Available in \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_JOB_PARTITION\fR
    Partition that job runs in.
    Available in \fBProlog\fR, \fBPrologSlurmctld\fR and \fBEpilogSlurmctld\fR.
    .TP
    \fBSLURM_JOB_UID\fR
    User ID of the job's owner.
    .TP
    \fBSLURM_JOB_USER\fR
    User name of the job's owner.
    .TP
    \fBSLURM_SCRIPT_CONTEXT\fR
    Identifies which epilog or prolog program is currently running.

    .SH "UNKILLABLE STEP PROGRAM SCRIPT"
    This program can be used to take special actions to clean up the unkillable
    processes and/or notify system administrators.
    The program will be run as \fBSlurmdUser\fR (usually "root") on the compute
    node where \fBUnkillableStepTimeout\fR was triggered.

    Information about the unkillable job step is passed to the script using
    environment variables.
    .TP
    .LP
    Given network topology information, Slurm allocates all of a job's
    resources onto a single leaf of the network (if possible) using a best\-fit
    algorithm.
    Otherwise it will allocate a job's resources onto multiple leaf switches
    so as to minimize the use of higher\-level switches.
    The \fBTopologyPlugin\fR parameter controls which plugin is used to
    collect network topology information.
    The only values presently supported are "topology/3d_torus" (default for
    Cray XT/XE systems, performs best\-fit logic over three\-dimensional topology),
    "topology/none" (default for other systems,
    best\-fit logic over one\-dimensional topology),
    "topology/tree" (determine the network topology based
    upon information contained in a topology.conf file,
    see "man topology.conf" for more information).
    Future plugins may gather topology information directly from the network.
    The topology information is optional.
    If not provided, Slurm will perform a best\-fit algorithm assuming the
    nodes are in a one\-dimensional array as configured and the communications
    cost is related to the node distance in this array.

    .SH "RELOCATING CONTROLLERS"
    If the cluster's computers used for the primary or backup controller
    will be out of service for an extended period of time, it may be
    desirable to relocate them.
    In order to do so, follow this procedure:
    .LP
    1. Stop the Slurm daemons
    .br
    2. Modify the slurm.conf file appropriately
    .br
    3. Distribute the updated slurm.conf file to all nodes
    .br
    4. Restart the Slurm daemons
    .LP
    There should be no loss of any running or pending jobs.
    Ensure that any nodes added to the cluster have the current
    slurm.conf file installed.
    .LP
    \fBCAUTION:\fR If two nodes are simultaneously configured as the
    primary controller (two nodes on which \fBSlurmctldHost\fR specify
    the local host and the \fBslurmctld\fR daemon is executing on each),
    system behavior will be destructive.
    If a compute node has an incorrect \fBSlurmctldHost\fR
    parameter, that node may be rendered
    unusable, but no other harm will result.

    .SH "EXAMPLE"
    .LP
    #
    .br
    # Sample /etc/slurm.conf for dev[0\-25].llnl.gov
    .br
    Epilog=/usr/local/slurm/epilog
    .br
    Prolog=/usr/local/slurm/prolog
    .br
    FirstJobId=65536
    .br
    InactiveLimit=120
    .br
    JobCompType=jobcomp/filetxt
    .br
    JobCompLoc=/var/log/slurm/jobcomp
    .br
    KillWait=30
    .br
    MaxJobCount=10000
    .br
    MinJobAge=3600
    .br
    PluginDir=/usr/local/lib:/usr/local/slurm/lib
    .br
    ReturnToService=0
    .br
    SchedulerType=sched/backfill
    .br
    SlurmctldLogFile=/var/log/slurm/slurmctld.log
    .br
    SlurmdLogFile=/var/log/slurm/slurmd.log
    .br
    SlurmctldPort=7002
    .br
    SlurmdPort=7003
    .br
    SlurmdSpoolDir=/var/spool/slurmd.spool
    .br
    StateSaveLocation=/var/spool/slurm.state
    .br
    SwitchType=switch/none
    .br
    TmpFS=/tmp
    .br
    WaitTime=30
    .br
    JobCredentialPrivateKey=/usr/local/slurm/private.key
    .br
    .na
    JobCredentialPublicCertificate=/usr/local/slurm/public.cert
    .ad
    .br
    #
    .br
    # Node Configurations
    .br
    # Partition Configurations
    .br
    #
    .br
    PartitionName=DEFAULT MaxTime=30 MaxNodes=10 State=UP
    .br
    PartitionName=debug Nodes=dev[0\-8,18\-25] Default=YES
    .br
    PartitionName=batch Nodes=dev[9\-17]  MinNodes=4
    .br
    PartitionName=long Nodes=dev[9\-17] MaxTime=120 AllowGroups=admin

    .SH "INCLUDE MODIFIERS"
    The "include" key word can be used with modifiers within the specified
    pathname. These modifiers would be replaced with cluster name or other
    information depending on which modifier is specified. If the included file
    is not an absolute path name (i.e. it does not start with a slash), it will
    searched for in the same directory as the slurm.conf file.
    .TP
    \fB%c\fR
    Cluster name specified in the slurm.conf will be used.
    .TP
    \fBEXAMPLE\fR
    .RE
    ClusterName=linux
    .br
    include /home/slurm/etc/%c_config
    .br
    # Above line interpreted as
    .br
    # "include /home/slurm/etc/linux_config"

    .SH "FILE AND DIRECTORY PERMISSIONS"
    There are three classes of files:
    Files used by \fBslurmctld\fR must be accessible by user \fBSlurmUser\fR
    and accessible by the primary and backup control machines.
    Files used by \fBslurmd\fR must be accessible by user root and
    accessible from every compute node.
    A few files need to be accessible by normal users on all login and
    compute nodes.
    While many files and directories are listed below, most of them will
    not be used with most configurations.
    .TP
    \fBEpilog\fR
    Must be executable by user root.
    It is recommended that the file be readable by all users.
    The file must exist on every compute node.
    .TP
    \fBEpilogSlurmctld\fR
    Must be executable by user \fBSlurmUser\fR.
    It is recommended that the file be readable by all users.
    .TP
    \fBJobCredentialPublicCertificate\fR
    Readable to all users on all nodes.
    Must not be writable by regular users.
    .TP
    \fBMailProg\fR
    Must be executable by user \fBSlurmUser\fR.
    Must not be writable by regular users.
    The file must be accessible by the primary and backup control machines.
    .TP
    \fBProlog\fR
    Must be executable by user root.
    It is recommended that the file be readable by all users.
    The file must exist on every compute node.
    .TP
    \fBPrologSlurmctld\fR
    Must be executable by user \fBSlurmUser\fR.
    It is recommended that the file be readable by all users.
    The file must be accessible by the primary and backup control machines.
    .TP
    \fBResumeProgram\fR
    Must be executable by user \fBSlurmUser\fR.
    The file must be accessible by the primary and backup control machines.
    .TP
    \fBslurm.conf\fR
    Readable to all users on all nodes.
    Must not be writable by regular users.
    .TP
    \fBSlurmctldLogFile\fR
    Must be writable by user \fBSlurmUser\fR.
    The file must be accessible by the primary and backup control machines.
    .TP
    \fBSlurmctldPidFile\fR
    Must be writable by user root.
    Preferably writable and removable by \fBSlurmUser\fR.
    The file must be accessible by the primary and backup control machines.
    .TP
    \fBSlurmdLogFile\fR
    Must be writable by user root.
    A distinct file must exist on each compute node.
    .TP
    \fBSlurmdPidFile\fR
    Must be writable by user root.
    A distinct file must exist on each compute node.
    .TP
    \fBSlurmdSpoolDir\fR
    Must be writable by user root.
    A distinct file must exist on each compute node.
    .TP
    \fBSrunEpilog\fR
    Must be executable by all users.
    The file must exist on every login and compute node.
    Must be executable by all users.
    The file must exist on every compute node.
    .TP
    \fBTaskProlog\fR
    Must be executable by all users.
    The file must exist on every compute node.
    .TP
    \fBUnkillableStepProgram\fR
    Must be executable by user \fBSlurmUser\fR.
    The file must be accessible by the primary and backup control machines.

    .SH "LOGGING"
    .LP
    Note that while Slurm daemons create log files and other files as needed,
    it treats the lack of parent directories as a fatal error.
    This prevents the daemons from running if critical file systems are
    not mounted and will minimize the risk of cold\-starting (starting
    without preserving jobs).
    .LP
    Log files and job accounting files,
    may need to be created/owned by the "SlurmUser" uid to be successfully
    accessed.  Use the "chown" and "chmod" commands to set the ownership
    and permissions appropriately.
    See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
    about the various files and directories used by Slurm.
    .LP
    It is recommended that the logrotate utility be used to ensure that
    various log files do not become too large.
    This also applies to text files used for accounting,
    process tracking, and the slurmdbd log if they are used.
    .LP
    Here is a sample logrotate configuration. Make appropriate site modifications
    and save as /etc/logrotate.d/slurm on all nodes.
    See the \fBlogrotate\fR man page for more details.
    .LP
    .nf
    ##
    # Slurm Logrotate Configuration
    ##
    /var/log/slurm/*.log {
    	compress
    	missingok
    	nocopytruncate
    	nodelaycompress
    	nomail
    	notifempty
    	noolddir
    	rotate 5
    	sharedscripts
    	size=5M
    	create 640 slurm root
    	postrotate

    .LP
    This file is part of Slurm, a resource management program.
    For details, see <https://slurm.schedmd.com/>.
    .LP
    Slurm is free software; you can redistribute it and/or modify it under
    the terms of the GNU General Public License as published by the Free
    Software Foundation; either version 2 of the License, or (at your option)
    any later version.
    .LP
    Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
    WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
    details.

    .SH "FILES"
    /etc/slurm.conf

    .SH "SEE ALSO"
    .LP
    \fBcgroup.conf\<B>fR(5), \fBgetaddrinfo\fR(3),
    \fBgetrlimit\fR(2), \fBgres.conf\fR(5), \fBgroup\fR(5), \fBhostname\fR(1),
    \fBscontrol\fR(1), \fBslurmctld\fR(8), \fBslurmd\fR(8),
    \fBslurmdbd\fR(8), \fBslurmdbd.conf\fR(5), \fBsrun\fR(1),
    \fBspank\fR(8), \fBsyslog\fR(3), \fBtopology.conf\fR(5)