SUSE Manager 3.2 › Advanced Topics › Optimization and Scalability

5 Optimization and Scalability

5.1 Optimizing Apache and Tomcat

5.1.1 Apache’s httpd MaxClients Parameter
5.1.2 Tomcat’s maxThreads Parameter

5.2 Big Scale Deployment (1000 Minions or More)

5.2.1 General Recommendations
5.2.2 Tuning Proposals

5.1 Optimizing Apache and Tomcat #

Warning: Altering Apache and Tomcat Parameters

Apache and Tomcat Parameters should only be modified with support or consulting as these parameters can have severe and catastrophic performance impacts on your server when improperly adjusted. SUSE will not be able to provide support for catastrophic failure when these advanced parameters are modified without consultation. Tuning values for Apache httpd and Tomcat requires that you align these parameters with your server hardware. Furthermore testing of these altered values should be performed within a test environment.

5.1.1 Apache’s httpd MaxClients Parameter #

The MaxClients setting determines the number of Apache httpd processes, and thus limits the number of client connections that can be made at the same time (SUSE Manager uses the pre-fork MultiProcessing Modules). The default value for MaxClients in SUSE Manager is 150. If you need to set the MaxClients value greater than 150, Apache httpd’s ServerLimit setting and Tomcat’s maxThreads must also be increased accordingly (see below).

Warning

The Apache httpd MaxClients parameter must always be less or equal than Tomcat’s maxThreads parameter!

If the MaxClients value is reached while the software is running, new client connections will be queued and forced to wait, this may result in timeouts. You can check the Apache httpd’s error.log for details:

[error] Server reached MaxClients setting, consider increasing the MaxClients setting

The default MaxClients parameter can be overridden on SUSE Manager by editing the server-tuning.conf file located at /etc/apache2/ .

Example server-tuning.conf file:

# prefork MPM
   <IfModule prefork.c>
           # number of server processes to start
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
           StartServers         5
           # minimum number of server processes which are kept spare
           # http://httpd.apache.org/docs/2.2/mod/prefork.html#minspareservers
           MinSpareServers      5
           # maximum number of server processes which are kept spare
           # http://httpd.apache.org/docs/2.2/mod/prefork.html#maxspareservers
           MaxSpareServers     10
           # highest possible MaxClients setting for the lifetime of the Apache process.
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
           ServerLimit        150
           # maximum number of server processes allowed to start
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
           MaxClients         150
           # maximum number of requests a server process serves
           # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
           MaxRequestsPerChild  10000
   </IfModule>

5.1.2 Tomcat’s maxThreads Parameter #

Tomcat’s maxThreads represents the maximum number of request processing threads that it will create. This value determines the maximum number of simultaneous requests that it is able to handle. All HTTP requests to the SUSE Manager server (from clients, browsers, XMLRPC API scripts, etc.) are handled by Apache httpd, and some of them are routed to Tomcat for further processing. It is thus important that Tomcat is able to serve the same amount of simultaneous requests that Apache httpd is able to serve in the worst case. The default value for SUSE Manager is 200 and should always be equal or greater than Apache httpd’s MaxClients. The maxThreads value is located within the server.xml file located at /etc/tomcat/ .

Example relevant lines in server.xml :

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="200" connectionTimeout="20000"/>
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="200" connectionTimeout="20000"/>

Note: Tuning Notes

When configuring Apache httpd’s MaxClients and Tomcat’s maxThreads parameters you should also take into consideration that each HTTP connection will need one or more database connections. If the RDBMS is not able to serve an adequate amount of connections, issues will arise. See the following equation for a rough calculation of the needed amount of database connections:

((3 * java_max) +  apache_max + 60)

Where:

3 is the number of Java processes the server runs with pooled connections (Tomcat, Taskomatic and Search)
java_max is the maximum number of connections per Java pool (20 by default, changeable in /etc/rhn/rhn.conf via the hibernate.c3p0.max_size parameter)
apache_max is Apache httpd’s MaxClients
60 is the maximum expected number of extra connections for local processes and other uses

5.2 Big Scale Deployment (1000 Minions or More) #

In the following sections find considerations about a big scale deployment. In this context, a big scale compromises 1000 minions or more.

5.2.1 General Recommendations #

SUSE recommends the following in a big scale SUSE Manager deployment:

SUSE Manager servers should have at least 8 recent x86 cores, 32 GiB of RAM, and, most important, fast I/O devices such as at least an SSD (2 SSDs in RAID-0 are strongly recommended).
Proxies with many minions (hundreds) should have at least 2 recent x86 cores and 16 GiB of RAM.
Use one SUSE Manager Proxy per 500-1000 clients. Keep into account that download time depends on network capacity. Here is a rough example calculation with physical link speed of 1 GB/s:
```
400 Megabytes  *        3000       /      119 Megabyte/s        / 60
= 169 Minutes
```
This is:
```
Size of updates * Number of minions / Theoretical download speed / 60
```
Depending on hardware you can accept hundreds of minion keys.
Plan time for onboarding minions— at least one hour per 1000 minions.
It is not recommended onboarding more than approx. 1000 minions directly to the SUSE Manager server— proxies should be used instead. This is because every minion can use up to 3 TCP connections simultaneously, and too many TCP connections can cause performance issues.
If the following error appears in output of dmesg, you probably have an excessive number of minions attached to a single SUSE Manager server or proxy for the ARP cache to contain all of their addresses:
```
kernel: neighbour table overflow
```
In that case, increase the ARP cache values via sysctl , for example, by adding the following lines to /etc/sysctl.conf :
```
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384
net.ipv4.neigh.default.gc_interval = 60
net.ipv4.neigh.default.gc_stale_time = 120
```

Tip: Start Small and Scale Up

Always start small and scale up gradually. Keep the server monitored in order to identify possible issues early.

5.2.2 Tuning Proposals #

SUSE proposes the following tuning settingsin a big scale SUSE Manager deployment:

Increase Tomcat memory to face a long queue of Salt return results. Items in the Salt return queue might accumulate in a short time: Set 8 GiB instead of the current default 1 GiB.
Several RHN parameters should be changed from their defaults:
- The number of Taskomatic workers should be increased, and thus parallelizing work on a high number of separate jobs (onboarding, staging).
- Quartz should check for runnable jobs more frequently to reduce latency (onboarding, staging, Action execution).
- Tomcat’s Salt return result workers should be increased, and thus parallelizing work on a high number of Salt return results (patching).
- The number of PostgreSQL connections available to Java applications (Tomcat, Taskomatic) should increase accordingly. Otherwise extra workers will starve waiting for a connection.
- Salt’s presence ping timeouts should be increased because responses might come back later than the defaults.
Salt master worker threads should be increased to 100 from 15, and thus parallelizing more requests (otherwise Tomcat and Taskomatic workers will starve waiting for the API to do something).
Increasing the number of Salt master worker threads more requires more RAM and does not bring benefits. Apache maximum request time should be increased.

Print this page