Using the Zip Package

You need to use the Jakarta EE Full & Web Distribution package of WildFly when downloading the zip package. If you have not downloaded WildFly already, you can use the following command to do so:

Using Galleon

Instead of downloading the WildFly zip package containing everything (all default server configurations and all JBoss modules), using Galleon tooling you can choose to only install the components needed by EJBCA.

Set Allowed Memory Usage

By default, 512 MB of heap (RAM) is allowed to be used by the application server. This is not sufficient to run EJBCA. We recommend to allocate at least 2048 MB of RAM. To increase the default value, run the following command:

Set the Transaction Node ID

Set the transaction node ID to a unique value. The node ID is used by the transactions subsystem and ensures that the transaction manager only recovers branches that match the specified identifier. This identifier must be unique between WildFly instances sharing either an object store or access common resource managers (i.e. when EJBCA is operating in a cluster).

Create a Master Password

Create a script which outputs the master password to stdout and ensure the script can only be executed by the wildfly user.

Create the Credential Store

Create a credential store in /opt/wildfly/standalone/configuration encrypted with the password echoed by the wildfly_pass script.

MariaDB

PostgreSQL

Microsoft SQL Server

Other Databases

If you are using another type of database, copy the JDBC driver to the deployment directory and make note of the driver class and driver-name shown in the server log, for later use when adding the data source. For example:

MariaDB

For MariaDB, run the following:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

PostgreSQL

For PostgreSQL, run the following:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Microsoft SQL Server

For Microsoft SQL Server, run the following:

Microsoft SQL Server With Kerberos Authentication

Create a configuration file for Kerberos.

Configure WildFly system properties.

Set up Kerberos using Elytron.

Create a datasource with a Kerberos authentication context.

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Other Databases

If you are using another type of database, adapt the statement above with the correct --driver-name, --connection-url, --driver-class and --check-valid-connection-sql.

Logging Configurations

Choose one of the logging configurations below.

Additional Logging Configuration

You may additionally want to add the following configuration:

Add Access Logging

To log all requests processed by the server, similar to the Apache access log, add the following configuration:

Remove the Console Handler

Console logging is not used when running WildFly with systemd, and removing it can increase logging performance.

Remove Old Log Files

Log files are rotated daily by default, but old log files are not deleted automatically. You can delete old log files using a cronjob:

Make the file executable:

Enable Syslog Shipping

To additionally send the server log over UDP to a syslog server such as Graylog or Logstash, configure a syslog handler in WildFly. Set the hostname and port of the syslog server, as well as the desired log level according to the following example:

Enable Audit Logging To File

You can write the EJBCA audit log to a separate file. E.g. to log to /opt/wildfly/standalone/log/cesecore-audit.log, rotate every 500 MB and keep the last two rotated files:

Remove Existing TLS and HTTP Configuration

Run the following commands in JBoss CLI to remove existing TLS and HTTP configuration:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Configure TLS

Configure TLS according to the following instructions.

Add HTTP(S) Listeners

To add HTTP(S) listeners:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080, 8442 and 8443 for incoming TCP traffic.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Configure TLS

Configure TLS according to the following instructions.

Add HTTP(S) Listeners

To add HTTP(S) listeners:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080 and 8443 for incoming TCP traffic.

Alternatively, you may open port 80 and port 443 and redirect traffic to 8080 and 8443 respectively.

Add Support for Sending Email

To add support for sending email using an SMTP smarthost, configure EjbcaMail. Specify the hostname, address, port and SMTP credentials according to the following example:

Remove AJP

Clean up all AJP related configurations that you are not using, or configurations already configured when you attempted to add them.

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Enable AJP Connector

Enabling the AJP connector is only needed if you terminate the TLS connection at a proxy (such as Apache or nginx) running in front of WildFly:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure TLS Certificate Enrollment With ACME

The Elytron subsystem in WildFly can enroll for TLS server certificates (manually configured in keystore.jks above) using ACME. If you enroll from EJBCA, see ACME for more information on how to enable and configure ACME.

To prove ownership of the domain name in the CSR, WildFly must be able to serve an HTTP challenge over port 80 put in the server root directory. The easiest way to get this to work is to configure WildFly with 2-port separation.

1. If the ACME server has a server certificate that chains up to an internal root not trusted by Java you need to manually add it to Java's truststore.

   Add a CA to the Java truststore in CentOS / RHEL

   keytool -importcert -noprompt -trustcacerts -alias internal-root -file InternalRoot.pem -keystore /etc/pki/java/cacerts -storepass changeit

2. Create a keystore where the keypair of the ACME account will be stored.

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=accountsPassword, secret-value="foo123")'

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=accountsKS:add(path=keystore/accounts.p12,relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=accountsPassword},type=PKCS12)'

3. Define the ACME directory to use. If you are enrolling directly against an EJBCA server, use something like https://my.ca/ejbca/acme//directory.

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/certificate-authority=AcmeCA:add(url="<ACME_DIRECTORY_URL>")'

4. Register an ACME account with the CA. The email address may be used by the CA to send announcements and warn about certificates that are about to expire.

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/certificate-authority-account=AcmeAccount:add(certificate-authority=AcmeCA,alias=https,key-store=accountsKS,contact-urls=[mailto:admin@example.com])'

5. Enroll for a certificate to test the configuration. Replace <HOSTNAME> with the hostname of your machine. You can specify multiple hostnames by separating them with a comma.

   Enroll with an RSA keypair

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=https,domain-names=[<HOSTNAME1>],certificate-authority-account=AcmeAccount,agree-to-terms-of-service)'
   Enroll with an EC keypair

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=https,domain-names=[<HOSTNAME1>],algorithm=EC,key-size=256,certificate-authority-account=AcmeAccount,agree-to-terms-of-service)'

6. Create a script to automatically renew the certificate when its about to expire.

   • Install jq.

     Install jq in CentOS / RHEL

     yum install jq -y

   • Create the script called renew-wildfly-certificate.sh and put it in /etc/cron.daily to make it run daily. Adjust the parameters in the script to match your configuration.

     /etc/cron.daily/renew-wildfly-certificate.sh

     #!/bin/sh

     #

     # Renew the WildFly TLS server certificate using ACME.

     # Assumes an ACME account has been configured in Elytron.

     # Requires 'jq' to be installed.

     #

      

     # The number of days before expiry where renewal should be attempted.

     days=30

     # Specify the type of keypair to generate. Possible values: [ "secp256r1", "RSA2048" ]

     key_specification="RSA2048"

     # The domain name(s) to put in the CSR. Separate with comma ','.

     domains="domain1.com,domain2.com"

     # The keystore alias of the certificate to renew

     alias="https"

     # The name of the ACME account in standalone.xml

     account="AcmeAccount"

      

     output=$(/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=elytron/key-store=httpsKS:should-renew-certificate(alias=$alias,expiration=$days)" --output-json)

     logger "$output"

     renew=$(echo "$output" | jq '.result["should-renew-certificate"]')

     if [ "$renew" = "true" ]; then

     logger "Attemping to renew domains [$domains]."

     if [ "$key_specification" = "secp256r1" ]; then

     output=$(/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=$alias,domain-names=[$domains],algorithm=EC,key-size=256,certificate-authority-account=$account,agree-to-terms-of-service)" --output-json)

     logger "$output"

     fi

     if [ "$key_specification" = "RSA2048" ]; then

     output=$(/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=$alias,domain-names=[$domains],certificate-authority-account=$account,agree-to-terms-of-service)" --output-json)

     logger "$output"

     fi

     fi

   • Make the script executable.

     chmod +x /etc/cron.daily/renew-wildfly-certificate.sh

Increase Deployment Timeout

If you are using HSMs with smart card authentication or if the database needs to reindex when WildFly boots, you may have to increase the deployment timeout to be able to deploy EJBCA correctly. The deployment timeout is specified in seconds, the command below sets it to 5 minutes.

Increase Maximum Upload Size

Increase the maximum size of POST requests to 25 MB. This may be needed if you need to upload large domain blocklists, zip packages with profiles, statedumps, etc. Increasing the maximum upload size may also be needed for publishing relatively large CRLs to an external VA.

If 3-port separation is used, increase the maximum upload size for the httpspriv HTTPS listener.

If 2-port separation is used, increase the maximum upload size for the https HTTPS listener instead.

URL Rewriting

URL rewriting allows you to serve CRLs, CA certificates and OCSP responses on your own URLs. This can be done using rewrite rules in Undertow.

Configure OCSP Logging

If you have enabled the OCSP audit log or the OCSP transaction log, you should configure an asynchronous logger in WildFly for best performance.

For the OCSP transaction log:

For the OCSP audit log:

Archive rotated log files to save disk space using a cron job:

Make the file executable:

Remove Welcome Content

Removes the hardcoded welcome page in WildFly. This is not needed if Galleon was used.

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

You can also remove the actual files and save some disk space with:

Remove the ExampleDS Datasource

To optionally remove the ExampleDS datasource:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove Unused Subsystems and Extensions

Optionally remove the unused subsystems and extensions.

If you used the Jakarta EE Full & Web Distribution zip package:

If you used Galleon:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove AJP

Clean up all AJP related configuration if you are not using it.

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Only Deploy at Startup

Historically application servers are really bad at cleaning up memory from previous deployments and hot (re-)deploy is discouraged in production. To avoid manual deployment with the management interface, we can specify that the deployment directory should be scanned once at application server startup by setting the scan-interval to 0. This also prevents an attacker from loading a malicious JAR file by dropping it in the deployments directory.

Disable Management Web Console

If you only plan on using the JBoss CLI anyway there is little reason to keep this around.

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Add a Request Limiter

You can limit the number of concurrent connections by adding a request limiter in Undertow. For example, to allow 100 connections to be processed at the same time, and allow up to 300 connections to be queued before new connections are rejected:

Restrict Access to Services

You can whitelist IP addresses or block access to some services completely using predicates and handlers in Undertow.

For example, to only allow access to the CA web from localhost:

To block access to the public web pages and the EJBCA documentation:

Redirect to Application for Unknown URLs

Known URLs for EJBCA starts with /ejbca, /crls, /certificates or /.well-known (EST and ACME) according to the following example:

Enable HTTP Strict Transport Layer Security

The HTTP Strict-Transport-Security response header (HSTS) defined in RFC 6797 tells the browser to only access the server using HTTPS. If you are not serving any resources on the domain over HTTP such as OCSP responses, you can enable this to improve security. Start with a small max-age value (such as 60 seconds) and then gradually increase.

You should also redirect any HTTP traffic to HTTPS. The example assumes port 80 is used for HTTP, port 443 for HTTPS and that the server can be accessed using the domain name example.com.

Enable OCSP Revocation Checking

WildFly can check the validity of client certificates against the OCSP responder defined by the certificate's AIA extension:

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Enable WildFly Audit Logging

WildFly can audit log changes made to the management model and output audit log events in JSON format to the file /opt/wildfly/standalone/data/audit-log.log. Audit logging is disabled by default. To enable it, run:

Remove WildFly Audit Logging

You may remove the WildFly audit logging configuration completely if you are not using it.

Switch to Debug Logging Globally

To enable debug logging globally, if you need to troubleshoot, follow these steps.

1. Make a note of the current log level for the packages org.cesecore and org.ejbca.
   

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:read-attribute(name=level)'

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:read-attribute(name=level)'

2. Switch to debug logging.

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:write-attribute(name=level, value=DEBUG)'

   /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:write-attribute(name=level, value=DEBUG)'

Once you are done troubleshooting, switch back to the log level you used previously. Note that having debug logging enabled globally will produce a lot of output, avoid using this configuration in production and consider enable debugging for individual packages instead.

Switch to Debug Logging for Individual Packages

EJBCA is divided into different packages, and instead of raising the log level globally, you can be a bit more surgical in your approach and pick individual packages to troubleshoot. For available packages and classes, you can dynamically generate a list of JBoss CLI commands for enabling debug logging based on import statements and package declarations in the source code, with the most "popular" packages first:

If you want something more flashy, you can pipe the output to something like fzf, allowing you to filter and select the package or class name from a list. For example, add the following to your config.fish:

Or if you are using Bash, add this to your .bashrc:

This will add a command called wflogging, which opens fzf where you can select a class or package name in a list. Press Alt+T to take a snapshot of the existing log configuration, Ctrl+A to add logging, Ctrl+R to remove logging, and Ctrl+D to enable debugging for the selected item. The current configuration in the application server is shown in the preview window on the right.

Log Requests and Responses for the Web Service API

To log the SOAP messages received by and sent from EJBCA:

To remove the configuration when you are done:

Log Requests and Responses for the EJBCA REST API

There is a servlet filter bundled with the EJBCA REST API which can be used to dump the full request and response.

Log Requests and Responses for the ACME API

There is a servlet filter bundled with the EJBCA ACME implementation which can be used to dump the full request and response.

Monitor the Peer Systems Connections Pool

Each peer connector maintains a pool of TLS connections. Each request will occupy one connection from the pool, and requests are delayed if the pool is empty. If you suspect that more pooled connections are needed:

Enable Hibernate Statistics

Hibernate is the ORM library used by EJBCA to create SQL queries. You can enable Hibernate statistics to get the SQL query, as well as the time it takes to execute it, written in the server log.

Hibernate statistics are quite verbose and there may be database-specific tools better suited to troubleshoot database performance, however if you want to enable it:

To remove the configuration:

Enable Profiling Using Glowroot

1. Download Glowroot and put it in your WildFly directory.
   

   wget https://github.com/glowroot/glowroot/releases/download/v0.13.6/glowroot-0.13.6-dist.zip -O /tmp/glowroot.zip

   unzip -q /tmp/glowroot.zip -d /opt/wildfly

   chown -R wildfly:wildfly /opt/wildfly/glowroot

2. Register Glowroot as a Java agent and restart WildFly.
   

   sed -i '/-Djdk.tls.ephemeralDHKeySize=2048/ a \ \ \ JAVA_OPTS=\"$JAVA_OPTS -javaagent:/opt/wildfly/glowroot/glowroot.jar"' /opt/wildfly/bin/standalone.conf

   systemctl restart wildfly

3. Glowroot should now be available on http://localhost:4000.