Docker Exec and Maven AppAssembler

Docker is a powerful tool for defining, building, deploying and running software in containers for self-contained distribution and isolation purposes. AppAssembler is a Maven plugin that is primarily used for generating start scripts for Java or other JVM applications. Docker and AppAssembler are great tools to include in your build/deployment toolkit. In this post, I am going to discuss some details regarding how we integrate Docker and Java services using Maven’s AppAssembler plugin at Qualtrics.

Launching your Application

There are two ways to launch your application in Docker; either via the CMD or the ENTRYPOINT instruction in your Dockerfile. These instructions tell Docker what to do when your image is launched using docker run. When you ask Docker to shut down your application using docker stop, it first sends SIGTERM to allow the application to exit gracefully. After a configurable amount of time (you can specify the interval with -t or --timeout in seconds), Docker will forcefully shut down your process with SIGKILL. Therefore, it is important that the SIGTERM signal reach your application so it can perform a graceful shutdown. The exact form of your CMD or ENTRYPOINT statement in your Dockerfile determines whether you automatically receive the SIGTERM signal or not.

The “bad” versions above launch your application by wrapping it in a shell; by default a shell process does not pass signals to its children. The “good” versions exec your application, making it the root process and thus able receive signals directly from Docker. The exec command replaces the current running process with the specified process, as opposed to forking which creates a new child process. If you are unsure what your application image is doing, simply shell into a running container and run ps:

Process does not receive signals:

$ docker exec -it efbc20916f74 /bin/sh

/ # ps

PID USER TIME COMMAND

1 root 0:00 /bin/sh -c top

7 root 0:00 top

8 root 0:00 /bin/sh

17 root 0:00 ps

Process receives signals:

$ docker exec -it 577a9dd09e33 /bin/sh

/ # ps

PID USER TIME COMMAND

1 root 0:00 top

6 root 0:00 /bin/sh

13 root 0:00 ps

Notice that in the first example a shell process (/bin/sh) is PID 1 while in the second (top) is PID 1. The extra shell processes (PID 8 and 6 respectively) are the interactive shells from which I ran ps (PID 17 and 13 respectively). Docker sends signals to the root process and shell processes do not propagate signals to their children. Therefore, the top process in the first example does not receive signals while it does in the second example.

In the case of our Java applications, the CMD or ENTRYPOINT will exec the start script generated by AppAssembler. The start script will, in turn, exec Java, resulting in the Java process becoming the root process in the container. You can see this in the AppAssembler Unix daemon script template (full template found on Github):

exec "$JAVACMD" $JAVA_OPTS @EXTRA_JVM_ARGUMENTS@ \
 -classpath "$CLASSPATH" \
 -Dapp.name="@APP_NAME@" \
 -Dapp.pid="$$" \
 -Dapp.repo="$REPO" \
 -Dapp.home="$BASEDIR" \
 -Dbasedir="$BASEDIR" \
 @MAINCLASS@ \
 @APP_ARGUMENTS@"$@"@UNIX_BACKGROUND@

Passing Arguments to the Containers

We pass two types of arguments to our containers:

1. Arguments for the JVM
2. Arguments for the application Main class.

The application arguments are usually very minimal, as most of our configuration is picked up from files or other sources. However, when needed, application arguments can be easily specified. One option is to pass arguments to the start script from your CMD or EXEC via "$@" in the template. For example:

ENTRYPOINT [ "/opt/my-app/bin/my-app", "arg1", "arg2" ]

Alternatively, or in addition to the above, you can specify the arguments in the AppAssembler plugin configuration in your pom.xml file in which case they are inserted into the template via @APP_ARGUMENTS@. For example:

<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>appassembler-maven-plugin</artifactId>

 <configuration>

   <programs>

     <program>
       <mainClass>com.example.Main</mainClass>
       <commandLineArguments>
         <commandLineArgument>arg1</commandLineArgument>
         <commandLineArgument>arg2</commandLineArgument>
       </commandLineArguments>

     </program>

   </programs>

 </configuration>

</plugin>

Passing arguments to the JVM offers similar options. First, you probably noticed that the JAVA_OPTS environment variable is expanded in the template immediately following the Java command. Defining this environment variable in your Dockerfile using the ENV directive is the simplest way to pass arguments to the JVM when using AppAssembler’s launch script:

ENV JAVA_OPTS="-Xms128m -Xmx256m"

> docker run my-container

> docker run my-container -e "JAVA_OPTS=-Xms128m -Xmx512m"

As you can see, you can provide default JVM arguments and then override them from the command line. However, the JAVA_OPTS environment variable must either be set explicitly within the Dockerfile or specified explicitly on the commandline. You cannot build the environment variable at runtime using a combination of build default and command line input. For example, the following is not possible:

ENV JAVA_XMS="128m"

ENV JAVA_XMX="256m"

ENV JAVA_OPTS="-Xms${JAVA_XMS} -Xmx${JAVA_XMX}"

> docker run my-container -e "JAVA_XMX=512m"

The JAVA_OPTS environment variable is defined at build time and is not reinterpreted when the container is run. This is important because the first common workaround is to interpret JAVA_OPTS just before you run your application by modifying your CMD or EXEC directive appropriately. For example:

ENV JAVA_XMS="128m"

ENV JAVA_XMX="256m"

CMD JAVA_OPTS="-Xms${JAVA_XMS} -Xmx${JAVA_XMX}"; echo ${JAVA_OPTS}

> docker run -it my-container -e "JAVA_XMX=512m"

-Xms128m -Xmx512m

This solves the immediate problem, but you have just unwittingly introduced a shell as the root process in your Docker container. We can exec our way around this with something like this:

ENTRYPOINT ["sh", "-c"]

CMD [ \

 "JAVA_OPTS=\"-Xms${JAVA_XMS} -Xmx${JAVA_XMX}\" \

 exec \

 /opt/my-app/bin/my-app" ]

This works great! It provides flexibility to pass both JVM and application arguments while still allowing your application process to receive signals from Docker. Cosmetically, though, it leaves something to be desired, as your Dockerfile is using an inline shell exec and interpolating environment variables. After all, isn’t that why we generated a start script with AppAssembler?

Cosmetic Improvements

One way to improve the appearance is to specify the JVM arguments in the AppAssembler plugin configuration and insert them into the start script via @EXTRA_JVM_ARGUMENTS@; for example:

<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>appassembler-maven-plugin</artifactId>

 <configuration>

   <programs>

     <program>
       <mainClass>com.example.Main</mainClass>
       <commandLineArguments>
         <commandLineArgument>arg1</commandLineArgument>
         <commandLineArgument>arg2</commandLineArgument>
       </commandLineArguments>

     </program>

   </programs>

 </configuration>

</plugin>

Then in your Dockerfile declare the defaults:

ENV JAVA_XMS="128m"

ENV JAVA_XMX="256m"

ENTRYPOINT ["/opt/my-app/bin/my-app"]

The start script rendered from the template now passes the desired JVM arguments while still allowing us to use the exec form of either CMD or EXEC in our Dockerfile. However, while this strategy provides a cleaner look to our Dockerfile, our Dockerfile no longer documents the environment variables our application accepts; these are now split across the Dockerfile and pom.xml (or generated start script).

Our Solution

In order to keep the shell foo out of our Dockerfile and to fully document our application in its Dockerfile, our solution was to customize the AppAssembler template. In particular, our template accepts both JVM and application arguments and performs variable interpolation on each argument. First, here is the custom portion of our template (the complete custom template can be found here):

# Separate arguments for java (JAVA_OPTS) from application (APP_OPTS)

BEFORE_OPTS=""

AFTER_OPTS=""

SPLIT_FOUND=

for ARG in "$@"; do

 VALUE=

 eval VALUE="${ARG}"

 if [ -z "${SPLIT_FOUND}" ]; then

   if [ "${VALUE}" = "--" ]; then

     SPLIT_FOUND="true"

   else

     BEFORE_OPTS="${BEFORE_OPTS} '${VALUE}'"

   fi

 else

   AFTER_OPTS="${AFTER_OPTS} '${VALUE}'"

 fi

done

if [ -z "${SPLIT_FOUND}" ]; then

 JAVA_OPTS=""

 APP_OPTS="${BEFORE_OPTS}"

else

 JAVA_OPTS="${BEFORE_OPTS}"

 APP_OPTS="${AFTER_OPTS}"

fi



# Build a single array of arguments for exec

ARGUMENTS="${JAVA_OPTS} \

 @EXTRA_JVM_ARGUMENTS@ \

 -classpath '${CLASSPATH}' \

 -Dapp.name='@APP_NAME@' \

 -Dapp.pid='$$' \

 -Dapp.repo='${REPO}' \

 -Dapp.home='${BASEDIR}' \

 -Dbasedir='${BASEDIR}' \

 @MAINCLASS@ \

 @APP_ARGUMENTS@ \

 ${APP_OPTS}"

eval "set -- ${ARGUMENTS}"



# Exec the java process replacing this shell

exec "${JAVACMD}" "$@" @UNIX_BACKGROUND@

Next, this is how you configure AppAssembler to use the custom template assuming you have installed the custom template into your target directory. We use the Maven Dependency Plugin to unpack the template from a shared resource package.

<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>appassembler-maven-plugin</artifactId>

 <configuration>

    <unixScriptTemplate>${project.basedir}/target/customTemplate</unixScriptTemplate>

   <programs>

     <program>
       <mainClass>com.example.Main</mainClass>
     </program>

   </programs>

 </configuration>

</plugin>

Finally, this is how you now invoke the application from your Dockerfile:

ENV JAVA_XMS="128m"

ENV JAVA_XMX="256m"



ENTRYPOINT [ \

 "/opt/my-app/bin/my-app", \

 "-Xms${JAVA_XMS}", \

 "-Xmx${JAVA_XMX}", \

 "--" \

 "HardCodedArg1", \

 "HardCodedArg2" ]

To execute your application in this form run:

> docker run -it my-container -e "JAVA_XMX=512m" VariableArg1 VariableArg2

The custom AppAssembler template splits the JVM and application arguments based on a special -- marker argument. Then it passes all the arguments prior to the marker to the JVM and all the arguments following marker to the application. If no marker argument is found then all the arguments are passed to the application. The additional arguments passed to Docker run on the command are appended to the list passed to the application in your Dockerfile.

This custom template combined with the exec in the Dockerfile allows you to define all your environment variables and arguments in the Dockerfile including interpretation of any environment variables as part of both JVM and application arguments. This approach suited our goals the best, but if you take away anything from this post it’s that there are several ways to approach this problem. Hopefully, I’ve shown you something that helps you define a pattern that works best for your needs!