Is there a Java library for testing command line applications?

I doubt that there is such a library, but, depending on what you expect from your students there are things you might be able to build.

First, it is possible to compile and run one Java program from another. Here is a page that describes the how to do it: https://www.journaldev.com/937/compile-run-java-program-another-java-program

Once you can do that, you can, perhaps, read a set of regular expression patterns from a file and match them against what the student program produces using the Pattern class.

There is expect. It is designed to test interactive command-line systems. For non-interactive programs it is easier.

expect is not java, but as pointed out in the comments. The language is irrelevant, as the testing is done from the outside.

This answer is based on the using features and commands available in a Linux environment. The basic idea is to use file redirection to feed input to the program under test (PUT) and collect output, both stdout and stderr. This describes a small system that we have used at Colorado State University that I wrote. If anyone is interested, I can supply both the scripts and some documentation. Using them would require modifications on the part of the tester to accommodate location of submissions, etc that are particular to CSU.

The basic idea is to execute a single test case, collect its output and assess the result. The test framework is language agnostic as it is used to test programs written in a variety of languages. It is written in bash and takes advantage of many Linux utilities. A test case consists of a single line of text containing a testName, number of points for the test, and the actual Linux command necessary to run the test. The "language" of the test case has some simple macro capability to make it easy to write multiple test cases easily. For example, the macro $inputFile is expanded to input/$testName, and likewise for $output. The tester creates multiple input files named the same as the testName in the directory input.The required output redirection is done by the framework, so that the actual command is often as simple as java SomeProgram < $inputFile or ./myprog < $inputFile.

Assessment is done using diff, comparing the output of the PUT to that of a master solution. diff has lots of options to ignore case, white space, etc to loosen up the actual comparison. The master solution's output is collected by simply running the framework and renaming its output as the master. The diff is presented in a colored side-by-side format as that is easier for the students to understand than a standard diff output. A student gets no/full points depending on the diff. A typical program has many test cases. Optional post-processing to assess points is possible as is the ability to add in material generated manually.

In a complete test, the framework takes test cases from a text file, processed them one by one, and collects all output in a single file. At the completion, the file is post-processed to extract the individual test case scores and prepend them to the raw output. The student sees a total score, individual test case scores and finally the results of each test case.

To test the entire class, the framework processes a list of the student ID's from a text file. Each ID corresponds to a directory containing the submission for that student. The submission consists of a single file, though that file is frequently a tar/jar/zip.

The actual build of the PUT is driven by a Makefile. This may be supplied by the person running the tests or many be a required part of the submission. The framework simple performs a make and runs the resulting code. Part of the specification of the assignment is the name/class of the executable. For testing scripts, the Makefile may simply insure that the execute permission is set.