Add instructions for adding new regression test cases

docs/source/testing/index.rst

@@ -3,35 +3,31 @@
 Testing OpenFAST
 ================
 
-The OpenFAST test suite consists of glue code and module level regression tests
-and unit tests. The regression tests compare locally generated solutions to
-a set of baseline solutions. The unit tests ensure that individual subroutines
-are functioning as intended.
+OpenFAST is a complex software with many moving parts. In order to
+maintain stability in new and existing code, a test suite is included
+directly in the source code. Two primary levels of tests exist:
+regression tests at the highest level and unit tests at the lowest
+level. The regression tests compare locally generated results with
+stored "baseline" results. These tests give an indication of whether the
+full-system or sub-system response has changed. The unit tests focus on
+a single subroutine or code block. These tests need not be physically
+realistic and focus on the mathematics and exersizing of an algorithm.
+The objective of the included tests is to quickly catch bugs or unexpected
+changes in results. Additionally, the tests can help programmers
+design their module and subroutine interfaces in a sustainable and
+maintainable manner.
 
 All of the necessary files corresponding to the regression tests are contained
 in the ``reg_tests`` directory. The unit test framework is housed in
 ``unit_tests`` while the actual tests are contained in the directory
 corresponding to the tested module.
 
-Configuring the test suite
---------------------------
-Portions of the test suite are linked to the OpenFAST repository through a
-`git submodule`. Specifically,
-
-- `r-test <https://github.com/openfast/r-test>`__
-- `pFUnit <https://github.com/Goddard-Fortran-Ecosystem/pFUnit>`__
-
-.. tip::
-
-    Be sure to clone the repo with the ``--recursive`` flag or execute
-    ``git submodule update --init --recursive`` after cloning.
-
-The test suite can be configured with CMake similar to OpenFAST. The default
-CMake configuration is suitable for most systems, but may need customization
-for particular build environments. See the :ref:`understanding_cmake` section
-for more details on configuring the CMake targets. While the unit tests must
-be built with CMake due to its external dependencies, the regression test
-may be executed without CMake.
+The OpenFAST GitHub repository uses `GitHub Actions <https://github.com/openfast/openfast/actions>`_
+to automatically execute the test suite for new commits and pull
+requests. This cloud computing resource is available to all
+GitHub users and is highly recommended as part of the development
+workflow. After enabling GitHub Actions in an OpenFAST repository, simply
+pushing new commits will trigger the tests.
 
 Test specific documentation
 ---------------------------
@@ -41,50 +37,29 @@ Test specific documentation
    unit_test.rst
    regression_test.rst
 
-Continuous integration
-----------------------
-A TravisCI configuration file is included with the OpenFAST source code at ``openfast/.travis.yml``.
-The continuous integration infrastructure is still under development, but the
-status for all branches and pull requests can be found on the
-`TravisCI OpenFAST page <https://travis-ci.org/OpenFAST>`_.
-
-For development and testing purposes, a version of the TravisCI test can be run
-locally with Docker. The code snippet below outlines starting a TravisCI image
-on Docker.
-
-.. code-block:: bash
+Obtaining and configuring the test suite
+----------------------------------------
+Portions of the test suite are linked to the OpenFAST repository through a
+`git submodule`. Specifically, the following two repositories are included:
 
-    # Running a travis ci image on docker locally
+- `r-test <https://github.com/openfast/r-test>`__
+- `pFUnit <https://github.com/Goddard-Fortran-Ecosystem/pFUnit>`__
 
-    # Run this on your local machine's command line
-    BUILDID="build-1"
-    INSTANCE="travisci/ci-garnet:packer-1512502276-986baf0"
-    docker run --name $BUILDID -dit $INSTANCE /sbin/init
-    docker exec -it $BUILDID bash -l
+.. tip::
 
-    # Now you're inside your docker image
-    sudo apt-get update
-    sudo apt-get install python3-pip
-    sudo -E apt-get -yq --no-install-suggests --no-install-recommends install gfortran libblas-dev liblapack-dev
-    git clone --depth=50 https://github.com/OpenFAST/openfast.git OpenFAST/openfast
-    cd OpenFAST/openfast
+    Be sure to clone the repo with the ``--recursive`` flag or execute
+    ``git submodule update --init --recursive`` after cloning.
 
-    # Modify this line for the commit or pull request to build
-    git fetch origin +refs/pull/203/merge:
+The test suite is configured with CMake similar to the general OpenFAST
+build process with an additional CMake flag:
 
-    git checkout -qf FETCH_HEAD
-    git submodule update --init --recursive
+.. code-block:: bash
 
-    export FC=/usr/bin/gfortran-7
-    export DOUBLE_PRECISION=ON
-    export TRAVIS_BUILD_INTEL=YES
-    export TRAVIS_COMPILER=gcc
-    export CC=gcc
-    gcc --version
-    pyenv shell 3.6.3
+    # BUILD_TESTING     - Build the testing tree (Default: OFF)
+    cmake .. -DBUILD_TESTING:BOOL=ON
 
-    source ~/.bashrc
-    pip3 install numpy
-    mkdir build && cd build
-    cmake .. -DBUILD_TESTING=ON -DDOUBLE_PRECISION=$DOUBLE_PRECISION -DBUILD_SHARED_LIBS=ON
-    make -j 8 install
+Aside from this flag, the default CMake configuration is suitable for most systems.
+See the :ref:`understanding_cmake` section for more details on configuring
+the CMake targets. While the unit tests must be built with CMake due to its external
+dependencies, the regression test may be executed without CMake, as described in
+:ref:`python_driver`.

docs/source/testing/regression_test.rst

@@ -17,14 +17,14 @@ results. To account for system, hardware, and compiler
 differences, the regression test attempts to match the current machine and
 compiler type to the appropriate solution set from these combinations:
 
-================== ========== ============================
- Operating System   Compiler   Hardware
-================== ========== ============================
- **macOS**          **GNU**    **2017 MacbookPro**
- CentOS 7           Intel      NREL Eagle - Intel Skylake
- CentOS 7           GNU        NREL Eagle - Intel Skylake
- Windows 10         Intel      Dell Precision 3530
-================== ========== ============================
+================== ============== ============================
+ Operating System   Compiler       Hardware
+================== ============== ============================
+ macOS 10.15        GNU 10.2       2020 MacbookPro
+ Ubuntu 20.04       Intel oneAPI   Docker
+ Ubuntu 20.04       GNU 10.2       Docker
+ Windows 10         Intel oneAPI   Dell Precision 3530
+================== ============== ============================
 
 The compiler versions, specific math libraries, and more info on hardware used
 to generate the baseline solutions are documented in the
@@ -392,3 +392,53 @@ description is given in :ref:`regression_test_windows`.
    :hidden:
 
    regression_test_windows.rst
+
+.. _new_regression_test_case:
+
+Adding test cases
+-----------------
+In all modes of execution, the regression tests are ultimately driven by a
+series of Python scripts located in the ``openfast/reg_tests`` directory
+with the naming scheme ``execute<Module>RegressionTest.py``.
+The first step to adding a new regression test case is to verify that
+a script exists for the target module. If it does not, an issue
+should be opened in `OpenFAST Issues <https://github.com/openfast/openfast/issues>`_
+to coordinate with the NREL team on creating this script.
+
+The next step is to add the test case in the appropriate location in
+the `r-test` submodule. The directory structure in r-test mirrors the
+directory structure in OpenFAST, so module-level tests should be placed
+in their respective module directories and glue-code tests go in
+``r-test/glue-codes/openfast``. Note the naming scheme of files for
+existing tests and adapt the new test case files accordingly. Specifically,
+the main input file and output file names may be expected in a particular
+convention by the Python scripts. Also, consider that any relative paths
+within the input deck for the new test case must work within the r-test
+directory structure.
+
+Once the test directory exists, the test case must be registered with
+the appropriate drivers. For OpenFAST glue-code tests, this happens both in
+CMake and a standalone list of test cases. For CMake, edit the file
+``openfast/reg_tests/CTestList.cmake``. The additional test should be
+added in the section corresponding to the module or driver at the
+bottom of that file. For the Python driver, the new test case must
+be added to ``openfast/reg_tests/r-test/glue-codes/openfast/CaseList.md``.
+At this point, the registration with CTest can be verified:
+
+.. code-block:: bash
+
+    # Move into the build directory
+    cd openfast/build
+
+    # Run CMake to take the new changes to the test list
+    cmake .. -DBUILD_TESTING=ON  # If the BUILD_TESTING flag was previously enabled, this can be left off
+
+    # List the registered tests, but don't run them
+    ctest -N
+
+For module regression tests, the only option for execution is with the
+CMake driver, so follow the instructions above to edit ``CTestList.cmake``.
+
+Finally, the new test cases in the r-test submodule must be added to the
+r-test repository. To do this, open a new issue in `r-test Issues <https://github.com/openfast/r-test/issues>`_
+requesting for support from the NREL team to commit your test.