OpenFAST · HaymanConsulting · Jul 25, 2017 · May 1, 2017 · May 1, 2017 · May 1, 2017
diff --git a/.gitignore b/.gitignore
@@ -27,9 +27,12 @@
 *.exe
 *.out
 *.app
+*.pyc
 
 # CMake specific files
 build/**/*
+install/*
+ctest-build/*
 
 vs-build
 build
diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,4 @@
+[submodule "reg_tests/r-test"]
+	path = reg_tests/r-test
+	url = https://github.com/openfast/r-test.git
+	branch = dev
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -105,7 +105,15 @@ endforeach(IDIR IN ITEMS ${FAST_MODULES_EXTERNAL})
 
 option(BUILD_DOCUMENTATION "Build documentation." OFF)
 if(BUILD_DOCUMENTATION)
-   add_subdirectory(docs)
+  add_subdirectory(docs)
+endif()
+
+option(BUILD_TESTING "Build the testing tree." OFF)
+if(BUILD_TESTING)
+  # add_subdirectory(reg_tests) this should be 'include'. with add_subdirectory,
+  # the CTestTestlist.cmake file is placed in the reg_tests subdirectory
+  # and cannot be found by ctest 
+  include(reg_tests/CMakeLists.txt)
 endif()
 
 add_subdirectory(glue-codes)

diff --git a/CTestConfig.cmake b/CTestConfig.cmake
@@ -0,0 +1,19 @@
+## This file should be placed in the root directory of your project.
+## Then modify the CMakeLists.txt file in the root directory of your
+## project to incorporate the testing dashboard.
+##
+## # The following are required to submit to the CDash dashboard:
+##   ENABLE_TESTING()
+##   INCLUDE(CTest)
+
+set(CTEST_PROJECT_NAME "openfast")
+set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
+
+set(CTEST_DROP_METHOD http)
+set(CTEST_DROP_SITE "my.cdash.org")
+set(CTEST_DROP_LOCATION "/submit.php?project=OpenFAST")
+set(CTEST_DROP_SITE_CDASH TRUE)
+
+# saved for debugging
+#set(CTEST_DROP_SITE "localhost")
+#set(CTEST_DROP_LOCATION "/cdash/public/submit.php?project=openfast")
diff --git a/docs/index.rst b/docs/index.rst
@@ -1,24 +1,28 @@
 .. OpenFAST documentation master file, created by
    sphinx-quickstart on Wed Jan 25 13:52:07 2017.
 
-OpenFAST Documentation 
+OpenFAST Documentation
 ======================
 
-This page is under construction. Our intention is that this page will eventually host documents on compiling OpenFAST, theory behind OpenFAST, software-development requirements, reporting issues, etc.
+This page is under construction. Our intention is that this page will eventually host
+documents on compiling OpenFAST, theory behind OpenFAST, software-development
+requirements, reporting issues, etc.
 
-Please contact Michael.A.Sprague@NREL.gov with questions regarding the OpenFAST development plan.
+Please contact `Michael.A.Sprague@NREL.gov <mailto:Michael.A.Sprague@NREL.gov>`_. with questions regarding the OpenFAST
+development plan.
 
-For possible bugs, enhancement requests, or code questions, please submit an issues at the OpenFAST Github repository https://github.com/OpenFAST/OpenFAST
+For possible bugs, enhancement requests, or code questions, please submit an issues at
+the `OpenFAST Github repository <https://github.com/OpenFAST/OpenFAST>`__.
 
-Users may find the established FAST8 through the NWTC Information Portal: https://nwtc.nrel.gov/
+Users may find the established FAST8 through the NWTC Information Portal:
+https://nwtc.nrel.gov/
 
 .. toctree::
    :maxdepth: 2
 
    OpenFAST Github Organization Page <https://github.com/OpenFAST>
    OpenFAST Github Repository <https://github.com/OpenFAST/OpenFAST>
    Installing OpenFAST  <source/install.rst>
-   Module Documentation <source/module.rst>
+   Testing OpenFAST  <source/test.rst>
    Source Code Documentation <source/code.rst>
    Building this Documentation Locally <source/build_doc.rst>
-
diff --git a/docs/source/test.rst b/docs/source/test.rst
@@ -0,0 +1,60 @@
+Testing OpenFAST
+================
+
+The OpenFAST test suite consists of system and module level regression tests and
+unit tests. The regression test compares locally generated solutions to a set of
+baseline solutions. The unit tests ensure that individual subroutines are
+functioning as intended.
+
+All of the necessary files corresponding to the regression test are contained in
+the ``reg_tests`` directory. The unit test framework is housed in ``unit_tests``
+while the actual tests are contained in the directory corresponding to the tested
+module.
+
+Dependencies
+------------
+- Python 2.7/3+
+- Numpy
+- CMake and CTest (Optional in regression test)
+
+Configuring the test suite
+--------------------------
+A portion of the test suite is linked to the OpenFAST repository through
+``git submodule``. Specifically,
+
+- `r-test <https://github.com/openfast/r-test>`__
+- `pFUnit <http://pfunit.sourceforge.net>`__
+
+Be sure to clone the repo with the ``--recursive`` flag or execute
+``git submodule update --init --recursive`` after cloning.
+
+The test suite can be built with `CMake <https://cmake.org/>`__ similar to
+OpenFAST. The default CMake configuration is useful, but may need customization
+for particular build environments. CMake variables can be configured in the `CMake
+GUI <https://cmake.org/download/>`__ or through the command line interface with
+the command ``ccmake``. If the entire OpenFAST package is to be built, the test
+related CMake variables can be configured during the OpenFAST CMake
+configuration. However, if only the test suite will be built, configure CMake
+using the ``reg_tests`` project with ``ccmake path/to/reg_tests`` or selecting
+``reg_tests`` as the source directory in the CMake GUI.
+
+The test specific CMake variables are
+
+- BUILD_TESTING
+- CTEST_OPENFAST_EXECUTABLE
+- CTEST_[MODULE]_EXECUTABLE
+
+Look at the `Installing OpenFAST <install.html>`__ page for more details on
+configuring the CMake targets.
+
+While the unit tests must be built with CMake due to its external dependencies,
+the regression test may be executed without building with CMake. See the
+`regression test <test/regression_test.html>`__ section for more info.
+
+Test specific documentation
+---------------------------
+.. toctree::
+   :maxdepth: 2
+
+   Unit Test <test/unit_test.rst>
+   Regression Test <test/regression_test.rst>
diff --git a/docs/source/test/regression_test.rst b/docs/source/test/regression_test.rst
@@ -0,0 +1,119 @@
+Regression test
+===============
+
+The regression test executes a series of test cases which fully describe
+OpenFAST and its module's capabilities. Each locally computed result is compared
+to a static set of baseline results. To account for machine and compiler
+differences, the regression test attempts to match the current machine and
+compiler type to the appropriate solution set from these combinations
+
+- macOS with GNU compiler (default)
+- Red Hat Enterprise Linux with Intel compiler
+- Windows with Intel compiler
+
+The automated regression test runs
+`CTest <https://cmake.org/Wiki/CMake/Testing_With_CTest>`__ and can be executed
+by running either of the commands ``make test`` or ``ctest`` from the build
+directory. If the entire OpenFAST package is to be built, CMake will configure
+CTest to find the new binary at ``openfast/build/glue-codes/fast/openfast``.
+However, if the intention is to build only the test suite, the OpenFAST binary
+should be specified in the CMake configuration under the ``CTEST_OPENFAST_EXECUTABLE``
+flag. There is also a corresponding ``CTEST_[MODULE]_NAME`` flag for each module
+included in the regression test.
+
+The regression test can be executed manually with the included driver
+``reg_tests/manualRegressionTest.py``.
+
+In both modes of execution a subdirectory is created in the build directory
+called ``reg_tests`` where all of the input files for the test cases are copied
+and all of the locally generated outputs are stored.
+
+Ultimately, both CTest and the manual execution script call a series of Python
+scripts in ``reg_tests`` and ``reg_tests/lib``. One such script is ``lib/pass_fail.py``
+which reads the output files and computes a norm on each channel reported.
+If the maximum norm is greater than a predetermined tolerance, that particular
+test is reported as failed. The failure criteria is outlined in pseudocode below.
+
+::
+
+  for j in range(nChannels)
+     norm_diff[j] = L2norm(localSolution[j]-baselineSolution[j])
+     rms_baseline[j] = L2norm(baselineSolution[j])
+
+  norm = norm_diff / rms_baseline
+
+  if max(norm) < tolerance:
+    success
+
+Running the regression test with CTest
+--------------------------------------
+When driven by CTest, the regression test can be executed by running various
+forms of the command ``ctest`` from the build directory. The basic commands are
+
+- ``ctest`` - Run the entire regression test
+- ``ctest -V`` - Run the entire regression test with verbose output
+- ``ctest -R [TestName]`` - Run a test by name
+- ``ctest -j [N]`` - Run all tests with N tests executing in parallel
+
+Each regression test case contains a series of labels associating all of the
+modules used. The labeling can be seen in the test instantiation in
+``reg_tests/CTestList.cmake`` and called directly with
+
+- ``ctest -L [Label]``
+
+These flags can be compounded making useful variations of ``ctest`` such as
+
+- ``ctest -V -L aerodyn14`` - Runs all cases that use AeroDyn14 with verbose output
+- ``ctest -j 16 -L aerodyn14`` - Runs all cases that use AeroDyn14 in 16 concurrent processes
+- ``ctest -V -R 5MW_DLL_Potential_WTurb`` - Runs the case with name "5MW_DLL_Potential_WTurb"
+
+Regression test from scratch
+----------------------------
+
+- Build OpenFAST and the test suite
+
+::
+
+  git clone --recursive https://github.com/openfast/openfast.git
+  cd openfast/reg_tests/r-tests/openfast
+  python compileDISCON.py
+  cd ../../
+  mkdir build && cd build
+  # Configure CMake - BUILD_TESTING, CTEST_OPENFAST_EXECUTABLE, CTEST_[MODULE]_EXECUTABLE
+  cmake ..
+  make
+  ctest
+
+
+- Build only the test suite
+
+::
+
+  git clone --recursive https://github.com/openfast/openfast.git
+  cd openfast/reg_tests/r-tests/openfast
+  python compileDISCON.py
+  cd ../../
+  mkdir build && cd build
+  # Configure CMake - CTEST_OPENFAST_EXECUTABLE, CTEST_[MODULE]_EXECUTABLE
+  cmake ../reg_tests
+  ctest
+
+- `Windows with Visual Studio regression test <regression_test_windows.html>`__
+
+Follow the link above for a detailed procedure. It is summarized below though
+excluding the procedure to build OpenFAST itself.
+
+::
+
+  git clone --recursive https://github.com/openfast/openfast.git
+  cd openfast
+
+  ## Build the ServoDyn external controller libraries
+  # Open the Visual Studio Solution (DISCON.sln) located in ``openfast\vs-build\DISCON``
+  # Choose Release and x64 for the Solutions Configuration and Solutions Platform
+  # Build Solution
+
+  ## Execute the OpenFAST regression Tests
+  # Open a command prompt which is configured for Python (like Anaconda)
+  cd openfast\reg_tests
+  python manualRegressionTest.py ..\build\bin\FAST_x64.exe Windows Intel
diff --git a/docs/source/test/regression_test_windows.rst b/docs/source/test/regression_test_windows.rst
@@ -0,0 +1,81 @@
+Running OpenFAST Regression Tests on Windows
+============================================
+
+| 1) Clone the openfast repo and initialize the testing database
+|    a) Open a git command shell window [like git bash]
+|    b) Change your working directory to the location above where you want your local repo to be located [the repo will be placed into a folder called openfast at this location]
+|    c) Type:  git clone https://github.com/openfast/openfast.git [this creates a local version of the openfast repo on your computer]
+|         You should see something like this:
+|          Cloning into 'openfast'...
+|          remote: Counting objects: 23801, done.
+|          remote: Compressing objects: 100% (80/80), done.
+|          remote: Total 23801 (delta 73), reused 102 (delta 50), pack-reused 23670
+|          Receiving objects: 100% (23801/23801), 92.10 MiB | 18.99 MiB/s, done.
+|          Resolving deltas: 100% (13328/13328), done.
+|          Checking connectivity... done.
+|    d) Type: cd openfast  [change your working directory to the openfast folder]
+|    e) Type: git checkout dev [this places your local repo on the correct branch of the openfast repo]
+|    f) Type: git submodule update --init --recursive [this downloads the testing database to your computer]
+|         You should see something like this:
+|          Submodule 'reg_tests/r-test' (https://github.com/openfast/r-test.git) registered for path 'reg_tests/r-test'
+|          Cloning into 'reg_tests/r-test'...
+|          remote: Counting objects: 3608, done.
+|          remote: Compressing objects: 100% (121/121), done.
+|          remote: Total 3608 (delta 22), reused 161 (delta 21), pack-reused 3442
+|          Receiving objects: 100% (3608/3608), 154.52 MiB | 26.29 MiB/s, done.
+|          Resolving deltas: 100% (2578/2578), done.
+|          Checking connectivity... done.
+|          Submodule path 'reg_tests/r-test': checked out 'b808f1f3c1331fe5d03c5aaa4167532c2492d378'
+
+
+| 2) Build The Regression Testing DISCON DLLs
+|    a) Open the Visual Studio Solution (DISCON.sln) located in openfast\vs-build\DISCON folder
+|    b) Choose Release and x64 for the Solutions Configuration and Solutions Platform, respectively
+|    c) From the menu bar select Build->Build Solution
+|    d) You should now see the files DISCON.dll, DISCON_ITIBarge.dll, and DISCON_OC4.dll in your openfast\build\reg_tests\openfast\5MW_Baseline\ServoData folder
+
+
+| 3) Build OpenFAST using Visual Studio
+|    a) Open the Visual Studio Solution (FAST.sln) located in openfast\vs-build\FAST folder
+|    b) Choose Release and x64 for the Solutions Configuration and Solutions Platform, respectively
+|    c) From the menu bar select Build->Build Solution
+|       i)  If this is the first time you have tried to build openfast, you will get build errors!!! [continue to steps (ii) and (iii), otherwise if FAST builds successfully, continue to step (3d) ]
+|       ii) Cancel build using the menubar Build->Cancel
+|            [ VS is confused about the build-order/dependency of the project files in FASTlib., but canceling and restarting VS, it somehow as enough info from the partial build to get this right, now]
+|       iii) Close your Visual Studio and then Repeat Steps (a) through (c)
+|    d) You should now see the file FAST_x64.exe in your openfast\build\bin folder
+
+
+| 4) Execute the OpenFAST regression Tests
+|    a) Open a command prompt which is configured for Python [ like Anaconda3 ]
+|    b) Change your working directory to openfast\reg_tests
+|    c) Type: python manualRegressionTest.py ..\build\bin\FAST_x64.exe Windows Intel
+|         You should see this: executing case AWT_YFix_WSt
+|    d) The tests will continue to execute one-by-one until you finally see something like this:
+|         ('AWT_YFix_WSt', 'FAIL')
+|         ('AWT_WSt_StartUp_HighSpShutDown', 'FAIL')
+|         ('AWT_YFree_WSt', 'FAIL')
+|         ('AWT_YFree_WTurb', 'FAIL')
+|         ('AWT_WSt_StartUpShutDown', 'FAIL')
+|         ('AOC_WSt', 'FAIL')
+|         ('AOC_YFree_WTurb', 'FAIL')
+|         ('AOC_YFix_WSt', 'FAIL')
+|         ('UAE_YRamp_WSt', 'FAIL')
+|         ('UAE_Rigid_WRamp_PwrCurve', 'FAIL')
+|         ('WP_VSP_WTurb_PitchFail', 'FAIL')
+|         ('WP_VSP_ECD', 'FAIL')
+|         ('WP_VSP_WTurb', 'FAIL')
+|         ('WP_Stationary_Linear', 'PASS')
+|         ('SWRT_YFree_VS_EDG01', 'FAIL')
+|         ('SWRT_YFree_VS_EDC01', 'FAIL')
+|         ('SWRT_YFree_VS_WTurb', 'FAIL')
+|         ('5MW_DLL_Potential_WTurb', 'FAIL')
+|         ('5MW_DLL_Potential_WTurb_WavesIrr', 'FAIL')
+|         ('5MW_DLL_Potential_WSt_WavesReg', 'FAIL')
+|         ('5MW_DLL_Potential_WTurb_WavesIrrFixedYawGrowth', 'FAIL')
+|         ('5MW_DLL_WTurb_WavesIrr', 'FAIL')
+|         ('5MW_DLL_WTurb_WavesIrr_WavesMulti', 'FAIL')
+|         ('5MW_DLL_WTurb_WavesIrr', 'FAIL')
+|         ('5MW_WSt_WhiteNoise_OC4', 'FAIL')
+|         ('5MW_BD_DLL_Potential_WTurb', 'FAIL')
+|    e) If an individual test succeeds you will see 'PASS' otherwise you will see 'FAIL' after that test's name
diff --git a/docs/source/test/unit_test.rst b/docs/source/test/unit_test.rst
@@ -0,0 +1,4 @@
+Unit test
+=========
+
+Coming soon