-
Notifications
You must be signed in to change notification settings - Fork 622
Verification Case Setup Example
An example step-by-step procedure for authoring a verification case, from setting up the input files to compiling the FDS Verification Guide.
= Prerequisites =
Before getting started, are you conversant in [http://www.mathworks.com/help/techdoc/ Matlab] and [http://www.latex-project.org/ LaTeX]? No? Do yourself a favor: take a week out of your busy life and learn the basics of these tools. I guarantee your investment will payoff a thousand fold!
= Introduction =
The purpose of this page is to document, through example, all of the minute steps required to get a verification case from conception to full automation, such that the FDS Verification Guide can be built without error by pressing a single button. Similar procedures also apply to the FDS Validation Guide. Here we will follow the development of the "heated channel" test case series developed by Kiyoung Moon of Yonsei University.
One difference between the verification suite and the validation suite is that the user is expected to run all the verification cases in order to compile the guide. That is, from a pristine download of the FDS code project, the output files necessary to generate the plots in the guide are not committed to the repository. For validation this is not the case. All necessary output files are committed and just need to be processed by the appropriate Matlab scripts.
The reader should take note of the files that need to be committed. In the text below, these files will be highlighted with a (commit) tag. The most common problem we encounter when compiling the guides is that an author will assume their case is fully committed because they are able to compile the guide on their machine, but if they have not committed all the necessary files the guide will not compile for the other project members.
Let's get started.
= Case Setup =
For this particular test series Kiyoung wants to compare FDS results for the mean streamwise temperature profile in the wall-normal direction to the heated channel flow DNS calculations of Kim and Moin (1987). He will run four cases at different molecular Prandtl numbers (Pr). His goal is to verify the implementation of a new wall heat flux model.
== Naming Convention ==
The first thing to think about is what the case names will be. This is more important than it may seem at first. The reason is that the names work their way into the depths of the processing scripts through attachments to output data files, etc., and making any changes later can prove to be a nightmare. So, do all of us a favor and think carefully about the naming convention. The names ought to be descriptive, but somewhat compact. Here are a few guidelines:
Kiyoung's four cases are named:
- [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Verification/Turbulence/heated_channel_Pr_0p10_32.fds heated_channel_Pr_0p10_32.fds] (commit)
- [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Verification/Turbulence/heated_channel_Pr_0p71_32.fds heated_channel_Pr_0p71_32.fds] (commit)
- [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Verification/Turbulence/heated_channel_Pr_1p00_32.fds heated_channel_Pr_1p00_32.fds] (commit)
- [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Verification/Turbulence/heated_channel_Pr_2p00_32.fds heated_channel_Pr_2p00_32.fds] (commit)
The "32" relates to the number of grid points: these cases are 32^3^. Later, it will be easy to add 64^3^ cases with the same convention.
== Choosing a Subdirectory ==
Next, think about which verification subdirectory should be used to store the input files. In this example, we could choose [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FVerification%2FHeat_Transfer Verification/Heat_Transfer] or [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FVerification%2FTurbulence Verification/Turbulence]. We choose the latter, since the purpose of the test cases is to verify a new heat flux model for turbulent flow.
Note that ALL verification cases must live within a directory one level below Verification. DO NOT create subdirectories under Turbulence, for example, to help organize the cases. If new directories are needed, put them under Verification.
Copy all the input files in the test series to the verification subdirectory.
== Input File Guidelines ==
The first issue to discuss about input files is the text editor used to create them. Be careful not to commit DOS files with "^M" carriage returns. This happens commonly with Windows-based text editors. The safe bet is to look at your files in vi before committing.
Minimize input parameters. Do not add a parameter to the input file if it is already the FDS default.
Minimize comments. Some users write dissertations inside their input files. Save this for the guide write up.
Minimize the size of the output files. In most verification cases we do not need many slice files or time history devices. When possible, use line devices (POINTS=...) to gather statistics. This both reduces the clutter in the input file and reduces the size of the output. Note that the device files for verification cases are not committed---this issue is more relevant to validation cases---but it is a good habit to get into.
== Edit FDS_Cases.sh ==
For the cases to run automatically in the verification suite, they must be added to the master list in [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Verification/FDS_Cases.sh Verification/FDS_Cases.sh] (commit). This list is organized alphabetically by subdirectory name. Append your cases to the end of the appropriate subdirectory. In this case, Kiyoung added the following:
{{{ $QFDS -d Turbulence heated_channel_Pr_0p10_32 $QFDS -d Turbulence heated_channel_Pr_0p71_32 $QFDS -d Turbulence heated_channel_Pr_1p00_32 $QFDS -d Turbulence heated_channel_Pr_2p00_32 }}}
== Commit the Files ==
Once you are happy with the input files and they run with the debug version of the latest code, you are ready to commit them to the repository. Do not forget to update your repository just prior to committing---someone else might be editing the master list. Commit all input files and the FDS_Cases.sh file.
= Post-processing =
In this section we will discuss the process of comparing the FDS results with an expected or (even better) analytical solution. There are a few ways this can be done. The easiest path is if the expected results and the FDS results can be plotted with the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/scripts/dataplot.m dataplot.m] script provided in the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%253Fstate%253Dclosed Utilities/Matlab] directory. This is usually the case if we are just looking at profiles. However, if a significant amount of post-processing calculations must be done before direct comparisons can be made, it may be necessary to create your own Matlab script to generate the plots for the FDS Verification Guide. Note that at present Matlab is the standard. This is the only way we have to make everything look identical throughout the guides.
== Organize the Data File ==
If you have an analytical solution or data for comparison, it is best to create a simple, comma-delimited file (.csv) with the data organized into columns with simple, clear header names. Basically, the same naming conventions follow within the data file as we listed above for input filenames: no decimals, spaces, etc. Once the data have been organized, you may commit the file to the same verification subdirectory where the input files live. In this case, we are committing:
[http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FVerification%2FTurbulence FDS-SMV/Verification/Turbulence/heated_channel_dns_data.csv] (commit)
== Aesthetics of the Plots ==
Because there are literally thousands of plots in the FDS guides, it is important that there exists some level of continuity for the reader and we are therefore very picky about how the plots look. All the things you have heard over the years about high quality graphics and good presentations come into play here. If you are rusty, have a look at [http://www.edwardtufte.com/tufte/ Edward Tufte's web site].
== Using dataplot.m ==
As we mentioned before, the easiest thing to do when adding verification cases is to organize your output so that the dataplot script can be used to generate the plots. In this way, all the fonts, line widths, etc., are handled automatically. To add a plot via dataplot, you must add a line to the spreadsheet [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%253Fstate%253Dclosed FDS_verification_dataplot_inputs.csv] (commit). All the examples you could need are there in the spreadsheet. Find a logical place to insert a row for your specific plot. The key is to note the column where the plot will be saved. It will be saved in the Manuals directory, NOT the Verification directory. In our case, the plot should be printed to [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide%2FSCRIPT_FIGURES Manuals/FDS_Verification_Guide/SCRIPT_FIGURES]/plotname.pdf. Note: DO NOT COMMIT THIS FILE. It is generated automatically.
Plots that may change when the code is modified are stored in [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide%2FSCRIPT_FIGURES SCRIPT_FIGURES]. Other images that are needed in the guides are permanently archived in the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide%2FFIGURES FIGURES] directory.
== Error statistics in FDS_verification_dataplot_inputs.csv ==
The following columns in FDS_verification_dataplot_inputs.csv are used to generate error information for regression statistics. The inputs that can be used are listed alongside each column name. When the verification script runs, these statistics are output to FDS_verification_scatterplot_outputs.csv and placed in an appendix in the FDS Verification Guide.
Quantity
|| Relative error || Outputs error as abs((x,,i,,-y,,i,,)/y,,i,,,) where x,,i,, is the predicted metric and y,,i,, is the expected metric. || || Absolute error || Outputs error as abs(x,,i,,-y,,i,,). ||
Metric
|| all || Compares the values of all points along each curve. || || area || Compares the value of the area under each curve. || || end || Compares the value of the last point on each curve. || || end_x_y || Compares the value of the last point on each specified curve, where x is the number of the expected curve that you want to compare to curve y of the predicted metric. The values x and y correspond to the order that the curves are listed in the d1{{{}}}!Dep_Col_Name and d2{{{}}}!Dep_Col_Name columns. For example, to compare the end points of curves 1 and 3, you would specify end_1_3. || || max || Compares the maximum (peak) value of each curve. || || mean || Compares the mean value of each curve. || || mean_x_y || Compares the mean value on each specified curve, where x is the number of the expected curve that you want to compare to curve y of the predicted metric. For example, to compare the mean of curves 2 and 2, you would specify mean_2_2. || || min || Compares the minimum value of each curve. || || threshold || Compares the independent value when the dependent value reaches some threshold. For example, to compare the time to threshold temperature for cables, the d1{{{_}}}!Comp_Start value is set to the threshold temperature, and then the time at which this temperature is attained is reported. ||
!Error_Tolerance - Specify the error tolerance that should not be exceeded. For example, for a 1% relative error, this would be 0.01; for an absolute error of 20 degrees C, this would be 20.
!Group_Key_Label - Used to group common verification cases. This should be set to the directory that the verification case resides in (e.g., Pyrolysis).
!Group_Style - Used for scatterplots of validation cases, can be safely ignored for verification cases.
!Fill_Color - Used for scatterplots of validation cases, can be safely ignored for verification cases.
== Adding Matlab Scripts ==
In this case, since we need to post-process the FDS data (non-dimensionalize) to compare with the DNS results, it makes sense to write a special script. If you are Matlab savvy, this is not too hard. If not, it is a great way to learn Matlab! The most difficult part of all this for the uninitiated seems to be keeping the repository directory structure straight in their heads while creating the plotting routines. This is critical, because ultimately the script will live here:
[http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%2Fscripts FDS-SMV/Utilities/Matlab/scripts/]
But... it will be launched via the master script which lives here:
[http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%253Fstate%253Dclosed FDS-SMV/Utilities/Matlab/]
These relative directories can be confusing at first. But follow this example, and you will get things right from here on.
The first thing to do is read in the analytical (expected) data and the FDS data, perform whatever calculations are necessary, and plot the results. In the next section we will worry about the details of how the plot looks.
As usual, think of a good name for your script and add it to the master list of special scripts under "Special cases" in [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/FDS_verification_script.m FDS_verification_script.m] (commit). While you are developing the script, it is usually helpful to comment out all the other calls above your script. This allows you to run your script in "production mode" from exactly the directory where it will live permanently. Mark my words: If you decide to do all your development offline in some other directory, you will have all kinds of trouble merging everything in once you think you are finished. Please, do not think that because you can generate an Excel plot that you are anywhere close to finished.
We actually have a couple of options for where we could add the script. If the type of problem you are working on is completely new, then adding the directly to the master list is probably best. For example, we could add our new script [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/scripts/heated_channel.m heated_channel.m] (commit) to the master list as follows:
{{{ % Special cases
... (other scripts, comment out during development) run scripts/heated_channel }}}
Note how the scripts must be run when launched from within the master script since the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/scripts/heated_channel.m heated_channel.m] lives a level below.
Our problem happens to fit nicely with the other scripts that are run in the special script called [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/scripts/wall_model.m wall_model.m]. Therefore, we choose to add our script there:
{{{ % McDermott % 5-20-2009 % wall_model.m % % This script generates all plots necessary for the wall model section % of the verification guide.
law_of_the_wall % Figure: The law of the wall poiseuille_convergence % Figure: Convergence for 2D Poiseuille flow fds_moody_chart % Figure: The FDS Moody chart yplus % Figure: Title heated_channel % Figure: Heated Channel Flow }}}
== Creating your own script ==
In this example, we are going to write our own plotting script because some degree of post-processing calculations are necessary to non-dimensionalize the velocity and temperature results for comparison to the DNS data. The script we created is [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/scripts/heated_channel.m heated_channel.m] (commit) and this script is committed to the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%2Fscripts Matlab/scripts] directory.
Here are a few things to remember when creating your own script:
Add [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk/Utilities/Matlab/scripts/plot_style.m plot_style] at the top of your script. This sets many of the default Matlab parameters, like !FontSize and !FontName. Refer to our example script to see how to use these parameters to control the axis fonts and so on.
Check to make sure any necessary files exist and exit gracefully if they do not. This prevents subsequent cases from being omitted if one particular case fails in our nightly build. Here is a basic example to follow:
{{{ if ~exist(filename) display(['Error: File ' filename ' does not exist. Skipping case.']) return end }}}
Add the SVN number to the top right of the plot. The X_SVN_Position and Y_SVN_Position are set differently for linear, loglog, semilogx, and semilogy plots. Browse the examples in the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%2Fscripts Matlab/scripts] directory to find one that matches your situation and simply copy the syntax. The only thing you should need to modify is the SVN_Filename. If you have a series of cases all on one plot, simple choose one of the case CHIDs as your filename.
Print the plot to pdf. Here again, find an example in [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%2Fscripts Matlab/scripts] and copy it.
== Generating the Plots ==
Before you are done with Matlab, you should be able to delete all the files in the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide%2FSCRIPT_FIGURES SCRIPTS_FIGURES] directory, cd (change directory) into the [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FUtilities%2FMatlab%253Fstate%253Dclosed Utilities/Matlab] directory within the Matlab Command Window
{{{
cd /FDS-SMV/Utilities/Matlab }}}
and run
{{{
FDS_verification_script }}}
A pdf of your plot should be created in [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide%2FSCRIPT_FIGURES SCRIPTS_FIGURES]. If you have gotten this far, congratulations, time to actually do some writing.
If your plot does not show up, or if Matlab chokes, the most likely culprit is the path, either the path you set to the data directory or the path you set to the output plot directory.
= Editing the Verification Guide =
The LaTeX source for the guide is located in [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide Manuals/FDS_Verification_Guide/FDS_Verification_Guide.tex] (commit). Edit this file with whatever editor you choose (I use TeXShop on a Mac). Decide where your section should reside. In our case, we will add a section to the Wall Flows chapter called Heated Channel Flow.
As with the Matlab script, conforming to the conventions we have established in the guide is basically a matter of copying and editing someone else's section. Here are a few things to look out for:
Follow the [http://www.nist.gov/pml/pubs/sp811/index.cfm NIST Guide for the Guide for the Use of the International System of Units (SI)] for mathematical style.
When starting the section, first clear the page, then add your section or subsection, and add the \textorpdfstring command.
{{{ \clearpage
\subsection{Heated Channel Flow (\texorpdfstring{{\bf heated_channel}}{heated_channel})} }}}
Group similar plots with the tabular environment. Look at other examples within the guide for side-by-side plots; most use a width of 3.2 inches. Add descriptive figure captions.
{{{
\begin{figure}[h]
\begin{tabular*}{\textwidth}{l@{\extracolsep{\fill}}r}
\includegraphics[width=3.2in]{SCRIPT_FIGURES/heated_channel_uplus.pdf} &
\includegraphics[width=3.2in]{SCRIPT_FIGURES/heated_channel_Tplus.pdf}
\end{tabular*}
\caption[Mean velocity and temperature profiles for heated channel flow]{\label{fig_heated_channel} Comparison of
Use the following rules for captions and short captions (short captions show up in the table of contents):
* End your actual figure or table caption with a period.
* Do not use a period at the end of the short caption (this conflicts with the dots to the page number).
* The short caption should not span more than one line, and it is better not to use the entire line.
* If you reference a case name in the captions, use {\bf casename}, not {\ct ...}.
If you cite references (hard to imagine that you would not have one or two), then be sure to add these (in alphabetical order) to the bibtex database in Manuals/Bibliography/FDS_general.bib (commit).
{{{ @TECHREPORT{Kim:1987, author = {J. Kim and P. Moin}, title = {{Transport of Passive Scalars in Turbulent Channel Flow}}, institution = {National Aeronautics and Space Administration}, year = {1987}, month = {September}, type = {Technical Memorandum}, number = {89463}, address = {Moffet Field, California}, } }}} Make sure to double check the reference at the end of the document. Look for correct capitalization, consistent naming conventions (J. Kim not John Kim), etc. Sometimes you will need to put curly brackets around acronyms. For example, NASA might come out as Nasa if you do not use {NASA}.
== Compile the Document ==
Depending on your computer setup, there are different ways to compile the document. But the basic sequence of operations is the same. You have to run LaTeX once, then run BibTeX once, then LaTeX two more times to resolve all references. In Linux it looks like this (note that you do not include the extension for the bibtex run): {{{ $ pdflatex FDS_Verification_Guide.tex $ bibtex FDS_Verification_Guide $ pdflatex FDS_Verification_Guide.tex $ pdflatex FDS_Verification_Guide.tex }}} In between each run you will see a bunch of cryptic log messages that fly by too fast to read. Hence the next step.
== Check for Errors ==
Before you commit your edits to the guide, (of course first update your repository) in a bash shell cd to [http://code.google.com/p/fds-smv/source/browse/trunk/FDS/trunk#trunk%2FManuals%2FFDS_Verification_Guide Manuals/FDS_Verification_Guide] and run {{{ $ ./make_guide.sh }}} This script will compile the LaTeX document by running pdflatex, bibtex, pfdlatex, pdflatex, which is the sequence needed to resolve all references in the document. The script then checks for warnings or error messages associated with include files (like plots) that are not found. If all goes well, you will get something like {{{ FDS Verification Guide built successfully! }}} If all is not well, you may see something like {{{ ! LaTeX Error: File `SCRIPT_FIGURES/heated_channel_Tplus' not found. }}}
== Commit the Document ==
Once you have compiled the guide without errors or warnings, go ahead and update your repository and commit the .tex and .bib files.
= Review =
If everything has been done correctly, then our automated user, Firebot, will be able to run your cases, generate your plots, and compile the guide while you slumber. If something has been missed, we will all wake to the following email: Firebot angry! Well, mildly upset. Not to worry. The great thing about the repository is that we all iterate and fix our errors as soon as possible. This is not to say we are cavalier about pushing up broken code or documents. But errors during development are somewhat expected.
Here is a checklist of the files we committed for this example:
- FDS input files, e.g., FDS-SMV/Verification/Turbulence/heated_channel_Pr_1p00_32.fds
- Experimental or analytical data files, e.g., FDS-SMV/Verification/Turbulence/heated_channel_dns_data.csv
- FDS-SMV/Verification/FDS_Cases.sh
- FDS-SMV/Utilities/Matlab/scripts/heated_channel.m (or add line to FDS-SMV/Utilities/Matlab/FDS_verification_dataplot_inputs.csv)
- FDS-SMV/Utilities/Matlab/scripts/wall_model.m (or add to master list in FDS-SMV/Utilities/Matlab/FDS_verification_script.m)
- FDS-SMV/Manuals/FDS{{{}}}!Verification_Guide/FDS{{{}}}!Verification_Guide.tex
- FDS-SMV/Manuals/Bibliography/FDS_general.bib
Thanks for your contribution!