Advances in verification and validation in computational fluid dynamics

CHEN Jiangtao; XIAO Wei; ZHAO Wei; ZHANG Peihong; YANG Fujun; JIN Tao; GUO Yongyan; WU Xiaojun; CHEN Jianqiang; WANG Ruili; LI Li

doi:10.6052/1000-0992-23-012

Volume 53 Issue 3

Sep. 2023

Turn off MathJax

Article Contents

Article Navigation > Advances in Mechanics > 2023 > 53(3): 626-660

Chen J T, Xiao W, Zhao W, Zhang P H, Yang F J, Jin T, Guo Y Y, Wu X J, Chen J Q, Wang R L, Li L. Advances in verification and validation in computational fluid dynamics. Advances in Mechanics, 2023, 53(3): 626-660 doi: 10.6052/1000-0992-23-012

Citation:

Chen J T, Xiao W, Zhao W, Zhang P H, Yang F J, Jin T, Guo Y Y, Wu X J, Chen J Q, Wang R L, Li L. Advances in verification and validation in computational fluid dynamics. Advances in Mechanics, 2023, 53(3): 626-660 doi: 10.6052/1000-0992-23-012

Citation:

PDF( 2472 KB)

Advances in verification and validation in computational fluid dynamics

doi: 10.6052/1000-0992-23-012

1.
China Aerodynamics Research and Development Centers, Mianyang Sichuan 621000, China
2.
Institute of Applied Physics and Computational Mathematics, Beijing 100094, China
3.
Xi’an Aeronautics Computing Technique Research Institute, AVIC, Xi’an 710000, China

More Information

Corresponding author: huang7766@sina.com; chenjq@cardc.cn
Received Date: 2023-03-16
Accepted Date: 2023-06-05

Available Online: 2023-06-06

Publish Date: 2023-09-30

Abstract

Abstract

Computational fluid dynamics (CFD) has played an increasingly important role in major engineering fields, and its credibility is the key constraint to its further extensive engineering application. It is widely accepted home and abroad that verification and validation is the only way to evaluate and guarantee the credibility of CFD. Through systematic verification and validation, the potential programming errors can be effectively identified, the reliability of numerical solving process can be guaranteed, the adequacy and prediction capability of mathematical models in the intended use can be objectively evaluated and improved when necessary. In this paper, with regard to two key issues, ‘‘what is verification and validation’’ and ‘‘how to perform verification and validation’’, the research progress of verification and validation in CFD is introduced from the aspects including basic concept, implementation processes, main methods, calibration model experiments and platform tools, with focusing on numerical error estimation and uncertainty quantification. At the end, the shortcomings of current research are reviewed and the key research directions are prospected.
- verification and validation,
- error estimation,
- uncertainty quantification,
- calibration model experiment

FullText(HTML)

References(144)

References

[1]	陈坚强. 2021. 国家数值风洞(NNW)工程关键技术研究进展. 中国科学:技术科学, 51: 1326-1347 (Chen J Q. 2021. Advances in the key technologies of Chinese national numerical windtunnel project. Scientia Sinica Technologica, 51: 1326-1347). doi: 10.1360/SST-2020-0334 Chen J Q. 2021. Advances in the key technologies of Chinese national numerical windtunnel project. Scientia Sinica Technologica, 51: 1326-1347). doi: 10.1360/SST-2020-0334
[2]	陈树生, 刘丽媛, 闫超, 等. 2017. CFD软件自动化验证确认云平台设计与实现. 航空学报, 38: 120209 (Chen S S, Liu L Y, Yan C, et al. 2017. Design and realization of automated testing cloud platform for CFD verification and validation. Acta Aeronautica et Astronautica Sinica, 38: 120209). Chen S, Liu L, Yan C, et al. 2017. Design and realization of automated testing cloud platform for CFD verification and validation. Acta Aeronautica et Astronautica Sinica, 38: 120209).
[3]	梁益华, 杨永, 朱朝. 2004. CFD 可信度分析平台 WiseCFD. 第十二届全国计算流体力学会议论文, 775-780 Liang Y H, Yang Y, Zhu C. 2004. CFD credibility analysis platform WiseCFD. XT, 775-780.
[4]	刘智益, 王晓东, 康顺. 2013. 叶顶间隙尺度的不确定性对压气机性能影响的CFD模拟. 工程热物理学报, 34: 628-631 (Liu Z Y, Wang X D, Kang S. 2013. CFD simulation of uncertain tip clearance effect on compressor performance. Journal of Engineering Thermophysics, 34: 628-631). Liu Z Y, Wang X D, Kang S. 2013. CFD simulation of uncertain tip clearance effect on compressor performance. Journal of Engineering Thermophysics, 34: 628-631).
[5]	邵帅, 李明, 王年华, 等. 2018. 基于非结构/混合网格模拟黏性流的高阶精度DDG/FV 混合方法. 力学学报, 50: 1470-1482 (Shao S, Li M, Wang N H, et al. 2018. High-order DDG/FV hybrid method for viscous flow simulation on unstructured/hybrid grids. Chinese Journal of Theoretical and Applied Mechanics, 50: 1470-1482). Shao S, Li M, Wang N H, et al. 2018. High-order DDG/FV hybrid method for viscous flow simulation on unstructured/hybrid grids. Chinese Journal of Theoretical and Applied Mechanics, 50: 1470-1482).
[6]	宋赋强, 阎超, 马宝峰, 等. 2018. 锥导乘波体构型的气动特性不确定度分析. 航空学报, 39: 121519 (Song F Q, Yan C, Ma B F, et al. 2018. Uncertainty analysis of aerodynamic characteristics for cone-derived waverider configuration. Acta Aeronautica et Astronautica Sinica, 39: 121519). Song F Q, Yan C, Ma B F. 2018. Uncertainty analysis of aerodynamic characteristics for cone-derived waverider configuration. Acta Aeronautica et Astronautica Sinica, 39: 121519 ).
[7]	汤涛, 周涛. 2015. 不确定性量化的高精度数值方法和理论. 中国科学:数学, 45: 891-928 (Tang T, Zhou T. 2015. Recent developments in high order numerical methods for uncertainty quantification. Scientia Sinica:Mathematica, 45: 891-928). doi: 10.1360/N012014-00218 Tang T, Zhou T. 2015. Recent developments in high order numerical methods for uncertainty quantification. Scientia Sinica: Mathematica, 45: 891-928). doi: 10.1360/N012014-00218
[8]	王年华, 张来平, 赵钟, 等. 2017. 基于制造解的非结构二阶有限体积离散格式的精度测试与验证. 力学学报, 49: 627-637 (Wang N H, Zhang L P, Zhao Z, et al. 2017. Accuracy verification of unstructured second-order finite volume discretization schemes based on the method of manufactured solutions. Chinese Journal of Theoretical and Applied Mechanics, 49: 627-637). Wang N H, Zhang L P, Zhao Z, et al. 2017. Accuracy verification of unstructured second-order finite volume discretization schemes based on the method of manufactured solutions. Chinese Journal of Theoretical and Applied Mechanics, 49: 627-637).
[9]	王瑞利, 林忠, 袁国兴. 2010. 科学计算程序的验证和确认. 北京理工大学学报, 30: 353-360 (Wang R L, Lin Z, Yuan G X. 2010. Verification and validation in scientific computing code. Transactions of Beijing Institute of Technology, 30: 353-360). Wang R L, Lin Z, Yuan X G. 2010. Verification and validation in scientific computing code. Transactions of Beijing Institute of Technology, 30: 353-360).
[10]	王运涛, 刘刚, 陈作斌. 2019. 第一届航空CFD可信度研讨会总结. 空气动力学报, 37: 247-261 (Wang Y T, Liu G, Chen Z B. 2019. Summary of the first aeronautical computational fluid dynamics credibility workshop. Acta Aerodynamica Sinica, 37: 247-261). Wang Y T, Liu G, Chen Z B. 2019. Summary of the first aeronautical computational fluid dynamics credibility workshop. Acta Aerodynamica Sinica, 37: 247-261.
[11]	邬晓敬, 张伟伟, 宋述芳, 等. 2015. 翼型跨声速气动特性的不确定性及全局灵敏度分析. 力学学报, 47: 587-595 (Wu X J, Zhang W W, Song S F, et al. 2015. Uncertainty quantification and global sensitivity analysis of transonic aerodynamics about airfoil. Chinese Journal of Theoretical and Applied Mechanics, 47: 587-595). doi: 10.6052/0459-1879-14-372 Wu X, Zhang W, Song S, et al. 2015. Uncertainty quantification and global sensitivity analysis of transonic aerodynamics about airfoil. Chinese Journal of Theoretical and Applied Mechanics, 47: 587-595). doi: 10.6052/0459-1879-14-372
[12]	熊芬芬, 王瑞利, 吴晓军, 等. 2023. 不确定性量化方法及应用. 北京: 科学出版社.
[13]	夏侯唐凡, 陈江涛, 邵志栋, 等. 2022. 随机和认知不确定性框架下的CFD 模型确认度量综述. 航空学报, 43: 025716 (Xiahou T F, Chen J T, Shao Z D, et al. 2022. Model validation metrics for CFD numerical simulation under aleatory and epistemic uncertainty. Acta Aeronautica et Astronautica Sinica, 43: 025716). (Xiahou T F, Chen J T, Shao Z D, et al. 2022. Model validation metrics for CFD numerical simulation under aleatory and epistemic uncertainty. Acta Aeronautica et Astronautica Sinica, 43: 025716).
[14]	肖思男, 吕震宙, 王薇. 2018. 不确定性结构全局灵敏度分析方法概述. 中国科学:物理学力学天文学, 48: 014601 (Xiao S N, Lv Z Z, Wang W. 2018. A review of global sensitivity analysis for uncertainty structure. Scientia Sinica Physica, Mechanica & Astronom1ca, 48: 014601). Xiao S N, Lv Z Z, Wang W. 2018. A review of global sensitivity analysis for uncertainty structure. Scientia Sinica Physica, Mechanica & Astronom1ca, 48: 014601).
[15]	肖钊, 韩旭, 杨刚. 2014. 基于区间技术的模型确认方法及应用. 机械工程学报, 50: 177-184 (Xiao Z, Han X, Yang G. 2014. Model validation method and its application based on the interval techniques. Journal of Mechanical Engineering, 50: 177-184). Xiao Z, Han X, Yang G. 2014. Model validation method and its application based on the interval techniques. Journal of Mechanical Engineering, 50: 177-184).
[16]	赵辉, 胡星志, 张健, 等. 2019. 湍流模型系数的不确定度对翼型绕流模拟的影响. 航空学报, 40: 122581 (Zhao H, Hu X Z, Zhang J, et al. 2019. Effects of uncertainty in turbulence model coefficients on flow over airfoil simulation. Acta Aeronautica et Astronautica Sinica, 40: 122581). Zhao H, Hu X Z, Zhang J, et al. 2019. Effects of uncertainty in turbulence model coefficients on flow over airfoil simulation. Acta Aeronautica et Astronautica Sinica, 40: 122581).
[17]	赵炜, 陈江涛, 肖维, 等. 2020. 国家数值风洞工程(NNW)验证与确认系统关键技术研究进展. 空气动力学学报, 38: 1165-1172 (Zhao W, Chen J T, Xiao W, et al. 2020. Advances in the key technologies of verification and validation system of national numerical windtunnel project,. Acta Aerodynamics Sinica, 38: 1165-1172). Zhao W, Chen J T, Xiao W, et al. 2020. Advances in the key technologies of verification and validation system of national numerical windtunnel project. Acta Aerodynamics Sinica, 38: 1165-1172).
[18]	章超, 刘骁, 陈江涛, 等. 2020. 烧蚀热响应计算中的不确定性传播分析方法研究. 宇航学报, 41: 1401-1409 (Zhang C, Liu X, Chen J T, et al. 2020. Study on uncertainty propagation anslysis method in ablative thermal response calculation. Journal of Astronautics, 41: 1401-1409). doi: 10.3873/j.issn.1000-1328.2020.11.005 Zhang C, Liu X, Chen J T, et al. 2020. Study on uncertainty propagation anslysis method in ablative thermal response calculation. Journal of Astronautics, 41: 1401-1409). doi: 10.3873/j.issn.1000-1328.2020.11.005
[19]	张伟, 王小永, 于剑, 等. 2018. 来流导致的高超声速气动热不确定度量化分析. 北京航空航天大学学报, 44: 1102-1109 (Zhang W, Wang X Y, Yu J, et al. 2018. Uncertainty quantification analysis in hypersonic aerothermodynamics due to freestream uncertainties. Journal of Beijing University of Aeronautics and Astronautics, 44: 1102-1109). doi: 10.13700/j.bh.1001-5965.2017.0303 Zhang W, Wang X Y, Yu J, et al. 2018. Uncertainty quantification analysis in hypersonic aerothermodynamics due to freestream uncertainties. Journal of Beijing University of Aeronautics and Astronautics, 44: 1102-1109). doi: 10.13700/j.bh.1001-5965.2017.0303
[20]	中国科学院. 2018. 新型飞行器中的关键力学问题. 北京: 科学出版社.
[21]	AIAA (American Institute of Aeronautics and Astronautics). 1998. Guide for the verification and validation of computational fluid dynamics simulations, AIAA-G-077-1998.
[22]	AIAA. 1999. Assessment of experimental uncertainty with application to wind tunnel testing, AIAA S-071A-1999.
[23]	AIAA. 2003. Assessing experimental uncertainty supplement to AIAA S-071A-1999, AIAA G-045-2003.
[24]	Arendt P D, Apley D W, Chen W. 2012. Quantification of model uncertainty: calibration, model discrepancy, and identifiability. Journal of Mechanical Design, 134: 100908. doi: 10.1115/1.4007390
[25]	ASME (American Society of Mechanical Engineers). 2005. Test uncertainty. ASME PTC 19.1-2005.
[26]	ASME. 2006. Guide for verification and validation in computational solid mechanics. ASME V&V 10-2006.
[27]	ASME. 2009. Standard for verification and validation in computational fluid dynamics and heat transfer.
[28]	ASME. 2012. An illustration of the concepts of verification and validation in computational solid mechanics. ASME V&V 10.1-2012.
[29]	ASME. 2019. Standard for verification and validation in computational solid mechanics. ASME V&V 10-2019.
[30]	Andrianov G, Burriel S, Cambier S, et al. 2007. OpenTURNS, an open source initiative to treat uncertainties, Risks'N statistics in a structured industrial approach. In: Proceedings of the ESREL’2007 Safety and Reliability Conference, Stavenger: Norway.
[31]	Avdonin A, Polofke W. 2019. Quantification of the impact of uncertainties in operating conditions on the flame transfer function with nonintrusive polynomial chaos expansion. Journal of Engineering for Gas Turbines and Power, 141: 011020. doi: 10.1115/1.4040745
[32]	Babuska I, Nobile F, Tempone R. 2007. A stochastic collocation method for elliptic partial differential equations with random input data. SIAM J Numer Anal, 45: 1005-1034. doi: 10.1137/050645142
[33]	Bhattacharyya B. 2022. Uncertainty quantification of dynamical systems by a POD-kriging surrogate model. Journal of Computational Science, 60: 101602. doi: 10.1016/j.jocs.2022.101602
[34]	Blatman G, Sudret B. 2010. An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis. Probab Eng Mech, 25: 183-97. doi: 10.1016/j.probengmech.2009.10.003
[35]	Blatman G, Sudret B. 2011. Adaptive sparse polynomial chaos expansion based on least angle regression. Journal of Computational Physics, 230: 2345-67. doi: 10.1016/j.jcp.2010.12.021
[36]	Blottner F G. 1990. Accurate Navier-Stokes results for the hypersonic flow over a spherical nosetip. Journal of Spacecraft & Rockets, 27: 113-122.
[37]	Brehm C, Hader C, Fasel H F. 2015. A locally stabilized immerse boundary method for the compressible Navier-Stokes equations. Journal of Computational Physics, 295: 475-504. doi: 10.1016/j.jcp.2015.04.023
[38]	Bungartz H J, Griebel M. 2004. Sparse grids. Acta Numer, 13: 1-123. doi: 10.1017/S0962492904000169
[39]	Burg C, Murali V K. 2004. Efficient code verification using the residual formulation of the method of manufactured solution. AIAA Paper, 2628.
[40]	Cary A W, Chawner J R, Earl P N, et al. 2021. CFD vision 2030 roadmap: progress and perspectives. AIAA Aviation Forum.
[41]	Casey M, Wintergerste T, Innotec S. 2000. Ercoftac special interest group on quality and trust in industrial CFD: Best practice guidelines. Brussels: European Research Community on Flow, Tutbulence and Combusion.
[42]	Choudhary A, Roy C J, Dietiker J F, et al. 2014. Code verification for multiphase flows using the method of manufactured solutions. International Journals of Multiphase Flow, 80: 150-163.
[43]	DoD. 1994. DoD Directive No. 5000.59: Modeling and simulation (M&S) management.
[44]	DoD. 1996. DoD Instruction 5000.61: Modeling and simulation (M&S) verification, validation, and accreditation (VV&A). Defense Modeling and Simulation Office, Office of the Director of Defense Research and Engineering.
[45]	DoD. 1997. DoD Modeling and simulation glossary. from www. msco. mil.
[46]	DoD. 2008. Department of defense standard practice: documentation of verification, validation, and accreditation (VV&A) for models and simulations. US Washington, DC, Department of defense.
[47]	Duan Q, Sorooshian S, Gupta V. 1992. Effective and efficient global optimization for conceptual rainfall-runoff models. Water Resour. Res., 28: 1015-1031. doi: 10.1029/91WR02985
[48]	Dunn M C, Shotorban B, Frendi A. 2011. Uncertainty quantification of turbulence model coefficients via Latin hypercube sampling method. Journal of Fluids Engineering, 133: 2913-2921.
[49]	Eça L, Hoekstra M. 2014. A procedure for the estimation of the numerical uncertainty of CFD calculations based on grid refinemen tstudies. Journal of Computational Physics, 262: 104-130. doi: 10.1016/j.jcp.2014.01.006
[50]	Eca L, Klaij C M, Vaz G, et al. 2016. On code verification of RANS solvers. Journal of Computational Physics, 310: 418-439. doi: 10.1016/j.jcp.2016.01.002
[51]	Eça L, Vaz G, Pereira F S, et al. 2019. Numerical and parameter uncertainties: Are they independent?// Verification and Validation Symposium, VVS 2019.
[52]	Eça L, Pereira F S, et al. 2020. On the role of discretization errors in the quantification of parameter uncertainties. Proceedings of ASME 2020 verification and validation symposium, VVS2020-8825, virtual, online.
[53]	Adams B M, Bauman L E, Bohnhoff W J, et al. 2009. Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: Version 5.4 user's manual. Sandia Technical Report SAND2010-2183.
[54]	Ferson S, Oberkampf W L. 2009. Validation of imprecise probability models. International Journal of Reliability and Safety, 3: 3-22. doi: 10.1504/IJRS.2009.026832
[55]	Ferziger J H, Peric M. 1996. Further discussion of numerical errors in CFD. International Journal for Numerical Methods in Fluids, 23: 1263-1274. doi: 10.1002/(SICI)1097-0363(19961230)23:12<1263::AID-FLD478>3.0.CO;2-V
[56]	Ferziger J H. 1998. A note on numerical accuracy. International Journal for Numerical Methods in Fluids, 8: 995-996.
[57]	Ferziger J H, Peric M. 2002. Computational methods for fluid dynamics/ 3rd edn. Berlin, Springer-Verlag.
[58]	Fishman G S. 1996. Monte carlo: concepts, algorithms, and applications. New York: Springer-Verlag.
[59]	Gan Y J, Duan Q Y, Gong W, et al. 2014. A comprehensive evaluation of various sensitivity analysis methods: A case study with a hydrological model. Environmental Modelling & Software, 51: 269-285.
[60]	Ghanem R, Spanos P. 1992. Stochastic finite elements: A spectral approach. New York: Springer-Verlag.
[61]	Ghia U, Bayyuk S, Roy C, et al. 2010. The AIAA code verification project-test cases for CFD code verification. AIAA Paper, 125.
[62]	Golub G H, Van Loan C F. 1996. Matrix computations, 3rd edn. Baltimore. The Johns Hopkins University Press.
[63]	Grier B, Alyanak E, White M, et al. 2014. Numerical integration techniques for discontinuous manufactured solutions. Journal of Computational Physics, 278: 193-203. doi: 10.1016/j.jcp.2014.08.031
[64]	Grier B, Figliola R, Alyanak E, et al. 2015. Discontinuous solutions using the method of manufactured solutions on finite volume solvers. AIAA J, 53: 2369-2378. doi: 10.2514/1.J053725
[65]	Hallissy B P, Hariharan N S, Laiosa J P, et al. 2014. CREATETM-AV quality assurance: best practices for validating and supporting computation-based engineering software. AIAA Paper, 0918.
[66]	Hebert S, Luke E A. 2005. Honey, I shrunk the grids! A new approach to CFD verification studies. AIAA Paper, 685.
[67]	Heeg J, Wieseman C D, Chwalowsiki P. 2016. Data comparisons and summary of the second aeroelastic prediction workshop. AIAA 2016-3121.
[68]	Hirsch C. 2008. The development of a framework for CFD validation and best practice: the QNET-CFD knowledge base. Chinese Journal of Aeronautic, 19: 105-113.
[69]	Huang H, Giacobello M. 2022. Uncertainty quantification in store separation analysis using kestrel, Design of experiments and surrogate modelling. AIAA2022-1318. AIAA Scitech 2022 Forum San Diego, CA & Virtual.
[70]	IEEE. 1984. IEEE standard dictionary of electrical and electronics terms. ANSI/IEEE Std 100-1984, New York.
[71]	Jiang X, Mahadevan S. 2007. Bayesian risk-based decision method for model validation under uncertainty. Reliability Engineering & System Safety, 92: 707-718.
[72]	Kawai S, Shimoyama K. 2014. Kriging-model-based uncertainty quantification in computational fluid dynamics. 32nd AIAA Applied Aerodynamics Conference.
[73]	Kennedy M C, O'Hagan A. 2001. Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B, 3: 425-464.
[74]	Kirkpatrick S, Gelatt C D, Vecchi A. 1983. Optimization by simulated annealing. Science, 220: 671-680. doi: 10.1126/science.220.4598.671
[75]	Lax P D. 1954. Weak solutions of nonlinear hyperbolic equations and their numerical computation. Commun. Pure Appl. Math., 7: 159-193. doi: 10.1002/cpa.3160070112
[76]	Ling Y, Mahadevan S. 2013. Quantitative model validation techniques: new insights. Reliability Engineering & System Safety, 111: 217-231.
[77]	Liu D, Litvinenko A, Schillings C, et al. 2017. Quantification of airfoil geometry-induced aerodynamic uncertainties--comparison of approaches. Society for Industrial and Applied Mathematics, 5: 334-352.
[78]	Liu S, Wang Y, Qin N, et al. 2020. Quantification of airfoil aerodynamic uncertainty due to pressure-sensitive paint thickness. AIAA J, 58: 1432-1440. doi: 10.2514/1.J058801
[79]	Liu W, Belytschko T, Mani A. 1986a. Probabilistic finite elements for nonlinear structural dynamics. Comput Methods Appl Mech Engrg, 56: 61-81. doi: 10.1016/0045-7825(86)90136-2
[80]	Liu W, Belytschko T, Mani A. 1986b. Random field finite elements. Internat J Numer Methods Engrg, 23: 1831-1845. doi: 10.1002/nme.1620231004
[81]	Liu X, Furrer D, Kosters J, et al. 2018. Vision 2040: A roadmap for integrated, multiscale modeling and simulation of materials and systems. NASA/CR-2018-219771.
[82]	Liu Y, Chen W, Arendt P, et al. 2011. Toward a better understanding of model validation metrics. ASME Journal of Mechanical Design, 133: 071005. doi: 10.1115/1.4004223
[83]	Loeven A, Bijl H. 2008. Airfoil analysis with uncertain geometry using the probabilistic collocation method. 49th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Schaumburg, IL, AIAA 2008-2070.
[84]	Mahaffy, J. 2007. Best practice guidelines for the use of CFD in nuclear reactor safety applications. OECD, Nuclear Energy Agency, Technical Report, NEA/CSNI/R(2007)5.
[85]	Marelli S, Sudret B. 2014. UQLab: A framework for uncertainty quantification in matlab. Int Conf on Vulnerability, Risk Analysis & Management, 2554-2563.
[86]	Marini M, Paoli R, Grasso F, et al. 2002. Verification and validation in computational fluid dynamics: The flownet databese experience. Bulletin of the JSME, 45: 15-22.
[87]	Mariotti A, Salvetti M V, Omrani P S, et al. 2016. Stochastic analysis of the impact of freestream conditions on the aerodynamics of a rectangular 5: 1 cylinder. Computers & Fluids, 136: 170-192.
[88]	Marshall D D. 2011. A scientific software verification library based on the method of manufactured solutions. AIAA Paper, 615.
[89]	NRC (National Research Council). 2012. Assessing the reliability of complex models: mathematical and statistical foundations of verification, validation, and uncertainty quantification. The National Academies Press, Washington: D.C.
[90]	Murali V K, Burg C. 2002. Verification of 2D Navier-Stokes codes by the method of manufactured solutions. AIAA Paper, 3109.
[91]	NASA (National Aeronautics and Space Administration). 2016a. Standard for models and simulations. NASA-STD-7009A, superseding NASA-STD-7009, 2008.
[92]	NASA. 2016b. Simulation credibility advances in verification, validation and uncertainty quantification. NASA/TP-2016-219422 JANNAF/GL-2016-0001.
[93]	Navah F, Nadarajah S. 2016. On the verification of high-order CFD solvers. VII European Congress on Computational Methods in Applied Sciences and Engineering, Crete: Greece.
[94]	Niederreiter H. 1992. Random number generation and quasi-monte carlo methods. Philadelphia: SIAM.
[95]	Niederreiter H, Hellekalek P, Larcher G, et al. 1998. Monte carlo and quasi-monte carlo methods. New York: Springer-Verlag.
[96]	Norman R, Philip D. 2006. The Navier-Stokes equations: a classification of flows and exact solutions. Cambridge University Press.
[97]	Oberkampf W L, Roy C J. 2010. Verification and validation in scientific computing. Cambridge University Press.
[98]	Oberkampf W L, Smith B L. 2017. Assessment criteria for computational fluid dynamics model validation experiments. ASME J. Verif. , Validation Uncertainty Quantif., 2: 031002. doi: 10.1115/1.4037887
[99]	Oberkampf W L, Trucano T G. 2000. Validation methodology in computational fluid dynamics. Fluids 2000 Conference AIAA 2000-2549.
[100]	Park I, Amarchinta H K, Grandhi R V. 2010. A bayesian approach for quantification of model uncertainty. Reliab Eng Syst Saf, 95: 777-785. doi: 10.1016/j.ress.2010.02.015
[101]	Nair P B, Keane A J. 2002. Stochastic reduced basis methods. AIAA J, 40: 1653-64. doi: 10.2514/2.1837
[102]	Platteeuw P, Loeven G, Bijl H. 2008. Uncertainty quantification applied to the k-epsilon model of turbulence using the probabilistic collocation method. 49th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Schaumburg, IL. Reston, Virginia: AIAA.
[103]	Raisee M, Kumar D, Lacor C. 2015. A non-intrusive model reduction approach for polynomial chaos expansion using proper orthogonal decomposition. Int J Numer Methods Eng, 103: 293-312. doi: 10.1002/nme.4900
[104]	Ricci P, Riva F, Theiler C, et al. 2015. Approaching the investigation of plasma turbulence through a rigorous verification and validation procedure: A practical example. Physics of Plasma, 22: 055704. doi: 10.1063/1.4919276
[105]	Richard G H, David C M, Jonathan W N. 2015. V&V framework. Sandia National Labs. Rept. SAND2015-7455, Albuquerque, NM.
[106]	Riley M E, Grandhi R V. 2011. Quantification of model-form and predictive uncertainty for multi-physics simulation. Computers and Structures, 89: 1206-1213. doi: 10.1016/j.compstruc.2010.10.004
[107]	Roache P J. 1998. Verification and validation in computational science and engineering. Albuquerque, NM, Hermosa Publishers.
[108]	Roy C J, Oberkampf W L. 2011. A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Computer Methods in Applied Mechanics and Engineering, 200: 2131-2144. doi: 10.1016/j.cma.2011.03.016
[109]	Roy C, Blottner F G. 2003. Methodology for turbulence model validation: application to hypersonic flows. Journal of Spacecraft and Rockets, 40: 313-325. doi: 10.2514/2.3966
[110]	Roy C J, Nelson C C, Smith T M, et al. 2004. Verification of Euler/Navier-Stokes codes using the method of manufactured solutions. International Journal for Numerical Methods in Fluids, 44: 599-620. doi: 10.1002/fld.660
[111]	Rumpfkeil M P, Hanazaki K, Beran P S. 2017. Construction of multi-fidelity locally optimized surrogate models for uncertainty quantification. 19th AIAA Non-Deterministic Approaches Conference.
[112]	Rumsey C L, Slotnick J P, Sclafani A J. 2019. Overview and summary of the third AIAA high lift prediction workshop. Journal of Aircraft, 56: 621-644. doi: 10.2514/1.C034940
[113]	Saltelli A, Tarantola S, Campolongo F, et al. 2004. Sensitivity analysis in practise: a guide to assessing scientific models. John Wiley & Sons, Ltd, Chichester, West Sussex.
[114]	Schlesinger S. 1979. Terminology for model credibility. Simulation, 32: 103-104. doi: 10.1177/003754977903200304
[115]	Schaefer J A, Cary A W, Mani M, et al. 2017. Uncertainty quantification and sensitivity analysis of sa turbulence model coefficients in two and three dimensions. 55th AIAA Aerospace Sciences Meeting.
[116]	Sentz K, Ferson S. 2002. Combination of evidence in dempster-shafer theory. Contemporary Pacific.
[117]	Shafer G. 1976. A mathematical theory of evidence. Princeton University Press.
[118]	Shu C W. 1998. Essentially non-oscillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws. Institute for Computer Applications in Science and Engineering (ICASE), 1697: 325-432.
[119]	Slotnick J, et al. 2014. CFD vision 2030 study: A path to revolutionary computational aerosciences. NASA/CR-2014-218178.
[120]	Smolyak S. 1963. Quadrature and interpolation formulas for tensor products of certain classes of functions. Dokl Akad Nauk SSSR, 4: 1042-1045.
[121]	Sod G. 1978. A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws. J. Comput. Phys., 27: 1-31. doi: 10.1016/0021-9991(78)90023-2
[122]	Stein M. 1987. Large sample properties of simulations using latin hypercube sampling. Technometrics, 29: 143-151. doi: 10.1080/00401706.1987.10488205
[123]	Swiler L P, Eldred M S. 2009. Efficient algorithms for mixed aleatory-epistemic uncertainty quantification with application to radiation-hardened electronics. Sandia National Labs, Rept. SAND2009-5805, Albuquerque, NM.
[124]	Swiler L P, Mayes R L, Eldred M S. 2009. Epistemic uncertainty in the calculation of margins. 50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA Paper, 2249, Palm Springs, CA.
[125]	Thomas J L, Diskin B, Rumsey C L. 2008. Toward verification of unstructured-grid solvers. AIAA J, 46: 3070-3079. doi: 10.2514/1.36655
[126]	Tinoco E N, Brodersen O P, Keye S, et al. 2018. Summary data from the sixth AIAA CFD drag prediction workshop: CRM cases. Journal of Aircraft, 55: 1352-1379. doi: 10.2514/1.C034409
[127]	Tong C. 2005. Psuade user's manual. Lawrence Livermore National Laboratory (LLNL), Livermore: CA.
[128]	Venditti D A and Darmofal D L. 2000. Adjoint error estimation and grid adaptation for functional outputs: application to quasi-one-dimensional flow. Journal of Computational Physics, 164: 204-227. doi: 10.1006/jcph.2000.6600
[129]	Vidanovic N, Rasuo B, Damljanovic D, et al. 2014. Validation of the CFD code for determination of aerodynamic characteristics of nonstandard AGARD-B calibration model. Thermal Science, 18.
[130]	Voyles I T, Roy C J. 2015. Evaluation of model validation techniques in the presence of aleatory and epistemic input uncertainties. AIAA 2015-1374. AIAA SciTech, Kissimmee, Florida.
[131]	Wang C, Duan Q Y, Gong W, et al. 2014. An evaluation of adaptive surrogate modeling based optimization with two benchmark problems. Environ. Model. Softw, 60: 167-179. doi: 10.1016/j.envsoft.2014.05.026
[132]	Wang C, Duan Q Y, Charles H T, et al. 2016. A GUI platform for uncertainty quantification of complex dynamical models. Environmental Modelling & Software, 76: 1-12.
[133]	Wang N, Yao W, Zhao Y, et al. 2018. A new interval area metric for model validation with limited experimental data. ASME Journal of Mechanical Design, 140: 1-11.
[134]	Wang Y J, Zhang S D. 2016. Uncertainty quantification of numerical simulation of flows around a cylinder using non-intrusive polynomial chaos. Chinese Physics Letters, 33: 090501. doi: 10.1088/0256-307X/33/9/090501
[135]	Loh W. 1996. On Latin hypercube sampling. Ann Statist, 24: 2058-2080.
[136]	Wiener N. 1938. The homogeneous chaos. Amer J Math, 60: 897-936. doi: 10.2307/2371268
[137]	Wilson G E, Boyack B E. 1998. The role of the PIRT process in experiments, code development and code applications associated with reactor safety analysis. Nuclear Engineering and Design, 186: 23-37. doi: 10.1016/S0029-5493(98)00216-7
[138]	Woods C N, Starkey R P. 2015. Verification of fluid-dynamic codes in the presence of shocks and other discontinuities. Journal of Computational Physics, 294: 312-328. doi: 10.1016/j.jcp.2015.03.055
[139]	Xiu D, Karniadakis G E. 2002. The wiener-askey polynomial chaos for stochastic differential equations. SIAM J Sci Comput, 24: 619-644. doi: 10.1137/S1064827501387826
[140]	Xiu D, Hesthaven J S. 2005. High-order collocation methods for differential equations with random inputs. SIAM J Sci Comput, 27: 1118-1139. doi: 10.1137/040615201
[141]	Xiu D. 2007. Efficient collocational approach for parametric uncertainty analysis. Commun Comput Phys, 2: 293-309.
[142]	Zhang D X. 2002. Stochastic methods for flow in porous media. New York: Academic Press.
[143]	Zhu H Y, Wang G, Liu Y, et al. 2020. Numerical investigation of transonic buffet on supercritical airfoil considering uncertainties in wind tunnel testing. International Journal of Modern Physics B, 34: 2040083. doi: 10.1142/S0217979220400834
[144]	Zio E, Apostolakis G E. 1996. Two methods for the structured assessment of model uncertainty by experts in performance assessments of radioactive waste repositories. Reliab Eng Syst Saf, 54: 225-241. doi: 10.1016/S0951-8320(96)00078-6