Backtesting of Risk models

Just adding the time/frequency dimension difference to what was said above: model backtesting is a model performance technique which takes place on an ongoing basis (in particular for VaR, breaches need to be discussed as soon as feasible after they take place; if there are too many over some period of time, they need to be escalated). Model validation takes place when the model first gets implemented and then every few years gets revalidated.

If you think the methodology (conceptual approach, statistical facts used in it) or implementation of the model backtesting are non-trivial (non-mechanical, quantitative, assumption based), one could classify it as 'model' and could undergo validation itself.

"Validation" means that someone analyses the model and pronounces it fit to be used, usually subject to conditions such as ongoing performance monitoring, and restrictions on input. Good industry practices include 1 for the validator to be independent from the model developer 2 to review and re-validate important models periodically.

"Ongoing performance monitoring" means that the performance of the model in production and/or on some regression test cases is monitored for signs of deterioration.

Some tools are used both in "validation" and in "OPM".

VaR "backtesting" - since VaR predicts that a book will lose no more than $\$X$ on some number of days, it's easy to compare the VaR predictions with the "hypothetical profit and loss" (without fees and other cash flows that VaR does not take into account). If the number of days when the book loses more than VaR ("VaR breaches") is very different from what VaR calculation predicted, then this needs to be analyzed and explained. This exercise is usually done both for VaR validation and as part of VaR OPM.

(Other examples of tests done both during validation and OPM may be - benchmarking the model outcomes against observations and the outcomes of other models ; testing that the greeks from pricing models plugged into Taylor expansion explain profit and loss.

An example of a test usually done only during validation is re-implementing the same model independenly and comparing the outcomes. Other validation tasks might include checking that the model documentation is OK (describes the model in enough detail that someone not familiar with the model might reproduce it; explains choices made by the modeler and alternative approaches). The tasks in the periodic review (not a full-blown validation) might include reviewing the OPM results and checking that the OPM plan still makes sense; checking published research and industry practices to see if the model needs revalidation.)

In my opinion model validation is broader than model backtesting. During model backtesting you test model performance on data that has been realised using only the data you could have used when using the model for risk management. This gives you an idea on model accuracy and it allows you to find out where the model or submodels are inaccurate or misleading.

Model validation can include backtesting and also takes a deeper look at the model itself. Things to look at during model validation:

• Does the logic make sense
• Are the methods used appropriate, do other more appropriate methods exist?
• How sensitive is the model to changes in parameters and data
• Is the model code correct and does it handle all edge cases well (or at least flags them for attention)

and any other thing that convince management and yourself that the model can be used for its intended purpose.