The Model Validator’s Manifesto

The actuary’s role in modelling is evolving and transforming. As model complexity has increased, the need for robust validation similarly increases. The modern-day actuary’s contribution is advancing to include independent oversight and effective challenge in the process of modelling.

Model validation has gained popularity due the increased reliance on many types of models throughout organisations. Regulators mandate the need for robust and reliable validation of certain models: internal capital models, reserving models and pricing models with the primary purpose of reassuring societal and investor confidence. While models will always have inherent limitations, the aim of validation is not to disregard these limitations but to provide confidence in the use of the model given the current knowledge and available information.

Validation is a process requiring several steps. Validators are expected to know the extent of validation required for each model, often referred to as the validation scope. This often depends on a model’s use: its material importance in the business, the frequency of model usage and the number of users.

As actuaries represent a data-based profession, clear rules and good validation is needed. Some good work has already been done; for example, the UK Model Risk working group wrote an extended discussion on model risk and use in 2015[1].

Other groups have taken this idea even further. The Financial Modeller’s Manifesto was produced by Emanuel Derman and Paul Wilmott. It is a Hippocratic Oath for all financial professionals and it encourages more responsibility in quantitative finance and risk management as a rejoinder to the subprime mortgage crisis.

In a somewhat similar humorous and witty vein, I present the Model Validator’s Manifesto and Hippocratic Oath:

The Model Validator’s Hippocratic Oath

1. I will avoid validating models that I am a parent or emotionally attached too.
2. I will make sure I have a good enough understanding of the whole governance process before validating it.
3. I will use easily repeatable checks in my validation so that can be understood by a lay person.
4. I promise to give an honest and fair view even if it makes me unpopular.
5. I will avoid the use of unnecessary technical language in proving that my model is right.
6. I will not be short-sighted and discard models which will be useful in other aspects of the business.

The ideas driving this Hippocratic Oath is described in greater detail below:

1. Independent Oversight – I will avoid validating models that I am a parent or emotionally attached too.

It is a fact that modellers easily fall in love with their models. This is also mentioned in The Financial Modellers’ Manifesto. The lack of independency from the model is one of the primary causes for model failures. In order for a validator to carefully evaluate the borders and caveats of a model, they should not be involved in its creation.

2. Model Governance – I will make sure I have a good enough understanding of the whole governance process before validating it.

There is no purpose of a functioning model that has no-one to be accountable for it! Good governance involves named individuals who have responsibility for the oversight and development of the model.

3. Outcomes Analysis – I will use easily repeatable checks in my validation so that can be understood by a lay person.

Simple checks are preferable to complex algorithms in the validation process. The processes in validation should be kept as simple as possible and well explained so future team members will be able to reproduce and understand the outcomes.

4. Conceptual Soundness – I promise to give an honest and fair view even if it makes me unpopular.

A good organisational culture will enable effective challenge between validators and model creators. The validator should not be intimated by the possibility of rejecting the conceptual soundness of an underlying model due to possibility of “falling out” with the model creators.

5. Terminology Considerations – I will avoid the use of unnecessary technical language in proving that my model is right.

In addition to an outcomes analysis, it is useful for the validator to produce a brief report verifying the validity of their checks. Unnecessary jargon and technical language which do not add value to the validation process should be avoided. Again, the readability of the governance documentation should also be addressed with the same criteria.

6. Expectation Management – I will not be short-sighted and discard models which will be useful in other aspects of the business.

The validator should not be ruthless with their empowered position of objectivity. It is expected that highly complex and material models use more labour hours in their creation and should have more time spent on their validation.

Accordingly, the validator should be critical and deep in the validation of a model ensuring they place limitations in context of the vested interest of stakeholders.

As political and financial uncertainty grows, the call on actuaries to validate models will likely increase. Validators are at the forefront of change in risk management with additional emphasis being placed on sound governance processes.

Similarly, in academia many fields see active research into better quantification of uncertainty. My research adds to this by looking at models and the risks that they pose. Better model management will reduce failures and systemic shocks. Having validators and risk actuaries adhere to the principles of this Hippocratic Oath will help achieve this. Models, like actuaries, should be understood and not underestimated.

[1] Model Risk – Daring to open up the black box

CPD: Actuaries Institute Members can claim two CPD points for every hour of reading articles on Actuaries Digital.