Model Validation and Bank Examiners’ Rising Expectations

Technology in the banking industry continues to evolve at a rapid pace. As a part of that evolution, banks are relying more heavily on various models for decision-making and internal control. This can include, but is not limited to, areas such as asset liability management, the Bank Secrecy Act and anti-money laundering compliance, fraud detection tools and credit loss calculations. It is critical that banks fully understand how their models are working and how results are generated. Questions to consider as a part of a model risk management include:

• How confident are we in the data used in the model, and what internal controls do we have in place to ensure data accuracy?
• Are we running appropriate scenarios that capture multiple possible outcomes?
• What are the key assumptions of the model, and how are they documented and reviewed?
• Are we using the appropriate model for the size and complexity of our bank, and have they received a third-party review?

Regulator expectations of controls surrounding models have heightened; in response,  banks should document policies and procedures as well as outline the internal control structure surrounding their models and how they validate the data. This will help safeguard that decisions based on model output have accurate source information.

One way to accomplish this is through model validation, which is essential in confirming the model’s behavior.

The Office of the Comptroller of the Currency’s model risk management handbook defines model validation as, “The set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. Effective validation helps ensure that models are sound. It also identifies potential limitations and assumptions and assesses their possible impact. As with other aspects of effective challenge, model validation should be performed by staff with appropriate incentives, competence, and influence.”

It also describes a sound validation process as having:

• Defined purpose and goals.
• Scope, validation approach, schedule, resources and types and extent of validation activities and tasks.
• Specific actions that must be taken to complete individual validation activities and tasks.
• Detailed and sufficient documentation to demonstrate that all validation procedures are appropriately completed.

Validations can be performed internally or externally. If using external parties, management teams should also be aware of the regulatory expectations of monitoring third parties.

The OCC model risk management handbook states, “Many third parties provide banks with reports of independent certifications or validations of the third-party model. Validation reports provided by a third-party model provider should identify model aspects that were reviewed, highlighting potential deficiencies over a range of financial and economic conditions (as applicable), and determining whether adjustments or other compensating controls are warranted. Effective validation reports include clear executive summaries, with a statement of model purpose and a synopsis of model validation results, including major limitations and key assumptions. Validation reports should not be taken at face value. Bank management should understand any of the limitations experienced by the validator in assessing the processes and codes used in the models.”

A final item to consider when evaluating models is back testing the results as an integral part of validating model behavior and accuracy. For example, it was very challenging to back test asset liability management models in the recent past when interest rates were stable. With the recent rise in interest rates, management could look at model outputs from before interest rates increased to see how accurately the model captured the effects on earnings and capital. If the model yielded inaccurate results, analyzing the inputs and calculation methodologies can help bankers identify the root cause of the model’s inaccuracy, so it can be adjusted to yield more accurate results.

The data a bank uses is at the core of any decision; its accuracy is critical to provide management teams confidence that they are making the best possible decisions for their bank. Implementing a strong internal control structure around models that includes validation is vital to supporting a bank’s success.