Catastrophe model evaluation needs a rethink. Where should the industry start?
A write-up of our recent InsTech webinar exploring how insurers can move beyond the search for the best catastrophe model towards more proportionate, continuous and decision-led approaches to model evaluation.
Holly Elizabeth
Head of Marketing
~The catastrophe modelling landscape has changed considerably over the past decade. Insurers now have access to more models, from more providers, covering a broader range of perils and increasingly complex risks.
More choice is positive; it allows insurers to develop more diverse and informed views of risk. However, this expansion is not only limited to increasing availability. Just this month at the ILS Convergence conference in Bermuda, Karen Clark announced that KC&C would be moving towards an annual model update cycle.
In an industry where evaluating and implementing a new model version can take several months, how can insurers adapt their evaluation processes to keep pace?
This was the central theme of a recent webinar hosted by InsTech, in which Maximum Information's Tom Philp and John Wardman joined David Singh of Beazley to explore more effective, proportionate and sustainable approaches to catastrophe model evaluation.
In this article, we explore the key debates featured in the panel and what this means for the industry. Read more below.
The growing availability of alternative catastrophe models raises an important question about catastrophe model selection and how organisations should use models when different providers offer different perspectives on the same peril.
During the webinar, David Singh highlighted the operational challenge this presents for insurers managing risk across global portfolios. Organisations need a stable modelling platform capable of supporting a consistent view of risk across different regions, classes of business and regulatory requirements.
However, this does not mean relying on a single model or scientific perspective. Instead, insurers should identify which catastrophe models are appropriate for the decisions being made.
"I'm hoping that we can shift as a market from this concept of best towards the concept of which models are
appropriate for the decision at hand." - Tom Philp, CEO, Maximum Information
Different models reflect different scientific assumptions, methodologies and treatments of uncertainty, each potentially contributing valuable information to an insurer's understanding of risk.
A catastrophe model used to inform reinsurance purchasing decisions across a large portfolio may require a different evaluation from one used for individual risk selection. Material uncertainties, required level of detail and consequences of model error depend on the intended application and use.
This is why assessing catastrophe model appropriateness should begin with the decision a model is intended to support, rather than its technical sophistication alone.
David also emphasised that insurers need to look beyond traditional vendor benchmarking and the evaluation of individual perils in isolation. Accumulations, dependencies, cross-class exposures and wider systemic and economic consequences must also be considered.
"Shifting from evaluating a model peril in isolation to include how well those models can support the firm's overall assessment
and understanding of risk and, ultimately, decision making is a key shift in my perspective."
David Singh, Beazley
Ultimately, model appropriateness cannot be established through technical performance alone. It depends on whether a model provides the evidence needed to support a particular decision.
Making catastrophe evaluation proportionate to the risk
If model selection should begin with the intended use, then the same principle should apply to how extensively a model is evaluated.
This is not an entirely new concept. The importance of materiality and proportionality is increasingly reflected in regulatory expectations, including the PRA's recently published supervisory statement SS5/25 on climate-related risks.
The underlying principle is that not all risks, or organisations, require the same level of assessment. The depth of evaluation should reflect the significance of the risk to the business, the nature and size of its portfolio, its risk appetite and the decisions that the model is intended to support.
However, applying proportionality requires judgement about which risks warrant detailed investigation and what evidence is sufficient. For teams with limited resources, establishing these requirements can become a substantial exercise.
This disparity was raised by John Wardman during the webinar, drawing on Maximum Information's experience working with insurers across the London market. While some organisations have established teams of scientists and engineers with sophisticated evaluation processes, others rely on small teams or individuals with limited time and access to information.
To help address this challenge, John has been exploring how the principles set out in SS5/25 could inform a more structured catastrophe model evaluation framework.
His work has focused on helping insurers answer three questions: Which peril-region combinations matter most to the business? How deeply should those models be evaluated? And are they appropriate for the time horizon of the decisions they support?
These principles provide a practical starting point for insurers developing an evaluation framework that reflects the materiality of their risks and the decisions they need to make.
A useful example is the distinction between non-peak perils and US hurricane risk.
For a peril with limited exposure and little potential to materially affect an insurer's capital, a high-level assessment of model outputs may be sufficient. For a major US hurricane portfolio, however, evaluation may require a much deeper examination of model components and the relationships between hurricane risk, storm surge, flooding and other downstream impacts.
As David Singh explained:
"There's little benefit to spending a disproportionate amount of time on models that aren't going to really change your capital
or it's you have minimal exposure there."
This approach also challenges the tendency to equate increasingly sophisticated evaluation methods with better evaluation.
During the discussion, Tom Philp observed that organisations often begin by asking whether they can improve their existing tests, obtain better data or undertake more detailed analyses.
This became particularly interesting when attendees were asked to identify their biggest challenge in evaluating models. The leading response was insufficient data or transparency to properly evaluate them.
50% of webinar respondents say insufficient data or transparency is their biggest challenge when evaluating models.
There is little doubt that limitations in data availability and model transparency can present genuine barriers to evaluation. But it is worth challenging whether these are always the most significant obstacles to reaching a defensible decision.
Do insurers always need more data, or do they first need greater clarity about what they are trying to establish?
For many evaluations, particularly where a peril has limited materiality, the information already available may be sufficient. More granular data and sophisticated tests may improve understanding without necessarily changing the decision.
Without a clear definition of sufficient evidence, the pursuit of additional data risks becoming an open-ended exercise.
This is not an argument for overlooking genuine model limitations. Rather, as Tom suggested, greater value may come from revisiting the foundations of the evaluation framework before making individual tests more sophisticated.
From periodic to continuous catastrophe model evaluation
As model updates become more frequent, the time and resources required to evaluate them are becoming increasingly difficult for insurers to manage.
During the webinar, David Singh highlighted how much time organisations spend understanding what has changed between model versions, often at the expense of evaluating what those changes actually mean for their business.
"Companies invest a disproportionate amount of time just understanding what's changed rather than actually focusing on [...] how
appropriate that change is to the actual business. Vendors need engaging" - David Singh, Beazley
This points to a wider problem with how model evaluation has traditionally been approached. Rather than building on previous work, teams can find themselves repeating analyses or reconstructing the reasoning behind earlier decisions, with each model update triggering another substantial evaluation exercise.
Yet the limitations of this approach extend beyond efficiency. Traditionally, evaluations have been completed, documented, approved and revisited only when a significant change occurs. Meanwhile, scientific understanding continues to develop, exposures change and new evidence emerges that may challenge an insurer's existing view of risk.
For Tom Philp, this was one of the most important shifts the industry needs to make.
"You had a model update, you're going to do an evaluation, you do the evaluation, get signed off, it's in a drawer.
And that's just not the world we're in anymore. It has to be a live thing that's continually evolving."
This is where technology has an important role to play. Automating repeatable analyses and maintaining an accessible record of previous evaluations can reduce the operational burden of model changes, allowing specialist teams to spend less time preparing analyses and more time interpreting their implications.
As John Wardman highlighted, combining more standardised evaluation frameworks with technology offers an opportunity to improve the quality of model evaluation without replacing the expertise and judgement of those undertaking it.
"When you pair some of the standardisation approach with tools like our automated model evaluation platform MagniPhi,
we can see that the greater contribution is going to be improving the conversation rather than replacing it."
- John Wardman, Maximum Information
This is the approach Maximum Information is developing through MagniPhi, which enables insurers to build on previous model evaluations, assess the implications of new model versions and maintain an evolving, evidence-based view of risk.
Ultimately, the objective is not simply to evaluate catastrophe models faster. It is to establish a process through which an insurer's view of risk can be continually tested, challenged and refined as the evidence evolves.
.png)
.png)
.png)
