5 Things that Make a Real Difference in Risk Modeling
Having recently completed another risk model this week, I was reflecting on some of the things that factor into building a model. And not just a good model, but a great model that can transform the way our customers do business.
I realized that 5 things were really important.
Understand the Data – I mean REALLY understand the Data.
The biggest mistake you can make in risk modeling is not understanding each and every field in a dataset and what drives the data into it. So many models are built in a vacuum where a scientist is not given access to the business. They review datasets without understanding how that data is used in the business operations. When this happens, assumptions are made about the data and sometimes those assumptions are wrong. When we build a model at PointPredictive, our scientists spend an inordinate amount of time with the client, understanding each and every data element. It’s a painstaking process but the end result is a model that works.
Focus on the REAL problem.
In risk modeling, it is quite easy to get side-tracked and end up building a model that focuses on the wrong problem. The advice we always give ourselves is – “follow the losses”. If you build a risk model that reduces risk and that in turn results in reducing losses then you are solving the real problem at hand. Focus on the REAL problem when building a model and you’ll be successful.
Iterate early and often, getting feedback from the business along the way.
Years ago, a customer conveyed to me a story of a failed project that ended up causing their business a lot of frustration. They waited months and months for model to be delivered to them. When the final model and results were delivered it didn’t address the business problem at all and as a result was never deployed. The fact is, great models are typically the product of lots and lots of small, incremental iterations. After each iteration, the business provides feedback and adjustments and improvements are made. Not only do the models improve from the frequent feedback over time but the business will end up understanding the models and that in turn will improve the odds that the model will gain acceptance.
Measure Performance of the model from every perspective that makes sense
I am not sure you can ever focus too much on model performance measurements. As a company, we spend a lot of time working with our customers to show them how our models perform. How much risk does the model detect? What is the false positive of the model at each score band? What is the number of cases created? How will the model reduce losses over time? These are all questions that we try to answer for customers and it makes a real difference.
Be there for Go-Live.
The first day a model is deployed is a critical one. It’s the day that everything comes together and the end users will start to use your model. It’s also when a lot could go wrong. Data feeds, systems, or network connections can cause problems that impact your model. Or, in some cases, the end users may have a problem understanding how to use the model or scores in their workflow. Being there for Go-Live to make sure everything works according to plan is a very critical phase in the model development process.
If you look at the things that make a real difference , in risk modeling, they are all about the same thing – bringing the business and science together. A great model is a collaboration between operations and analytics. The more you can bring the two together and collaborate throughout the entire modeling process, the better the end solution will be. That’s what we believe and that’s our focus.
For questions, please contact fmckenna@pointpredictive.com
