Elevate your enterprise data technology and strategy at Transform 2021.
Machine learning has become pervasive in businesses across industries as the technology has matured in recent years. A 2020 Deloitte study found that 67% of organizations surveyed have put machine learning to work, and 97% expected to deploy some form of it in the year ahead. With this expanding use, new considerations are emerging, namely the significant investment of resources needed for maintenance of models.
Individual models may number in the hundreds for even a mid-sized company, such as a bank. Each model requires staff attention and computing power every time it needs to be run or updated. Plus, as output volume from separate machine learning models increases, interpretation and decision making become even more complex. Our team at Credit Sesame was experiencing all of this as we added products and continued to grow our business. To make sure that our machine learning work could continue to power the organization forward, we took a step back from our routines and decided to look for scalable options.
The unified modeling we developed is an approach in which a single model, rather than a set of related but separate models, is created to power a process or product. A unified model, not to be confused with unified modeling language, is facilitated by pooling the needed data together into a single array that is passed into the model, allowing all results to be delivered in one run rather than by calling a series of models in sequence.
To develop our approach, we first selected a set of models that were likely candidates for unification. We realized there was a very high level of overlap between the top features. Next, we developed a plan for unifying the models by working with the features being passed into the combined model. Then we ran a proof of concept to test the accuracy of the unified model compared to the individual versions. We were pleased to see equal or improved accuracy from the unified model.
Our experience with this approach has shown that it’s possible to glean a number of benefits by shifting to unified machine learning models. We have seen quantifiable improvements, such as a 60% reduction in people hours needed to maintain and run models that power one of the company’s key offerings. Our gains have opened up team time, improved process efficiency, significantly reduced maintenance costs, and more.
However, applying a unified modeling approach does present a number of challenges. Unified modeling is not a one-solution-fits all, so it is essential to understand the appropriate use cases.
When a unified model makes sense
Model unification can be useful for many types of machine learning problems. Our experience with predictive models, which are widely used by organizations across industries, has shown three important conditions that should be met for taking a unified modeling approach:
- A prediction is needed for the same target variable across a large number of related entities, or partitions
- Each partition uses the same set of features
- The models need to be refreshed on a frequent basis
These conditions typically are present when you’re looking to predict the values of target variables for closely related entities for purposes of comparison, or ranking and selection — for example, if you need to predict, among many lenders, the one with the highest probability of approving a particular loan. Unified models produce custom predictions for each partition (e.g., in our example, it would simultaneously predict the approval probability for each loan product).
Potential benefits of a unified model approach
When the situation is right for unified modeling, there are a number of benefits you can achieve. We tested how a shift to this method could improve six key metrics. As shown in the table below, there would likely be a 23% overall improvement across the metrics combined, with four showing gains from unified modeling and one staying the same.
A more in-depth look at the benefits in our real world experience showed that for the important area of process efficiency, the impact was significant. As our team shifted from deploying dozens of models that each powered a particular offering to using just one unified model, we experienced a 75% reduction in total steps performed. The change allowed a 60% reduction in people hours, which created substantial cost savings and opened bandwidth for the team to pursue other projects.
Additionally, a unified model helps reduce maintenance cost dramatically. Rather than working with dozens of separate models each time a business need occurs, a data science team is much more easily able to maintain one integrated model by updating it more frequently on a proactive, regular cadence.
The speed in which results are delivered is also critical to any business, especially when outcomes are needed in real time. By unifying into one model, you improve latency since all predictions are delivered at one time. We observed improvements to latency of approximately 66% in our work. Moreover, these improvements became more pronounced as the number of partitions in the data set grew.
Accuracy is always an important consideration. In our shift to a unified model, we saw accuracy increase by as much as 4% across the partitions being used. In our experience, pooling together data across partitions and fitting a predictive model on this combined data does not deteriorate the quality of outcomes.
Proceed with caution
The benefits of unified modeling can be significant for an organization. Yet there are a number of considerations to keep top of mind when implementing this approach, including data imbalances, rollbacks, and cold start needs.
Data imbalances
When developing a classification model, it is common to encounter class imbalance in the target variable. For unified models, the data is pooled from several partitions, and there can be a second layer of imbalance because certain partitions may be overrepresented. A team can correct this by upsampling the data for underrepresented partitions to promote fairness.
Rollbacks
With unified models, teams lose some flexibility for addressing problems since it is not possible to pick and choose individual partitions to roll back (or roll forward). A team can address this issue by retraining the unified model outside of the regular refresh cycle. Alternatively, if necessary, the model can be reverted for all partitions at once, across the board. For example, if you’ve created a unified model to predict demand for the full range or a set of your company’s products, you may find, after deploying the model, issues with the results for one product. You will then need to either roll back or retrain the full model.
Cold start needs
Sometimes there may be a gap in historical data when a new partition is introduced or an old one is reactivated. While there is no straightforward solution for handling this situation, one option is to create proxies from existing partitions that can be used until enough data is collected for the new one. Organizations are likely to encounter this situation when introducing new products to their inventories.
Ongoing evolution
Unified modeling can bring significant benefits to an organization when the right criteria are met and implementation teams have strategies ready to address challenges that may emerge. As uses for machine learning continue to spread even further throughout organizations and grow in complexity, the discipline must continue to mature. Techniques like the unified modeling approach I’ve described here are a critical part of the ongoing evolution that will help meet increasing demand from organizations for machine learning solutions to solve critical business challenges and help create competitive advantages.
Pejman Makhfi is Chief Technical Officer of Credit Sesame.
