Contributions: Cinthia Tanaka, Edesio Alcobaça, Felipe Almeida, Caroline Custodio, Pedro Schoen, Otávio Vasques

Fraud prevention remains one of the most pressing challenges for financial institutions, as fraudsters constantly evolve their tactics. At Nubank, we leverage technology and data science to stay one step ahead.

In this post, you’ll get to know a little bit more about sequential modeling—an advanced machine learning technique that has transformed how we detect and prevent fraud.

Our team of experts, Carol, Pedro, and Otávio, from Nubank’s Fraud Data Science (DS) team, share their experiences with building and scaling sequential models. Learn how these models have helped us better understand fraud patterns and improve customer protection, based on the answers our experts gave to the questions prepared by Nubank Editorial. 

Q: What are sequential models? How do they differ from traditional machine learning approaches?

Pedro: The main difference between sequential models and traditional machine learning models is related to how we incorporate features. Instead of engineering features (e.g., aggregating a certain value within a specific time window, like 24 hours), we feed events (e.g., first a credit card purchase, then a loan) into the model. The time between events and the order in which they occur are critical signals for identifying scams and theft, making sequential models particularly effective for such use cases.

Check our job opportunies

Q: Why did you start investing in sequential models?

Carol: Sequential models offer a lot of advantages, especially when it comes to scalability and adaptability. For sequential models, we have a different challenge for building the real-time features. We don’t have a huge volume of features to create, it’s relatively easy to do the feature engineering. The problem is to receive a sorted array of events so that you can calculate the features for each event. However, once you have the structure ready, it’s easy to expand this because you have default keys that you receive and this doesn’t change when you add more events or data to the model – there’s no need to change the feature engineering, so that it’s scalable and easier to adapt to different geographies.

Pedro: Adding to what Carol mentioned, one key benefit of sequential models is that they allow us to uncover relationships between events that we might miss with traditional models. For example, knowing what happened first and what happened last, or how much time passed between those events, helps us identify fraud patterns that might otherwise go undetected.

Q: How did you implement sequential models?

Pedro: We used neural networks to implement them, in a structure that processes events with layers that also receive additional inputs related to traditional (tabular) features. Related to features, we can create any number of features per event such that each event has n pre-defined features. Every event has the same set of features and if we need something new, we have to include it into a map. We can also work on a structure that can incorporate traditional information alongside the sequential features. To design the training dataset, we captured events for a specific time window while limiting the number of events, to optimize the performance.

Carol: In production, we had to fetch data in a format that we are not used to. In general, for traditional models, we reach different service endpoints and the data serving-layer, but none of the data sources would give all the information we needed for our sequential model, so we couldn’t fetch data from one single place in this case. Our solution was to build a streaming application using Apache Flink to get all events for a certain customer given a time window.

Otávio: Without the streaming application, we would have to fetch these events individually, making calls to several different services. Our application listens to a Kafka topic which has all the Change Data Capture (CDC) related to events we are interested in. The tough part was that we had to get in contact with the teams that own the data for them to integrate them into the platform so that we could consume the events on this topic. The streaming application is responsible for doing the necessary joins so that we can build the SQL query using Apache Pinot.

Q: What were the challenges during implementation?

Pedro: One of the early challenges we faced was dealing with memory leaks after deploying the model. Fortunately, we were able to debug and resolve the issue, but it did take some time.

It was also difficult to validate if the online predictions were correct: if one particular feature in the event was wrong or if the order was not correct, the array wouldn’t match with the batch input, so we had to validate each feature inside each event. In terms of interpretability, we used SHapley Additive exPlanations (Shap), but we had to adapt them to work with the sequential data. For each event, we have a Shap value, but we can sum them to read them as the Shap value for usual features.

Carol, Otávio:  The main challenge of the project was to build the streaming application that we needed to efficiently consume the real-time data, since it was one of the first built at Nubank. Processing vast amounts of event data, ensuring compatibility between batch and online validation formats, and dealing with irregular response times were also significant hurdles we faced as we deployed it.

When thinking about new applications, one thing that tends to worry stakeholders is the cost behind it, but we haven’t seen a lot of change in that compared to traditional models. 

Q: How did sequential models impact fraud prevention?

Pedro: While it seems the sequential model’s structure is more complex, it’s simpler on the data side: billions of rows can be processed in a few hours. For the first proof-of-concept we did, the results were worse than our baseline, but since the structure was lightweight and easy to use, we started using it for other use cases where we got a better performance than the baseline. 

Q: Would you say the next step is to make sequential models global?

Pedro: We have seen that fraudster behavior is similar across countries and there’s no limitation on the model architecture’s side.

Carol: On the data consumption side, there is a need for adapting to the legal requirements (data privacy / data sharing) and the data sources of each country. It may happen that we need to integrate new data sources or to adapt certain events locally. For example, Pix (instant transfer method) only exists in Brazil.

Otávio: There might be domains in which this data standardization comes almost for free and these are the domains in which we already started working on a global version of a sequential model. For other domains, we might need to abstract certain events to make sure we have a country-agnostic structure.

Q: Finally, what lessons did you learn and would you share tips for other teams?

Carol, Otávio, Pedro: A few key lessons stand out. First, sequential models are much easier to repurpose for different use cases compared to traditional models. While the model benefits from GPU capabilities, keep in mind that the more data you have, the higher the resource demands.

Another important lesson was the streaming application — building one from scratch was a risky move, but it paid off. The key takeaway is that, when trying something new, it’s important to plan ahead and mitigate risks. We didn’t start by fetching all events using the streaming application. Instead, we assessed whether it was feasible for each data source and worked closely with other teams to ensure smooth integration.

Finally, to streamline the training of sequential models, we built a custom workflow using an internal tool (Common Python Workflows) with support from our infrastructure team. Our advice to other teams interested in exploring sequential models is to start with a clear use case and be ready for iterative development.

Conclusion

The implementation of sequential models at Nubank has significantly enhanced our ability to prevent fraud and protect our customers. By using advanced neural networks to process event-based data, we’ve built scalable and adaptable models that can respond to rapidly evolving fraud patterns. The lessons learned throughout this journey highlight the power of innovation and adaptability in the fight against fraud.

At Nubank, we’re committed to pushing the boundaries of Data Science and Machine Learning to safeguard our customers. Stay tuned for more insights from the Purple MinDS series, where we continue exploring the latest advancements in fraud prevention and beyond.

Check our job opportunies