Making AI initiatives work for Machine Learning Engineering

Written by: Felipe Fábio Frigeri and Camila Fragoso

Generative AI has brought a whole new set of capabilities to tech companies. At Nubank, where code turns ideas into products and better customer experiences, that opens up many possibilities for Machine Learning Engineering (MLE). Data preparation, dataset construction, experimentation, deployment, and monitoring all involve steps that AI could help improve, and these activities support models that inform business decisions and ultimately affect our customers.

With fast, readily accessible Large Language Models (LLMs), the temptation to automate almost every step is understandable. But being able to build a solution quickly raises another question: what will make it useful beyond the first successful experiment?

Without clear criteria, applications can start piling up without anyone knowing who will use them, how they fit into the workflow, or whether they still serve their original purpose. There is something almost poetic about it: the ease and power of creating sophisticated solutions can become the very reason we lose focus on the problems we wanted to solve.

In the PJ (Pessoa Jurídica, or business customers) Credit Card team, this led us to be more selective about which AI initiatives we pursue. Before committing to implementation, we wanted to explain which problem a project would address, how we would assess its usefulness, and whether the approach could help with other use cases. Following Nubank’s value of pursuing high standards and smart efficiency, we organized this discussion around three practical questions.

In this post, we share those questions and walk through a feature discovery project from the PJ Credit Card team. We explain the idea, how we answered each question, and how those answers helped shape the project’s scope and approach.

If you don’t know where you’re going, any AI will do

These questions give us a common starting point for discussing proposals before committing to building them. They also give us something to revisit as the projects develop.

1. What problem or workflow are we trying to improve?

Machine Learning Engineering work is a sequence of tasks that moves between structured processes and creative work. On the process side, regardless of the problem we are solving or the model we are building, some steps are always part of the job: understanding the business context, gathering data, constructing datasets, deploying models, and monitoring them.

Then there is the creative work: exploring a new technique from research, considering an algorithm another team has been using, or designing a sequence of experiments to improve the metrics we care about. These sides often overlap, but they give us two useful ways to approach the first question.

Which step of the modeling process could we make faster or more fluid?

This could mean simplifying experimentation, constructing datasets, validating business assumptions, preparing documentation, evaluating models, or deciding when to retrain them. In a modeling environment where much of what we do already has its own automation, it helps to be specific about where the additional improvement would come from. Which part still takes considerable effort? Where do we repeatedly spend time that could go into other work?

Could AI help us explore previously unexplored territory?

Another place to look for opportunities is the work we keep postponing: hypotheses, techniques, or data sources we would like to investigate but haven’t had the capacity to explore. This could involve new algorithms, feature engineering approaches, data sources, or data formats. It could also mean adapting existing models to a different context, for example, investigating where moving from batch predictions to online decisions would make sense. An idea that previously required more time or capacity than we had might now become feasible.

These two questions may seem to point in different directions, one toward time efficiency and the other toward creativity. But both come from the same belief: we should create more room for scientific thinking and experimentation. Making recurring tasks easier gives us more time to formulate hypotheses, explore different approaches, and test ideas. Expanding the scope of our work allows us to pursue questions we previously couldn’t take on. Both support a defining part of Machine Learning Engineering: turning mathematics and science into practical solutions that improve business decisions and our customers’ experience.

With that purpose clear, an AI-first project has one additional condition: AI needs to be central to what makes the proposed approach possible.

2. How can we measure success: time, scope, or both?

Once we have identified an opportunity, the next question is what would convince us that pursuing it made a difference.

Time is a natural starting point. In engineering, we often talk about time to market, and reducing the time needed to deliver a model is a useful outcome. The same reasoning applies to individual steps, such as constructing a dataset or evaluating an experiment. If that is the intended improvement, we should establish a baseline by looking at how long comparable work took before introducing the solution. Even an approximate measure gives us something concrete to revisit, provided the work still meets the quality standards we expect.

For initiatives that expand the scope of our work, we need to explain how we will assess the usefulness of the new capability. Being able to run more experiments gives us an observable measure, but we should also identify which questions those experiments help us answer and which decisions they could inform.

For example, we might systematically evaluate whether two customer populations would benefit from separate models, revisiting an assumption we previously had little capacity to challenge. We would need comparable evaluations and clear criteria for deciding whether a change is justified. Similarly, discovering more candidate features expands our possibilities, while evaluating their contribution tells us which ones are worth pursuing.

Where the opportunity is closer to a business initiative, we should connect the delivery to the outcome it could enable. For a new credit policy, that might mean identifying the previously underserved customers it could reach. The relationship will not always be direct, and approximations are acceptable, but we should be able to explain how the new capability contributes to the broader objectives of the team.

Some projects will do both: expand what we can take on and reduce the time needed to deliver it. When that happens, we should make both gains explicit. The measures may be approximate at first, but they should help us assess whether the initiative is making our work more effective and creating the opportunities that justified building it.

3. How reusable is our solution?

A well-defined problem and clear success criteria give an AI initiative a strong foundation. A third question worth asking from the start is how much of the solution could be useful in other contexts, teams, or problems.

At the beginning of this post, we discussed how the ease of building with AI can lead to solutions that are quickly forgotten. Another risk is building something so closely tied to its original context that adapting it becomes almost as much work as starting again. In modeling, we call it overfitting when a model learns the particularities of its training data so closely that it performs poorly on new data. There is a useful analogy here: an AI solution can become so dependent on one model, repository, or person’s workflow that its usefulness barely extends beyond that setting.

A practical way to approach this is to look at the recurring steps of Machine Learning Engineering. Preparing data, evaluating experiments, deploying models, and investigating monitoring alerts are familiar activities across many teams. The question becomes: is this something only I do, or could another ML Engineer be facing the same problem?

Even within our own team, that question can change the design. An automation built to evaluate one model might also support future versions or other models if we make its assumptions explicit and keep model-specific inputs separate from the shared evaluation logic. Other teams might be asking different business questions while still benefiting from parts of the same process.

Ease of adoption matters here too. Someone else needs to understand what the solution does, what it expects, and how to use it in their own work. Clear documentation and a simple interface help turn the possibility of reuse into something people can actually adopt.

A real AI-first project: automated feature discovery through network graphs

To show how these questions guide a project from its initial idea to implementation, we walk through an automated feature discovery initiative in the PJ Credit Card team. We discussed all three questions before we started coding, using our answers to define the scope, success criteria, and opportunities for reuse.

Finding new features beyond familiar paths

Our feature-discovery process had focused on reusing features already adopted by other models but never tested in PJ. We wanted to broaden that search to include discarded experiments and signals that had never been turned into features. Graph-based modeling addresses a specific part of this opportunity: learning interactions among existing features and exploring relationships in raw events. It complements, rather than replaces, the work of finding and reusing candidates from other models. Our goal is to give generative AI the autonomy to train predictive models and creatively propose new feature candidates based on insights from those experiments.

To investigate interactions among existing features, we chose the paper Automatic Feature Generation based on Graph Structure Learning (AFGSL) as a modeling reference. In phase 1, aggregated features would become graph nodes that exchange information, producing contextualized embeddings for a downstream risk model. These are learned representations, not automatically interpretable business features, and the connections are predictive rather than causal.

Phase 2 would explore transactions and events, comparing a non-graph embedding baseline with a temporal graph connecting companies and related entities. The graph model would produce relational embeddings; an optional interpretable readout would turn learned context into named features. These raw-event approaches extend beyond the AFGSL paper and aim to uncover signals absent from existing feature columns.

How we evaluate these outputs

These outputs call for different evaluations. For embeddings from either phase, the question is what they add to the downstream risk model, assessed through comparisons against appropriate baselines and checks on out-of-time performance, calibration, and stability. For features with defined meaning, an automated validation workflow can identify risk patterns the current model does not fully capture, providing an initial indication of additional signal rather than proof of a downstream gain. Evaluation would deepen as candidates progress, using only information available before each credit decision.

The longer-term goal is reuse beyond PJ. Once we demonstrate value in our models, shared discovery and evaluation components could support other risk teams, while model-specific inputs remain configurable. That keeps the initiative grounded in a concrete problem, evidence of usefulness, and a path to broader adoption.

What this work is really about

This is a glimpse of the work we do as Machine Learning Engineers at Nubank. At the heart of our daily work is pursuing new ideas, testing hypotheses, and turning research into better models and customer experiences. The questions in this post help us decide which opportunities to pursue and what would make them valuable for the business. If this sounds like an environment you’d enjoy and you’d like to contribute, explore our open roles at Nubank.