How we built a benchmarking framework to horizontally accelerate transaction model research

Authors: Arissa Yoshida, Daniel Braithwaite, Marcelo Buga, Taylor Foust

At Nubank, we research transformer-based transaction representation models that can be adapted to a growing set of downstream applications. Over the past few years, this work evolved from supporting only a handful of internal tasks to powering a much broader suite with more than 20 different benchmarks. This growth introduced a new challenge: how could we ensure that changes in data, architecture, and training consistently improved models overall, instead of only boosting performance on isolated tasks?

To answer that question, we built a benchmarking framework focused on horizontal model and data research. Our goal was to transform an operationally heavy manual experimentation process into an automated, reproducible, and statistically rigorous workflow.This framework allows us to test changes horizontally across multiple downstream tasks and multiple trials simultaneously. Instead of evaluating ideas benchmark by benchmark, we can now identify which improvements truly generalize across applications and which only work in specific scenarios..The result was a dramatic reduction in the operational overhead of experimentation. With the framework, our team became capable of running roughly five times more experiments per month, spending more time on research itself and less time operating pipelines.

The starting point: pursuing horizontal improvements

Our team established a clear objective to generate meaningful average improvements on a benchmark that represents multiple internal tasks. This meant moving away from overly specialized optimizations targeting a single application. Every new research direction (whether an additional data source, an architectural change, or a new training formulation) needed to demonstrate gains across multiple tasks at the same time.

We organized this work into two major fronts:

Data scaling: Incorporating new transaction data sources;
Architectural changes: Ranging from entirely new model paradigms to smaller adjustments such as activation functions, batch normalization, decoupled learning rates, increased context length, and alternative blending architectures.

However, every new hypothesis required dozens or even hundreds of training and evaluation runs distributed across multiple benchmarks. With an ambitious research scope and a limited compute budget, it became clear that the bottleneck was no longer only modeling, but rather, the experimentation process itself..

Check our job opportunies

The problem: manual experimentation doesn’t scale

Before the benchmarking framework, running a single experiment was a manual process that could take several days. Researchers needed to:

Configure and run data preparation;
Take the output, configure pre-training, submit the job, monitor it, track outputs;
Repeat the process for fine-tuning;
Run inference on test dataset;
Manually consolidate results for baseline comparisons.

In practice, most of the time was consumed by operational tasks: configuring jobs, copying parameters between stages, tracking outputs, and manually dealing with failures. Even repeating an experiment required almost the same effort as running it for the first time.

Tracking itself became another source of friction. Experiments were documented in massive spreadsheets with thousands of cells, where small mistakes could compromise large portions of the results.Under time pressure, researchers frequently monitored jobs and manually intervened when failures occurred. On top of that, the risk of subtle differences in configurations or code changes between executions introduced the potential for confounding factors, requiring careful experiment organisation.

The solution: a benchmarking framework

The aforementioned manual work required for experimentation also scales poorly as the number of downstream applications increase. For example, each experiment frequently requires multiple trials to establish statistical confidence and must be evaluated across the entire downstream benchmark suite, not just on a single task. Without automation, that meant multiplying every manual step by the number of trials and the number of benchmarks.Therefore, we designed a benchmarking framework to automate the experimentation process. Specifically, we defined four primary goals:

Reduce the operational cost of repeated trials to near zero;
Minimize the effort required to orchestrate new experiments;
Reduce tracking errors;
Allow researchers to focus on ideas instead of process.

And the framework was built on two main pillars:

1. The ability to represent the entire modeling pipeline in code, from data preparation to training, inference, and evaluation (as a branch of the codebase);

2. Branch deployments, which create isolated experimentation environments.

How the framework works

The workflow changed dramatically. Today, the researcher’s process is essentially:

Formulate a hypothesis;
Implement a code change;
Create a deployment from the git branch;
Run the pipeline.

Everything else happens automatically. The system:

Collects all required data;
Generates train, validation, and test datasets;
Executes pre-training;
Runs multiple fine-tuning trials;
Executes inference;
Evaluates metrics;
Automatically logs results and parameters into a centralized leaderboard.

Once a change is implemented, jobs run end-to-end, and every stage starts automatically as soon as its dependencies are completed. Additionally, because every experiment is tied to a specific git tag, reproducing or extending previous experiments became significantly simpler: researchers can recover the tag, create a new branch, and iterate from there.

Statistical rigor: detecting promising improvements

One of the most important design decisions behind the framework was integrating statistical significance screening directly into the system. In machine learning research, one of the most common mistakes is trusting occasional results that appear promising purely because of statistical luck, then building entire research directions on top of them. The screen exists to tell us which directions are worth that investment, not to certify any single result as a finished finding.

Before implementing the framework, we conducted a dedicated study to measure joint fusion variance [1] across repeated runs. Using 10 identical baseline runs, we observed that test AUC variance was relatively low, with a standard deviation close to 0.02.

From there, we conducted simulations to define a reliable statistical protocol. We found that using:

N = 5 baseline runs;
M = 2 challenger runs;
Welch’s t-test;

allowed us to reliably detect improvements close to 0.08 pp in AUC with 95% statistical power and a significance level of 0.05. Smaller gains near 0.04 pp are caught only about half the time, which is enough to mark a direction as promising and worth more runs, but not enough to treat a single comparison as conclusive.

Importantly, the 5-and-2 protocol is a minimum. A borderline result near 0.04 pp can be confirmed with more runs. Most of the standard error comes from the two-run challenger arm, so those are the runs to add: going from 2 to 5 moves the reliably detectable effect from about 0.08 pp to about 0.055 pp, and 10 challenger runs reach about 0.045 pp. The reporting tool re-runs the test at the new count automatically. Most experiments stay at the cheap screen, and only the directions worth confirming get the larger counts..

This methodology was built directly into the framework. Today, the reporting tool automatically performs statistical tests across the multiple trials of each experiment, delivering statistically validated results by default. A typical interface displaying these significance test results across a downstream task suite is shown below.

Benchmark setup

Benchmark evaluation follows a structured process based on:

Standardized baselines with five runs;
Shared multitask benchmarks;
Consistent evaluation across multiple trials.

The framework was also designed to grow continuously. Adding new benchmark tasks requires nothing more than opening a pull request.

[MX ML Meetup] Foundation Models @ Nubank - Frame 6.jpg

Results: evaluating experiment variants

With the benchmarking framework, we began systematically exploring both architectural and data improvements relative to our original baseline: a token-level decoder-only transformer that uses LoRA fine-tuning and a DCNv2 blending module.

On the architectural side, we evaluated changes on challengers such as:

Extended context length;
Training optimizations;
New architectures built from scratch.

On the data side, we tested four new transaction data sources, both individually and in combination.

The results showed that some changes improved only specific benchmarks, while others delivered little gain at smaller scales but became highly relevant in larger experiments. Without a structured framework, comparing all of these directions simultaneouslyAs part of the initial nuFormer research phase, we combined the most promising architectural changes with additional data sources and produced a significantly improved baseline with statistically validated gains across multiple internal benchmarks.

Impact

The framework delivered measurable improvements across several dimensions:

Roughly 5x faster experimentation velocity;
Elimination of spreadsheet-based manual tracking;
End-to-end pipeline automation;
Reduced need for manual overnight job monitoring;
Simpler onboarding for new researchers;
Full experiment reproducibility through git tags.

In practice, work that previously required roughly a month of manual experimentation can now be completed in less than a week, with fewer errors and greater statistical confidence.

What’s next

We continue expanding the framework with new capabilities, including AI-generated experiment summaries, cleaner development interfaces and better data sharing across experiments. As the number of downstream applications grows, the framework continues serving as a consistent horizontal validation layer across tasks, helping transform research advances into production impact at scale.

More broadly, this work reflects a core Nubank principle: fighting complexity not only in financial products, but also in the infrastructure that powers them. The framework creates an environment designed not only for humans, but also for automated agents. Once the experimentation cycle becomes fully closed-loop, agents can continuously help test, execute, monitor, and evaluate research hypotheses, enabling continuous 24/7 iteration and self-improvement.

We are already seeing promising early results from this setup, especially in areas such as:

GPU optimization, where Model FLOPs Utilization (MFU) increased from 10–15% to 30-40%;
Linear attention, where we matched AUC performance while reducing training costs and enabling longer-sequence training.

Our goal remains reducing the operational cost of experimentation even further, allowing researchers to focus on the highest-leverage part of the work: identifying the right questions, interpreting results, and deciding what to explore next. If successful, this could represent another step-change in experimentation velocity and model evolution.