BloombergGPT's ledger: what training your own model cost, and what adapting one scored

Barakaeli Lawuo, Jul 2, 2026

In March 2023 Bloomberg published both the recipe and the bill for a language model it trained itself. BloombergGPT has 50 billion parameters. The team started from a compute budget of 1.3 million GPU hours on 40GB A100 GPUs, trained on 64 AWS p4d.24xlarge instances holding 512 GPUs in total, and ran for 139,200 steps, about 53 days.¹ The run processed 569 billion tokens, about $2.36 \times 10^{23}$ floating point operations, and ended at roughly 80% of one pass through the 709 billion tokens available, because loss on held-out data had stopped improving.¹

Half of what it read was Bloomberg's own. FinPile, a 363 billion token collection of filings, news, press releases, financial web pages and other documents that Bloomberg analysts had curated over forty years, made up 51.27% of training. Public datasets supplied the other 345 billion tokens.¹ On five public financial benchmarks the model beat three open models of similar or larger size (GPT-NeoX, OPT-66B and BLOOM-176B) on four and came second on the fifth. Its average was 62.51; theirs ranged from 51.90 to 54.35.¹ On Bloomberg's internal sentiment tasks the gap was wider: 62.47 against 29.23 to 35.76.¹

That is one side of the question this post is about. Should a team train its own model, or take an existing one and adapt it? The Bloomberg paper helps answer it because it wrote down the costs, not only the scores. The other side came from a group at Queen's University and J.P. Morgan AI Research, who ran general-purpose models from OpenAI on the same financial benchmarks and set the results next to BloombergGPT's reported numbers.² Read together, the two papers work as a case study with a ledger attached.

Pretraining

Training a model from random weights on a very large pile of text so it learns to predict the next token. BloombergGPT is a pretrained model built from scratch.

Fine-tuning

Continuing to train an existing model on labeled examples of one task, so some or all of its weights change.

Zero-shot and few-shot prompting

Asking a model to do a task with no worked examples in the prompt (zero-shot) or with a handful of them (few-shot, e.g. 5-shot). No weights change.

Chain-of-thought (CoT)

A prompt that asks the model to write out its reasoning steps before the final answer.

F1 score

A score that balances precision (how many predicted labels are right) and recall (how many true labels were found). Weighted F1 averages the per-class scores by how many test examples each class has.

GPU hour

One GPU busy for one hour. It is the usual unit for a training budget, because cloud providers bill by it.

What 53 days of training bought

The authors sized the model with the Chinchilla scaling equations, which for their budget pointed to about 50 billion parameters and more than a trillion training tokens. Their roughly 700 billion tokens were too few for that. They could have added more public text, but they did not want FinPile to fall below half of training, so they chose the largest model that could see all their tokens while keeping about 30% of the compute budget in reserve for failures, retries and restarts.¹ That reserve is worth pausing on. A team that builds plans for its own run to go wrong.

And it did go wrong. The appendix the authors call Training Chronicles records a first run in which the weights of one LayerNorm layer changed trend for no reason the team could explain. After four failed attempts to fix it, they started over with more conservative settings, among them a lower peak learning rate (6e-5 instead of 1e-4) and tighter gradient clipping (0.3 instead of 1.0).¹ The second run went smoothly for about 42 days, needing only a restart after 28 days because the platform had a hard limit on job length. Around day 48 the validation loss stopped improving for a week. Several interventions did not fix that, and with 77% of the training data used and the training budget nearly spent, they stopped.¹ (The main text says about 80%; the chronicle says 77%.)

The payoff was real against models of its own class. On ConvFinQA, a test of multi-turn numerical questions over earnings reports, BloombergGPT scored 43.41 against 30.06, 27.88 and 36.31 for the three open models, and the authors call that gap especially large.¹ They credit the results, in decreasing order of impact, to the curated internal dataset, their own tokenizer, and an up-to-date architecture.¹ One detail sets up everything that follows: every model in those financial tables was run by the Bloomberg team with the same plain few-shot prompts, with no chain-of-thought, and all of them were open models.¹ The strongest commercial models of the day were not in the comparison.

The same benchmarks, run on GPT-4

Li and colleagues tested gpt-3.5-turbo (the ChatGPT model) and GPT-4, in versions current as of July 7, 2023, on eight datasets covering five kinds of task: sentiment analysis, headline classification, named entity recognition (NER), relation extraction and question answering.² They did not fine-tune ChatGPT or GPT-4. They used zero-shot, few-shot and chain-of-thought prompts only, took BloombergGPT's scores from its paper, and matched its setups where they could, including the number of examples in each prompt.²

GPT-4 came out ahead on four of the five tasks that both papers report. On FiQA, an aspect-level sentiment task on financial news and microblog posts, BloombergGPT's 5-shot weighted F1 was 75.07; ChatGPT with no examples scored 75.90, and GPT-4 with five scored 88.11.² On gold-price news headlines, GPT-4 5-shot reached 86.00 against BloombergGPT's 82.20, while ChatGPT 5-shot trailed at 74.84.² On ConvFinQA the gap was large: 59.86 for ChatGPT and 76.48 for GPT-4, both zero-shot, against 43.41.² NER was the exception. There BloombergGPT's 20-shot entity F1 of 60.82 beat GPT-4's 56.71 at the same shot count.²

The authors found it notable that both OpenAI models beat a model trained specifically on financial text, and offered a hedged explanation: "This might be due to the larger model size of the two models."² They also observed that GPT-4 was significantly better than ChatGPT on nearly all the financial benchmarks, and concluded that advances in general models carry over into finance.²

The fourth bar in each group is the one a build-or-adapt decision should stare at. It is a much smaller model fine-tuned on one task's labeled data, and it wins two of the five. A fine-tuned BERT scored 95.36 on headlines, 9 points above GPT-4.² On NER, a conditional random field (CRF, a classic sequence-labeling model) trained on FIN5, financial agreements similar to the test set, scored 82.70, higher than every language model. The same CRF trained on general CoNLL news data scored 17.20.² On REFinD relation extraction, which BloombergGPT did not report, a fine-tuned Luke-base reached 56.30 macro F1 against GPT-4's 46.87 with ten examples.² Li et al.'s own advice follows from this: use the general models for relatively simple financial tasks, and treat pretraining plus fine-tuning as the leading option for structured prediction such as NER and relation extraction.²

Question answering pointed the other way. On FinQA, where the model must compute an answer from a table and text in an earnings report, GPT-4 with chain-of-thought reached 78.03, above the fine-tuned FinQANet.² The FinQA paper puts expert humans at 91.16 and non-expert crowd workers at 50.68 on the same test.⁵ Li et al. also logged a ChatGPT error of the kind that matters in finance: every intermediate result correct, then $753 million + $785 million + $1,134 million summed to $3,672 million instead of $2,672 million.²

The ledger

Each path puts different lines on a team's books. Where a paper reports a figure, the line below gives it with a citation. Where no paper does, the line is my reading and says so.

Compute. Building meant 1.3 million GPU hours of budget, 512 GPUs for 53 days, and 30% of it held back for things going wrong.¹ Bloomberg gives no dollar figure. The FinGPT authors priced those GPU hours at an AWS rate of $2.3 per hour and estimated around $3 million per training run; that is their estimate, not Bloomberg's.³ For adapting an open model with lightweight fine-tuning, the same authors estimate around $300 per run.³ LoRA (low-rank adaptation) shows where such savings come from. It freezes the pretrained weights and trains small added matrices instead. On GPT-3 175B it cut training memory from 1.2TB to 350GB, shrank the saved checkpoint from 350GB to 35MB, and trained 25% faster than full fine-tuning.⁴ Prompting spends no training compute at all. In Li et al.'s runs, chain-of-thought moved FinQA accuracy from 48.56 to 63.87 for ChatGPT and from 68.79 to 78.03 for GPT-4.²

Data. A builder needs a corpus. Some FinPile documents, such as filings, are public but were nontrivial to collect and preprocess for training; some Bloomberg news had to be purchased; the rest is private. All of it was stripped of markup and templates, and each dataset was deduplicated.¹ An adapter needs labeled examples instead. FinQA's 8,281 question and answer pairs were written by eleven finance professionals hired for the job.⁵ ConvFinQA added 3,892 conversations with 14,115 questions.⁶ My reading: adapting does not remove the data cost. It changes its shape, from hundreds of billions of tokens to thousands of expert labels.

Evaluation. This line appears on both sides. Bloomberg found no standard benchmark for financial language tasks; the public FLUE suite had limited coverage, no suggested few-shot method, and some low-quality annotations, so the team built internal benchmarks as well.¹ Each internal sentiment dataset was labeled by two annotators with a third breaking ties, drawn from a dedicated team of financial experts, consultants, or both.¹ My reading: a team that adapts rather than builds still needs this test set, because it is the only way to see which bar in the chart above belongs to its own task.

Refresh. FinPile documents run from March 2007 to July 2022.¹ Anything later is outside the model's weights unless someone trains again. The FinGPT authors argue that financial data is time-sensitive and that retraining large models often is expensive and impractical, which is why they favor lightweight adaptation.³ LoRA makes a shared base model cheap to specialize many times: storing 100 adapted GPT-3 models takes about 354GB instead of about 35TB.⁴ It has a cost it states plainly: the full 350GB base model is still needed at deployment.⁴ My reading of Li et al.'s ChatGPT to GPT-4 gap: a team that adapts collects a better base model by switching and rerunning its test set, while a team that built pays for another run.

Control. Bloomberg did not release the model's weights. Its reason was that text can be extracted from model weights, so giving out the weights could amount to giving out FinPile.¹ My reading: this line favors building when the training data is the business, and it is the one line a price estimate does not capture.

Where the comparison stops holding

Every BloombergGPT number in the chart crosses from one paper into another. Li et al. did not run BloombergGPT; their tables say its results come from the original paper.² Their appendix lists the ways the two setups differ. The headline dataset they could obtain had six tags, while Bloomberg's test used a version with nine categories; the dataset's authors told them additional filtering produced the six-label version.² For FPB and FiQA, Bloomberg created its own train and test splits.¹ Li et al. held out 20% of the data for testing and describe their FiQA test split as their own.² On ConvFinQA, Bloomberg fed each model the gold conversation, while Li et al. appended the model's own earlier answers before each new question.^1,2

Checking the source papers, I found one more. Li et al.'s Table 8 lists FinQANet at 68.90 on FinQA and 61.24 on ConvFinQA. The FinQA paper's own Table 2 reports 61.24 on FinQA,⁵ and the ConvFinQA paper's Table 3 reports 68.90 on ConvFinQA.⁶ The two values appear to have swapped places. It does not change which model wins on ConvFinQA, but it is a reminder that a borrowed number should be traced back to where it was measured. The same table also disagrees with the paper's own prose: the text says chain-of-thought added about 10 points for ChatGPT and 15 for GPT-4, while the table shows gains of about 15 and 9.²

The Bloomberg authors drew the boundary themselves. Results from custom, non-LLM models on these public tasks existed, and they left them out: "we omit reporting them here due to differences in the evaluation setup. As a result, our claims are restricted to comparisons of LLMs."¹ They also note that unlabeled versions of their internal test data probably appear in FinPile, and that because some of FinPile is on the public web, the models they compared against may have seen it too.¹ The comparison that matters most to a team deciding between building and adapting, a fresh model against a fine-tuned small one on the same split and prompt, is exactly the comparison the paper says it did not make.