Experimenting with Language Models

During September 2026, our team built two distinct language models from the ground up: a statistical n-gram model for Ewe, a low-resource language spoken in Ghana and Togo, and a domain-adapted English model focused on healthcare and agriculture. Our headline result was that our Ewe model achieved 1.45 bits per character on held-out test data, matching the industry-standard KenLM benchmark within 0.2%.

This project was Prosit 1 for our Natural Language Processing course (ICS 554) in the Master's in Intelligent Computing Systems program at Ashesi University. In this post, I will walk through the project brief, what we built, the core architecture choices, my personal contributions, and where the work goes next.

The full implementation, benchmark results, and reproducible notebook are available on GitHub at ASU-MICS-2028/nlp-group-1-prosit-1.

The brief

The assignment placed us as research interns at Ankora, a Ghanaian AI lab developing speech recognition systems. Automatic speech recognition (ASR) engines rely on language models to evaluate and choose the most plausible word sequences from noisy audio. Ankora asked us to build two systems:

• A low-resource language model: Because data for local African languages is limited, the brief proposed a traditional n-gram statistical model instead of a deep neural network.
• A specialized English model: Tuned specifically to a key domain, such as healthcare or agriculture.

Our deliverables included a shared GitHub repository, a detailed technical report, and a 10-minute team presentation. The written report covered three parts: Section A on language modeling theory, Section B on the low-resource Ewe model, and Section C on domain adaptation. A viva test accounted for 35% of the overall grade: after submitting our code, we each answered written questions about it.

One prompt in the assignment stood out: Do you agree that n-gram models are inherently better than neural models for low-resource languages? Rather than arguing the point abstractly, we chose to run empirical scaling experiments and map the crossover point directly.

What we built

We selected Ewe (Èʋegbe) as our low-resource language and structured the project into six clear, reproducible stages inside a central notebook, with standalone scripts persisting all intermediate metrics:

1. Data audit and filtering: We started with 4.4 million web-scraped English-Ewe sentence pairs. A massive 77% of rows were duplicates, and 70% of the unique "Ewe" lines contained no Ewe letters at all, consisting mostly of English, spam, or neighboring languages. After cleaning, deduplication, and linguistic filtering, 295,198 clean Ewe sentences remained, which we partitioned into train, validation, and test splits.
2. Hand-calculated proof of concept: We manually calculated probabilities on a toy five-sentence corpus to verify the underlying mathematics before writing code.
3. From-scratch n-gram engine: We implemented vocabulary building, maximum-likelihood estimation, perplexity calculation, and text generation. Without smoothing, 81% of validation sentences yielded zero probability, producing infinite perplexity.
4. Smoothing and tokenization: We implemented six smoothing strategies (add-k, Good-Turing, Jelinek-Mercer interpolation, Katz backoff, Kneser-Ney, and modified Kneser-Ney) across orders 2 through 12. We also evaluated twelve tokenization setups, ranging from word-level vocabularies to byte-pair encoding (BPE) and character models.
5. Neural baseline comparisons: To rigorously test the n-gram versus neural debate, we trained LSTM and small transformer architectures from scratch across five training dataset sizes. We repeated the exact same pipeline on parallel English text and WikiText-103 to establish high-resource reference points.
6. Domain adaptation: We adapted Qwen2.5-0.5B and SmolLM2-135M to medical literature (PubMedQA abstracts) and agricultural advisory data (a farming Q&A set). We evaluated LoRA across three ranks, DoRA, and full fine-tuning, while monitoring catastrophic forgetting on general English.

Cleaning a 4.4-million-row web corpus left 295,198 Ewe sentences, 6.7% of the download.

As the n-gram order grows, more dev words get zero probability and generated text copies training sentences verbatim: 96% at order 9.

We later expanded our dataset with three additional Ewe sources: Bible and religious publications, dictionary example sentences, and transcribed spoken audio from the Waxal corpus. We also produced standardized data documentation covering provenance, licensing, privacy, and known biases.

Our team, Group 1, consisted of Michael Boadi, Alice Kara-Acquah, Bernard Marfo, Eric Sunu, and myself. Together, we co-authored Sections B and C of the report, while each member submitted an individual analysis for Section A. We condensed our findings into a 14-slide presentation centered on two questions: does statistical counting beat deep learning when data is scarce, and can you adapt small models to specialized domains on a laptop without degrading general language ability?

Decisions that shaped the work

Our best results came from deliberate engineering choices, every one of which we validated experimentally:

Build the n-gram engine from scratch, then audit against reference libraries: Relying solely on third-party libraries would have bypassed the learning objective of understanding smoothing algorithms. However, writing custom implementations introduces subtle numerical bugs. To ensure accuracy, we cross-checked our code against standard libraries: our Kneser-Ney matched NLTK within 0.06%, and our modified Kneser-Ney matched KenLM within 0.2% at order 5 (128.04 versus 128.25 perplexity).

Filter sentences by distinctive Ewe orthography: The Ewe alphabet uses unique characters like ɖ, ƒ, ɣ, ʋ, and ɔ that do not occur in English. Requiring at least one of these characters filtered out 70% of the noisy web crawl. We audited this heuristic using GlotLID, a language identification model, which confirmed that 92.4% of the retained sentences were genuine Ewe (with the remainder belonging to closely related languages like Ga, Aja, and Nzema that share those letters). Disabling this single filter degraded model perplexity by 25%.

Split splits by near-duplicate hashes rather than random sampling: The raw corpus repeated biblical verses with minor punctuation differences. A naive random split would place identical verses in both training and test sets, artificially deflating perplexity to 93.1 instead of an honest 100.8. We hashed normalized text strings (lowercased, with digits and punctuation stripped) to ensure near-duplicate sentences never crossed between splits.

Leaving out each cleaning step in turn: skipping deduplication costs the most, and a random split looks better only because it leaks.

Evaluate tokenizers using bits per character rather than raw perplexity: Raw perplexity naturally favors small vocabularies: character-level models score an artificially low perplexity of 6.9, whereas word models score 123.8, even though the word model produces vastly better language representations. Bits per character (BPC) normalizes across vocabulary sizes, creating a fair playing field. Under this metric, word-level tokenization outperformed subword BPE for Ewe n-grams.

Map the n-gram versus neural crossover using a scaling curve: Instead of a single static benchmark, we evaluated all three architectures across 1,000, 3,000, 10,000, 30,000, and 100,000 training sentences to discover where neural models overtake n-grams.

Adapt small pretrained models using LoRA: Training an English foundation model from scratch was computationally impractical, while full fine-tuning modified 48% of Qwen's parameters. Using LoRA at rank 8 adjusted only 0.6% of weights while coming within 4% of full fine-tuning performance. We ran all fine-tuning on Apple silicon using the MLX framework, conducted learning-rate sweeps, verified stability across three random seeds, and measured retention using WikiText-2. When we discovered prompt leakage in the agriculture dataset, we re-scored the model on 104 verified leak-free test prompts, confirming our original conclusions (6.66 versus 6.69 perplexity).

What I did

I developed the core experimental pipeline and integrated our team's parallel modules into a unified codebase:

• Section B pipeline: Wrote the data auditing and splitting scripts, the from-scratch n-gram counting engine, all six smoothing algorithms, the BPE tokenizers, and the validation cross-checks against NLTK and KenLM (which I compiled from source).
• Neural benchmarking: Trained the LSTM and transformer baselines across five data scales, conducting width and tokenizer sweeps across Ewe, parallel English, and WikiText-103.
• Section C domain adaptation: Implemented the parameter-efficient fine-tuning workflow for Qwen2.5-0.5B and SmolLM2-135M, running LoRA (ranks 2, 8, and 32), DoRA, full fine-tuning, learning-rate sweeps, seed replications, and qualitative inference evaluations.
• Sensitivity analyses: Authored validation experiments testing train/dev/test split ratios from 1/49/50 to 98/1/1, a leave-one-out data cleaning ablation study, GlotLID language verification, and the leak-free agricultural evaluation.
• Team integration: Folded in Eric Sunu's parallel contributions (additional corpus ingestion, an Ewe morphological stemmer, distilgpt2 LoRA adaptations, decoding latency benchmarks, and dataset documentation), ensuring all experiments complied with our evaluation protocol.
• Packaging and presentation: Structured main.ipynb to load cached experimental outputs instantly without requiring hours of re-training, built our 14-slide presentation deck using numbers directly sourced from result files, created a speaker study guide with anticipated panel questions, and managed artifact distribution via Google Drive.

Every single figure and metric in our report was generated directly from a logged experimental run, and every data-cleaning rule was backed by an ablation test demonstrating its impact.

Results and what I learned

The Ewe n-gram model is reliable: Our top-performing configuration was a modified Kneser-Ney model at order 5. It recorded 99.8 perplexity and 1.45 bits per character on the test split, closely matching its validation score (100.8) and confirming that it avoided overfitting.

N-grams win only when training data is extremely small: The n-gram model outperformed both neural architectures below approximately 10,000 sentences. At 30,000 sentences, the LSTM overtook n-grams, and at 100,000 sentences, both the transformer and wider LSTMs took the lead. Ankora's intuition holds for tiny corpora, but modern Ewe resources are already large enough to reward neural approaches.

Training sentences	n-gram (bits/char)	LSTM	Transformer
1,000	2.49	2.86	3.19
3,000	2.36	2.52	2.96
10,000	2.13	2.11	2.31
30,000	1.92	1.85	1.95
100,000	1.70	1.71	1.66

Dev perplexity against training size: the n-gram wins below about 10,000 sentences, neural models win above.

Data distribution matters more than algorithmic tweaks: Approximately 35% of our raw web data came from religious texts. When we tested the web-trained model on transcribed spoken Ewe, performance dropped from 1.45 to 2.12 bits per character, with 11% of spoken vocabulary completely unrecognized. Incorporating 17,000 spoken sentences lowered test score to 1.70 bits per character. For real-world speech recognition, domain matching outweighs smoothing refinements.

Domain adaptation is effective and efficient: LoRA rank 8 adaptation on Qwen2.5-0.5B lowered agriculture perplexity from 15.9 to 6.7 and healthcare perplexity from 9.1 to 8.5, while general English perplexity remained virtually untouched (moving from 19.7 to 20.4). Higher LoRA ranks improved domain metrics slightly but increased forgetting. Learning rate was the most critical hyperparameter: a rate of 1e-4 degraded performance across all test sets.

More adapter capacity learns the domain better but forgets more general English.

At a learning rate of 1e-4, the health adapter got worse in-domain and lost general English; 1e-5 was safest.

Fluency does not equal factual accuracy: Adapted language models adopt the tone and vocabulary of a domain without necessarily internalizing medical or agricultural facts. For example, when prompted with "The patient was admitted with", our agriculture-adapted model generated instructions on storing veterinary antibiotics. Lower perplexity indicates stylistic alignment, not factual reliability.

My biggest takeaway was that achieving impressive benchmark numbers is deceptively easy when evaluation sets leak or metrics are poorly matched. The hardest and most rewarding part of machine learning engineering is designing tests that remain ruthlessly honest.

Where I would take it next

The logical next step is integrating the Ewe language model into Ankora's core speech recognition pipeline:

• Evaluate on Word Error Rate (WER): Use the language model to rescore candidate hypotheses generated by an Ewe acoustic model. That is the metric that matters in production speech systems, not standalone perplexity.
• Collect more conversational, everyday Ewe audio: The performance drop on spoken speech was our biggest error source. Sourcing wider conversational recordings, like the full Waxal speech corpus, would deliver greater accuracy gains than model architecture changes.
• Conduct native speaker audits: Our data filtering relied on programmatic heuristics. Having native Ewe speakers review sample sentences would catch semantic errors that automated tools miss.
• Benchmark multilingual foundation models: Once training data exceeds 30,000 sentences, neural approaches clearly win. Fine-tuning models already pretrained on African languages would be a natural next step.
• Evaluate domain models on factual question-answering: Test the health and agriculture adapters on structured question-answering benchmarks with expert review, rather than relying solely on perplexity metrics.
• Clarify dataset licensing: Much of the scraped web data lacks clear open-source licensing, preventing public distribution of the resulting models. Resolving copyright permissions would allow other African language researchers to build on this work.