Build a Nigerian Language Model — Koinder Practitioner Track

What you will build, and why it is worth building

By the end of this module you will have a working model that reads a sentence and tells you whether it is Yoruba, Igbo, Hausa, Nigerian Pidgin or English. You will write it from scratch, in about eighty lines, with no machine learning library at all.

That last part is deliberate. When you have written the mathematics yourself, you can debug it. People who only ever called a library cannot.

Why this problem and not something else

The tools built abroad genuinely fail here

Most commercial language detectors handle Yoruba, Igbo and Hausa poorly or not at all, and generally do not recognise Nigerian Pidgin as a language — despite it having tens of millions of speakers. This is not a gap you are imagining. It is a gap you can close.

You can collect the data yourself

You do not need a research budget. You need a few thousand sentences, and you are surrounded by people who produce them.

The pattern transfers to everything

Text in, category out. Once you can do this, you can do sentiment, topic, spam, urgency, or any classification problem in any Nigerian language.

Step 1 — Collect the data. This is the real work.

Aim for at least 300 sentences per language to begin with, 1,000+ to be useful. Here is where to get them, in order of how much trouble they cause:

Easiest — your own WhatsApp

With permission, copy sentences from group chats. Real, current, informal language. This is the most valuable text you have access to and no laboratory abroad can get it.

Radio and television

Transcribe from BBC Yoruba, BBC Hausa, BBC Igbo, and local stations. Slow, but the quality is high.

Wikipedia in each language

Yoruba, Igbo and Hausa Wikipedias all exist. Free and open. Note the register is formal and written, which is a limitation you should record.

Ask people

Give twenty friends a form. "Write ten sentences about your day in your language." Two hundred sentences in a weekend, and it is natural speech.

The rule that matters

Store one sentence per line with its language label, in a plain text file. Nothing clever. A folder of five text files is a perfectly good dataset, and it is yours.

One warning that will save you a week — read the next section before collecting.

Step 2 — The mistake I made, measured

When I built the demonstration for this module, I wrote twenty sample sentences in each language. For English I wrote the translations of the Pidgin sentences. It seemed tidy.

What I had actually created

pidgin : How you dey today
english: How are you doing today

Same meaning. Same topic. Same sentence shape. I had made the two hardest languages to separate as similar as they could possibly be, and I did not notice.

Then I fixed only the collection — same model, same algorithm, same number of sentences — and replaced the translated English with ordinary independent English sentences:

English collected as…	Overall	Pidgin vs English
translations of the Pidgin	74%	42%
real independent sentences	93%	90%

Nineteen points overall. Forty-eight on the hard pair. From data collection alone.

I also tried fixing it with better features first — adding whole-word features on top of the character patterns. That bought four points. The collection fix bought forty-eight.

This is the lesson of the whole module. When your model is disappointing, the instinct is to reach for a better algorithm. Look at how the data was gathered first. It is almost always cheaper and almost always bigger.

Step 3 — The model. Eighty lines, no libraries.

The idea: different languages use different letter patterns. Yoruba is full of -wo- and -se-. Igbo has -nwa- and -chi-. Hausa has -ana- and -yya-. You count which patterns appear in which language, and for a new sentence you ask which language's counts explain it best.

That is Naive Bayes. It is over two hundred years old and it works.

The two helper functions

import math, re
from collections import defaultdict

def clean(t):
    t = t.lower()
    # keep Yoruba and Igbo diacritics — they carry real information
    t = re.sub(r'[^\w\sàáèéìíòóùúẹọṣńǹḿ̀́]', ' ', t)
    return re.sub(r'\s+', ' ', t).strip()

def grams(t, n=3):
    t = ' ' + clean(t) + ' '     # spaces mark word starts and ends
    return [t[i:i+n] for i in range(len(t)-n+1)]

Run this alone first. print(grams("Bawo ni o se wa")) and look at the output. Do not continue until you can see what it produced.

Training — just counting

class LangID:
    def __init__(self, n=3):
        self.n = n
        self.counts = defaultdict(lambda: defaultdict(int))
        self.totals = defaultdict(int)
        self.vocab  = set()
        self.prior  = defaultdict(int)

    def train(self, rows):                 # rows = [(sentence, language), ...]
        for text, lang in rows:
            self.prior[lang] += 1
            for g in grams(text, self.n):
                self.counts[lang][g] += 1
                self.totals[lang]    += 1
                self.vocab.add(g)

Notice there is no "learning algorithm" here. Training is counting. Most people are surprised by this, and it is worth sitting with.

Predicting — which language explains this best?

    def predict(self, text):
        V = len(self.vocab)
        scores = {}
        for lang in self.prior:
            # start with: how common is this language in my data?
            s = math.log(self.prior[lang] / sum(self.prior.values()))
            for g in grams(text, self.n):
                # +1 so an unseen pattern does not zero everything out
                s += math.log((self.counts[lang][g] + 1) / (self.totals[lang] + V))
            scores[lang] = s
        best = max(scores, key=scores.get)
        ordered = sorted(scores.values(), reverse=True)
        margin  = ordered[0] - ordered[1]      # how far ahead is the winner?
        return best, margin

The margin is not decoration. It is your confidence. A small margin means the model is guessing, and a system that knows when it is guessing can hand over to a human — which is Pattern 6 from Module 1, appearing in your own code.

Step 4 — Evaluate honestly, and be surprised

Split your data: about 75% to train on, 25% held back. Never test on sentences the model trained on — that measures memory, not learning.

Running the demonstration data gave 76% overall. That sounds mediocre. Now look at where the mistakes actually are:

truth \ guess	Yor	Igb	Hau	Pid	Eng
Yoruba	6	0	0	0	0
Igbo	0	6	0	0	0
Hausa	0	0	6	0	0
Pidgin	0	0	0	1	5
English	0	0	0	3	3

The model is PERFECT on Yoruba, Igbo and Hausa. It is near-random on one pair.

"76% accurate" told you none of that. It hid a model that is excellent at three jobs and useless at one. Always break your results down by group. A single number is almost always a lie of omission — and now you have seen it in your own results rather than being told.

Build the confusion matrix before you build anything else. It takes ten lines and it will tell you what to fix.

Step 5 — Now point it at something else. This is the real prize.

You have not learned to build a language detector. You have learned a shape: text in → category out, learned from labelled examples. Change the labels and the same eighty lines do a different job.

Same code, new labels

positive / negative → sentiment in Pidgin for customer service
complaint / enquiry / praise → route incoming WhatsApp messages
urgent / normal → triage a support queue
genuine / scam → flag fraudulent SMS in local languages

How to find an African AI problem worth building

Three conditions. When all three hold, you have something.

The knowledge exists but nobody wrote it down. A trader knows which fish is fresh. A nurse knows which child needs attention now. They cannot fully explain how. That is exactly where learning from examples beats writing rules.

Tools built elsewhere fail here. Not because they are bad, but because they never saw this. They were trained on other people's world.

You can reach data that others cannot. This is your genuine advantage, and it does not appear on anyone's CV. A laboratory abroad cannot get Nigerian WhatsApp conversations, market prices from Mile 12, or how a Lagos address is really written.

Nigerian addresses

"Behind the big church, off Ojota bus stop, second gate." That is a real, functional, unambiguous address here. Every Western geocoder fails on it completely. Anyone who solves informal address parsing for Nigeria has built something with obvious buyers.

Yoruba diacritic restoration

People type owo without tone marks, but owó is money and owo is broom. Restoring the marks from context is an unsolved, genuinely useful research problem, and the pattern is the same one you just learned.

Nigerian-accent speech

Speech recognition trained on American English performs poorly on Nigerian speakers. Collecting a few hundred hours of local speech with transcripts is laborious, unglamorous, and would be extremely valuable.

Crop disease on local varieties

Existing plant disease models were trained largely on varieties grown elsewhere. Nigerian cassava, yam and maize varieties look different. Photographs from real farms here are data that does not exist yet.

Market prices

Nobody has a clean live dataset of Nigerian food prices by market. Collecting one is boring, and it would be worth a great deal to traders, researchers and government alike.

Notice what every single one of those has in common. The hard part is the data, not the model. That is the whole lesson, and it is also your opportunity.

Your assignment

Collect 300 sentences per language. Do it yourself. Note where each came from.

Check for the trap I fell into: are any two of your sets translations of each other?

Train the model. Report accuracy AND the confusion matrix.

Try n = 2, 3 and 4 and write down what changed.

Put it behind the endpoint you build in the capstone, so a stranger can use it.

Publish the dataset. A cleanly labelled Nigerian language corpus is a contribution, and it is the kind of thing people remember your name for.

Then say this in an interview: "I built a Nigerian language classifier from scratch. It was 76% until I found my English and Pidgin samples were translations of each other. Fixing the collection took it to 93% — the model never changed." That is fifteen seconds, it contains a number, and it demonstrates the one instinct that separates a data scientist from someone who has watched a tutorial.

Build a Nigerian Language AI, From Nothing