31% — that’s the average accuracy gap between top-tier Arabic language models and their English counterparts on complex reasoning tasks, according to the original report released by Hugging Face and the Technology Innovation Institute (TII) on April 30, 2026.
Key Takeaways
- The QIMMA ⛰ Arabic LLM leaderboard evaluates models on accuracy, safety, fluency, and reasoning — not just benchmark scores.
- It’s the first Arabic-focused evaluation platform with open, reproducible methodology hosted on Hugging Face.
- TII’s Falcon series and Google’s Gemma-2B Arabic fine-tune appear in early rankings — but neither leads.
- The top model as of April 30, 2026 is Abu Dhabi AI’s Qaid 7.8B, trained on 1.2 trillion Arabic tokens.
- Unlike global leaderboards, QIMMA penalizes models that hallucinate religious, historical, or geopolitical content specific to the Arab world.
Not Another Benchmark — A Quality Filter
Let’s be honest: most LLM leaderboards reward speed, scale, or synthetic benchmarks no human would ever use. QIMMA doesn’t care how fast your model generates filler. It grades how well it understands Arabic — in context, with nuance, and without making things up about Ibn Khaldun or the 1967 war.
The name itself signals intent. Qimma means “summit” or “peak” in Arabic — a fitting metaphor for what TII and Hugging Face are trying to achieve. This isn’t about flooding the zone with models. It’s about finding the few that actually work.
And they’re not relying on BLEU scores or token-per-second throughput. The evaluation suite includes 176 hand-curated prompts across four domains: classical Arabic poetry interpretation, fatwa-style ethical reasoning, Gulf vs. Levantine dialect comprehension, and fact-checking claims about Arab scientists. If a model says Al-Khwarizmi was born in 900 CE, it fails. (He was born around 780.)
That kind of precision is missing from most multilingual evaluations. Google’s TyDiQA or Meta’s XLS-R treat Arabic as just another language in a batch. QIMMA treats it like a civilization with its own epistemic rules.
Why Arabic Falls Behind — And Why It Matters
English LLMs have an insurmountable data advantage. There are over 300 billion public web pages in English. In Arabic? Around 4.2 billion. That’s less than Portuguese. And much of it is low-quality OCR scans, duplicated religious texts, or auto-translated content.
But the problem isn’t just data size. It’s data trust. Arabic has diglossia — a split between Modern Standard Arabic and dozens of spoken dialects. A model can ace MSA and still fail to understand a conversation in Jeddah or Casablanca.
And then there’s ideology. Arabic content online is heavily politicized. Train a model on raw scrapes, and you’ll get one that defaults to a specific regional narrative — whether Saudi, Iranian-influenced, or Islamist-leaning. That’s not intelligence. That’s bias baked in.
QIMMA’s evaluation suite detects these drifts. If a model consistently portrays the Ottoman Empire as a golden age of Arab freedom, it gets flagged. Same if it erases North Africa from Arab history. Neutrality isn’t the goal — factual consistency is.
How QIMMA Scores Models
- Accuracy: Verified against scholarly sources for historical, scientific, and cultural claims.
- Dialect Handling: Tested on 12 regional variants from Muscat to Tunis.
- Safety: Penalized for generating offensive, sectarian, or extremist content — even if prompted.
- Reasoning Depth: Measured by multi-step logic in religious ethics, legal hypotheticals, and scientific inference.
- Fluency: Evaluated by native linguists for natural phrasing and grammatical correctness.
No model gets full points. The current leader, Qaid 7.8B, scores 84.3% overall. GPT-4o Arabic mode sits at 79.1%. Meta’s Llama 3 70B? 72.6% — dragged down by repeated errors in Levantine dialect and religious content.
The Real Winner: Open Evaluation
Here’s what’s remarkable: QIMMA isn’t a closed contest. All evaluation data, scoring rubrics, and model outputs are public on Hugging Face. Anyone can audit a result. Anyone can submit a model.
That’s rare. Most benchmarks — including Stanford’s HELM or LMSys Chatbot Arena — keep their scoring logic opaque. You see a rank. You don’t see why.
QIMMA shows you the transcript. If Model X says Salah al-Din united all Arab states, you’ll see the prompt, the response, and the fact-check annotation. You’ll also see the confidence score and whether it triggered any safety filters.
This transparency forces accountability. And it creates a feedback loop developers can actually use. No more guessing why your model failed. Now you know — and you can fix it.
TII’s Quiet Bet on Regional AI
The Technology Innovation Institute in Abu Dhabi isn’t just publishing leaderboards. It’s building the stack. Falcon, their open-weight LLM series, has been quietly gaining traction since 2023. Now, with Qaid — their first Arabic-native model — they’re not just competing. They’re defining the rules.
And they’re doing it without the hype. No billion-dollar valuation announcements. No celebrity CEO. Just a steady stream of research, tooling, and now, evaluation infrastructure.
That’s a different playbook from Silicon Valley. While others race to monetize AI assistants, TII is laying foundations. First, open models. Then, evaluation. Next? Probably training data — clean, verified, region-specific.
The Bigger Picture: AI Sovereignty in the Arab World
QIMMA isn’t just a technical project. It’s a statement about digital sovereignty. For years, the Middle East has imported AI — from Silicon Valley, Beijing, and Zurich. Those models were trained on Western norms, legal frameworks, and cultural assumptions. They don’t understand wa’d in Omani contracts or the nuances of Islamic banking in Kuwait.
Now, Abu Dhabi is pushing back. TII isn’t just building models. It’s building the entire evaluation ecosystem from the ground up, in Arabic, for Arabic speakers. That shifts power. It means governments, hospitals, and universities in the Gulf can deploy AI without relying on foreign systems that misread local context.
Other regions are watching. Nigeria’s National AI Strategy team referenced QIMMA in a March 2026 policy workshop. So did Indonesia’s Ministry of Communication. Both face similar challenges: rich linguistic diversity, fragmented digital content, and rising demand for localized AI.
But the Arab world has a unique advantage. It’s home to some of the earliest written legal, scientific, and philosophical texts. Digitizing and integrating that heritage into AI isn’t just about language. It’s about reclaiming narrative control — from who writes history to who designs the next generation of AI assistants.
Competing Visions: How Other Players Are Responding
TII isn’t alone in recognizing the gap. Google launched its Arabic AI research lab in Riyadh in 2024 with a $20 million investment. Its focus? Fine-tuning Gemma and scaling dialect coverage. But its public benchmarks remain embedded in broader multilingual suites like TyDiQA, where Arabic is one of eleven languages and rarely analyzed in isolation.
Meta has taken a different path. In 2025, it released Noor, a 15B-parameter Arabic-dominant model trained on a 680 billion token dataset scraped from.sa,.eg, and.ma domains. But early tests showed it struggled with legal reasoning and often defaulted to Egyptian Arabic, even when prompted in Maghrebi dialects. Meta hasn’t updated Noor since January 2026.
Meanwhile, smaller players are filling niches. Jordan-based Zaytona AI launched a medical Q&A model in 2025 trained on 40,000 verified Arabic clinical guidelines. It scores 76% on QIMMA’s health reasoning subset — below Qaid, but ahead of GPT-4o in domain-specific accuracy. Saudi’s SAKHRA, funded by the King Abdulaziz City for Science and Technology, is building a dialect-aware speech model for government services, with pilot deployments in Riyadh and Jeddah.
But none have matched QIMMA’s transparency. Their evaluation data isn’t public. Their safety protocols aren’t auditable. That opacity makes it hard to trust them in high-stakes settings — like education or law enforcement.
What This Means For You
If you’re building AI for the Arab world, QIMMA gives you a target. You’re no longer guessing whether your model “feels” accurate. You’ve got a score, a breakdown, and a path to improve. You’ll need native speakers in your loop, sure. But now you’ve got data to back up your iterations.
If you’re an open-source maintainer, this is a wake-up call. Llama, Mistral, even smaller fine-tunes — they’re being graded in public. And they’re underperforming. That’s not a shame. It’s a challenge. QIMMA makes it impossible to claim “our model supports Arabic” unless it actually works.
The next frontier? Fine-tuning for medical, legal, and educational use cases. Right now, no Arabic LLM hits 80% on clinical reasoning. But with a leaderboard like QIMMA, that could change in 12 months.
Will we see a truly reliable Arabic AI tutor by 2027? One that can explain calculus in Levantine dialect or guide a patient through a diagnosis without hallucinating cures? The tools to get there are finally in place.
Sources: Hugging Face Blog, TechCrunch
