Chandra 2.1: Improved Multilingual and Table Accuracy

We’re excited to announce the release of Chandra 2.1, which is smaller and faster, while improving on several key axes of accuracy that our customers care about. This release focuses on multilingual and table performance, which are particularly challenging for models to handle with 100% accuracy.

Chandra 2.1 is live on the Datalab API today. All results in this post were produced via the API in mode=balanced.

Tables

Perfect extraction of a complex table requires a model to perfectly handle multiple things:

• Understand the structure of the table, including the number of columns, rows, and keep track of spanning cells, which carry significant semantic meaning.
• Read the content of each cell, without missing any minor details, like decimals, diacritics, or punctuation. Dense tables have very small cells, which makes it hard for the model to read them accurately.
• Reconcile the content of each cell with the table structure, and produce the final output.

100% accurate table extraction is a critical requirement for many applications - A single mis-aligned column header in a medical document, or mis-reading $15,000 as $1500 in a financial document, can be catastrophic.

Existing table benchmarks often fail to capture the most complex of real-world tables, or fail to focus on the structural aspects of the table. We built out our own internal benchmark to address this, comprised of the hardest tables we’ve received from our customers. Chandra 2.1 achieves a much higher score than Chandra 2 on this benchmark, and we’re just getting started!

Chandra 2 Chandra 2.1

• Pass rate
  79.5%

  82.1%
  +2.6

Here are a few examples of the kind of complex tables that Chandra 2.1 is able to extract:

Multilingual

Chandra 2 was a major step up on multilingual accuracy, and Chandra 2.1 keeps pushing - particularly on Indic scripts and Arabic.

Chandra 2 Chandra 2.1

• Pass rate
  72.7%

  75.2%
  +2.5

The biggest gains land on Indic scripts.

Chandra 2 Chandra 2.1

• Hindi (hi)
  82.7%

  86.3%
  +3.6
• Bengali (bn)
  77.3%

  81.2%
  +3.9
• Tamil (ta)
  82.9%

  88.4%
  +5.5
• Telugu (te)
  64.1%

  71.4%
  +7.3
• Kannada (kn)
  70.9%

  78.3%
  +7.4
• Malayalam (ml)
  73.3%

  79.1%
  +5.8
• Gujarati (gu)
  74.9%

  83.0%
  +8.1
• Marathi (mr)
  82.8%

  84.5%
  +1.7
• Punjabi (pa)
  76.1%

  77.6%
  +1.5
• Odia (or)
  45.8%

  54.2%
  +8.4
• Nepali (ne)
  77.9%

  81.5%
  +3.6
• Sanskrit (sa)
  63.1%

  67.9%
  +4.8

Right-to-left scripts saw the largest single-language jumps — Arabic from 56.8% to 77.2%, and Urdu from 61.2% to 72.3%.

Chandra 2 Chandra 2.1

• Arabic (ar)
  56.8%

  77.2%
  +20.4
• Urdu (ur)
  61.2%

  72.3%
  +11.1

Try it out

Chandra 2.1 is already live. You can try it out in the playground or via the API, or contact us at support@datalab.to for on-premise deployments.

What’s next

We’re continuing to invest in the research that has led to Chandra 2.1, and expect to release more updates in the coming months. If you have documents with complex tables, messy multilingual content, or any other challenging content that you’d like to see improved, please contact us.