Language Detection and Translation for Multilingual AI Agents

Why Multilingual Support Matters for Agents

A customer support agent that only understands English excludes roughly 80% of the world's population from the conversation. Even in predominantly English-speaking markets, agents encounter messages in Spanish, French, Mandarin, and dozens of other languages from diverse user bases. A truly capable agent detects the user's language automatically, processes the request in the original language or translates it, and responds in the language the user prefers.

Language Detection

The first step in any multilingual pipeline is identifying which language the user is writing in. The lingua-py library provides fast, accurate detection across 75 languages.

from lingua import Language, LanguageDetectorBuilder

# Build a detector for common languages
detector = LanguageDetectorBuilder.from_languages(
    Language.ENGLISH,
    Language.SPANISH,
    Language.FRENCH,
    Language.GERMAN,
    Language.PORTUGUESE,
    Language.CHINESE,
    Language.JAPANESE,
    Language.KOREAN,
    Language.ARABIC,
    Language.HINDI,
).build()

def detect_language(text: str) -> dict:
    """Detect language with confidence scores."""
    confidence_values = detector.compute_language_confidence_values(text)
    results = [
        {"language": cv.language.name, "confidence": round(cv.value, 3)}
        for cv in confidence_values[:3]
    ]

    detected = detector.detect_language_of(text)
    return {
        "detected": detected.name if detected else "UNKNOWN",
        "top_candidates": results,
    }

print(detect_language("Necesito ayuda con mi cuenta"))
# {'detected': 'SPANISH', 'top_candidates': [
#   {'language': 'SPANISH', 'confidence': 0.98}, ...]}

print(detect_language("Je voudrais réserver une table"))
# {'detected': 'FRENCH', 'top_candidates': [
#   {'language': 'FRENCH', 'confidence': 0.97}, ...]}

Handling Short and Mixed-Language Text

Short texts (under 20 characters) and code-switched messages are notoriously difficult for language detectors. Here is a robust detection strategy.

def robust_detect(text: str, fallback: str = "ENGLISH") -> str:
    """Detect language with fallback for short or ambiguous text."""
    if len(text.strip()) < 10:
        return fallback

    confidence_values = detector.compute_language_confidence_values(text)
    if not confidence_values:
        return fallback

    top = confidence_values[0]
    if top.value < 0.6:
        return fallback

    # Check if top two are close (mixed language indicator)
    if len(confidence_values) > 1:
        gap = top.value - confidence_values[1].value
        if gap < 0.15:
            return fallback

    return top.language.name

Translation with Multiple Providers

Production agents should support multiple translation backends with automatic failover.

from abc import ABC, abstractmethod
from typing import Optional

class TranslationProvider(ABC):
    @abstractmethod
    async def translate(
        self, text: str, source: str, target: str
    ) -> Optional[str]:
        pass

class DeepLTranslator(TranslationProvider):
    def __init__(self, api_key: str):
        import deepl
        self.client = deepl.Translator(api_key)

    async def translate(
        self, text: str, source: str, target: str
    ) -> Optional[str]:
        try:
            result = self.client.translate_text(
                text, source_lang=source, target_lang=target
            )
            return result.text
        except Exception:
            return None

class OpenAITranslator(TranslationProvider):
    def __init__(self, api_key: str):
        import openai
        self.client = openai.AsyncOpenAI(api_key=api_key)

    async def translate(
        self, text: str, source: str, target: str
    ) -> Optional[str]:
        try:
            response = await self.client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{
                    "role": "user",
                    "content": (
                        f"Translate from {source} to {target}. "
                        f"Return only the translation:\n{text}"
                    ),
                }],
                temperature=0,
            )
            return response.choices[0].message.content
        except Exception:
            return None

Translation Quality Assessment

Not all translations are equal. An agent should verify translation quality before acting on translated content.

def assess_translation_quality(
    original: str,
    translated: str,
    back_translated: str,
) -> dict:
    """Assess translation quality using back-translation comparison."""
    from difflib import SequenceMatcher

    similarity = SequenceMatcher(
        None,
        original.lower(),
        back_translated.lower(),
    ).ratio()

    length_ratio = len(translated) / max(len(original), 1)
    length_reasonable = 0.5 <= length_ratio <= 3.0

    return {
        "back_translation_similarity": round(similarity, 3),
        "length_ratio": round(length_ratio, 2),
        "length_reasonable": length_reasonable,
        "quality_score": round(similarity * (1.0 if length_reasonable else 0.7), 3),
        "acceptable": similarity > 0.6 and length_reasonable,
    }

Building a Multilingual Agent Pipeline

Here is the complete pipeline that wraps language detection, translation, agent processing, and response translation into a seamless flow.

class MultilingualAgent:
    def __init__(self, agent, translators: list[TranslationProvider]):
        self.agent = agent
        self.translators = translators
        self.user_languages: dict[str, str] = {}

    async def translate_with_fallback(
        self, text: str, source: str, target: str
    ) -> str:
        for translator in self.translators:
            result = await translator.translate(text, source, target)
            if result:
                return result
        return text  # Return original if all translators fail

    async def handle_message(
        self, user_id: str, message: str
    ) -> str:
        # Step 1: Detect language
        detected = detect_language(message)["detected"]
        self.user_languages[user_id] = detected

        # Step 2: Translate to English if needed
        if detected != "ENGLISH":
            english_message = await self.translate_with_fallback(
                message, source=detected, target="ENGLISH"
            )
        else:
            english_message = message

        # Step 3: Process with agent (in English)
        response = await self.agent.process(english_message)

        # Step 4: Translate response back to user language
        if detected != "ENGLISH":
            return await self.translate_with_fallback(
                response, source="ENGLISH", target=detected
            )
        return response

This architecture centralizes the agent's reasoning in one language (English, typically) while presenting a multilingual interface to users. The key advantage is that you maintain one set of prompts, tools, and business logic rather than duplicating everything per language.

FAQ

Should I translate user messages to English before processing, or build a natively multilingual agent?

Translate to English for most use cases. Modern LLMs understand many languages, but their reasoning quality varies significantly by language — English typically produces the best results. The translate-process-translate pattern gives you consistent quality across all languages while requiring only one set of prompts and tools. Build natively multilingual only if translation latency is unacceptable or if you need to preserve language-specific nuances like legal terminology.

How do I handle code-switched messages where users mix two languages in one sentence?

Detect the dominant language and translate the entire message. Sentence-level language detection can split mixed messages, but it often makes errors at code-switch boundaries. A simpler and more reliable approach is to pass the full mixed message to an LLM-based translator with an instruction like "Translate any non-English portions to English while preserving the English parts."

What is the best way to handle language detection failures?

Default to English and ask the user explicitly. If detection confidence is below 60%, respond with a multilingual prompt: "I want to help you in your preferred language. / Me gustaria ayudarte en tu idioma preferido. / Je souhaite vous aider dans votre langue." This avoids the frustration of the agent guessing wrong and responding in an unexpected language.

---

#LanguageDetection #Translation #Multilingual #NLP #AIAgents #Python #AgenticAI #LearnAI #AIEngineering