Remote AI Benchmark Engineer

ph3strongAbout The Opportunity /strong /h3pWe are building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. Our goal is to measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows. /ppWe are seeking experienced native-speaking software engineers to design, build, and validate these benchmarks. You will create high-signal, high-quality tasks that genuinely test a model's ability to handle multilingual environments without relying on English translation crutches. /ppstrongNote this is a remote, freelance opportunity /strong /ph3strongWhat You’ll Deliver /strong /h3ullipTask Engineering: Evaluating Coding Agents. /p /lilipAsset Creation: Build realistic task environments using datasets and files in your native language. emCrucially, these assets must remain in the target language to genuinely measure multilingual handling. /em /p /lilipPrompting Translation: finding failure points where AI does not work, in your native language /p /lilipImplementation Verification: Support the development of robust solutions (reference implementations) and write highly reliable, deterministic verifier scripts (using rubric-based judging only when strictly necessary). /p /lilipCalibration Execution: Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus). /p /lilipQuality Assurance: Participate in a rigorous, 4-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity. /p /li /ulh3strongQualifications /strong /h3ullipstrongExperience: /strong 1+ years of industry experience in software engineering. /p /lilipstrongBackground: /strong Proven track record at leading technology companies and/or graduation from top-tier engineering universities. /p /lilipstrongLanguage: /strong Native or near-native fluency, with a deep understanding of its grammar, register, and phrasing rules. High English proficiency. /p /lilipstrongTechnical Stack: /strong Strong proficiency in Python, standard shell scripting, and data processing. /p /lilipstrongWorkflow: /strong Extensive experience with Terminal/CLI-based development workflows and a working familiarity with coding agents. /p /lilipstrongDomain Expertise: /strong Deep technical understanding of multilingual text processing pitfalls, including: /pullipEncoding/decoding robustness and Unicode normalization. /p /lilipLocale-dependent conventions (collation, casing, non-Gregorian dates). /p /lilipText I/O, toolchain interoperability, and safe string operations. /p /lilipem(For specific languages) /em Bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts. /p /li /ul /li /ulh3strongWhy Collaborate with Lilt? /strong /h3ullipstrongYour schedule, your rules. /strong As an independent contractor, work when you want, as much or as little as you want. No fixed hours, no check-ins, no micromanaging. /p /lilipstrongGet paid quickly and fairly. /strong We respect your time and your expertise. Competitive rates, prompt payments, no chasing invoices. /p /lilipstrongWork on projects that actually matter /strong. Contribute to cutting-edge AI and language technology that is shaping how humans and machines communicate. /p /lilipstrongBe part of something bigger. /strong Join a global community of linguists, subject matter experts, and language professionals who are advancing human knowledge together. /p /lilipstrongGrow without limits. /strong As a Lilt contractor you get access to diverse, innovative projects that expand your portfolio and sharpen your skills across industries and domains. /p /lilipstrongHave fun doing what you love. /strong Bring your language skills to life on projects that are as interesting as they are impactful.
We are building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. Our goal is to measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows. /p /li /ulh3strongWhat to Consider Before Applying /strong /h3ullipstrongNot ideal as a full time job or primary income source. /strong Work availability fluctuates with project demand, making this better suited as a supplemental income stream. As a 1099 contractor, you won't receive benefits such as health insurance, paid time off, or retirement contributions, and hours are not guaranteed. /p /lilipstrongRequires reliable availability and commitment. /strong Once you accept a task, we expect quality work and on-time delivery. Most tasks require a minimum of 2 hours