QA & AI Evaluation Engineer - Antwerp office

pDelen Private Bank is a private wealth manager with a well-established IT team of 200 colleagues within an organisation of 800 employees. We support our clients through discretionary wealth management and wealth planning, and are strongly committed to automation through our strategically developed in-house core banking software. /p h3Role and context /h3 pAs bQA AI Evaluation Engineer /b, you safeguard the quality, reliability and safety of the bAIHub Platform /b. This is Delen Private Bank’s central in-house AI platform, integrated into the bank’s processes and systems. /p pAI solutions do not always behave predictably. That is why quality here goes beyond "does the functionality work as intended?" You define how we measure whether an AI application is bcorrect, consistent, safe and useful /b. You also check that it stays that way after changes to models, prompts or data. /p pYou work closely with bAI engineers, developers, business analysts, domain experts and stakeholders /b. You combine bstrong QA expertise with analytical insight into AI systems /b. You translate business expectations into measurable evaluation criteria. You build and maintain evaluation datasets and embed automated evaluations in our delivery pipelines. /p h3Responsibilities /h3 h3Analysis Evaluation Design /h3 ul liWork with business analysts and domain experts to btranslate expectations about AI behaviour into measurable acceptance and evaluation criteria /b. /li libDefine evaluation datasets /b (golden datasets) that represent real-world usage scenarios. Include edge cases and failure scenarios. /li libdeterministic /b: e.g. format validation, schema checks, exact or structured matches, mandatory source citation, correct tool calls; /li libnon-deterministic /b: e.g. LLM-as-a-judge, semantic similarity, groundedness and relevance. /li libCalibrate LLM-as-a-judge evaluations /b against human judgements, so that automated scores stay reliable. /li liHelp shape the bevaluation strategy, coverage and quality thresholds /b per use case and for the platform as a whole. /li /ul h3Test Execution Validation /h3 ul liRun functional and non-functional tests on platform components: APIs, integrations, data processing and user interfaces. /li liEvaluate bRAG applications /b for retrieval quality, groundedness and correct source citation. /li liTest bagentic flows /b: correct tool selection, parameters, multi-turn behaviour and error handling. /li liPerform badversarial testing and red teaming /b: prompt injection, jailbreaks, leakage of confidential or personal data, unwanted or misleading output. /li liValidate that bguardrails /b and content filters work. /li libLog bugs in Azure DevOps /b, actively follow up on their resolution and validate fixed issues. /li libFacilitate UAT sessions /b with end users and collect structured feedback. /li liMonitor bregression on model upgrades and prompt changes /b, and compare versions objectively. /li libMaintain evaluation datasets based on online evaluations /b and production feedback. You turn unforeseen edge cases into permanent test cases within the standard eval sets. /li liTrack quality in production through bmonitoring and tracing /b (quality scores, drift, user feedback) and flag degradation early. /li liUse bAI tooling /b yourself to generate test cases and synthetic test data and to analyse results faster. /li liHelp build breusable evaluation standards, templates and guidelines /b, so that teams building on AIHub can measure the quality of their own use cases. /li liWrite bannotation guidelines /b and involve domain experts in reviewing and labelling output. /li liCommunicate findings clearly to technical and non-technical stakeholders. /li /ul h3Profile and competencies /h3 ul liExcellent Dutch and good English. French is a plus (multilingual evaluation). /li liExperience as a bQA Engineer /b in an agile IT environment, with strong analytical skills. /li libYou have hands-on experience testing or evaluating LLM applications /b, or you show clear maturity in this domain. /li liYou understand the challenges of bnon-deterministic systems /b and report on them reliably and with statistical rigour. /li liYou have a critical and creative "break it" mindset to find edge cases and abuse scenarios. /li liAccurate, structured and quality-driven. /li liTeam player with excellent communication skills. /li /ul h3Basic knowledge of or experience with: /h3 ul libPython /b for test automation and evaluation scripts (e.g. pytest) /li libLLM evaluation frameworks /b (e.g. LangSmith, Azure AI Foundry evaluations, promptfoo, DeepEval, Ragas, Langfuse) /li liConcepts such as bRAG, embeddings, prompt engineering, agents and tool calling /b /li libAPI testing /b (e.g. Postman, REST, OpenAPI) /li libAzure DevOps /b for test management and CI/CD /li libSQL / JSON /b for validating data and interfaces /li libObservability tooling /b (e.g. OpenTelemetry, Application Insights) /li liExperience with or affinity for