Medior AI Engineer

h3Mission Overview: /h3 pKeystone Solutions is seeking a Medior AI Engineer to join our consultancy mission at a client site. You will be part of the AI Center of Excellence, focusing on the Run phase of AI services in production. This role involves ensuring the reliability, security, and cost-efficiency of AI services built on a hybrid operating model, utilizing both managed Azure services and self-hosted open-source components. /p h3Key

Responsibilities:
/h3 ul liOperate the AI services in production: monitor availability, latency, and errors, handle incidents and service requests, analyze root causes, then fix or elevate. /li liKeep answer quality under control: run and maintain evaluation sets with business content owners, track answer quality, non-answer rate, and user feedback, and detect and investigate quality regressions. /li liOperate the ingestion pipelines: manage incremental refresh, failed documents, index health, and onboarding of new content sources or business domains. /li liOperate agent integrations: monitor tool and integration calls to business systems, handle failures and timeouts, and review action logs and exceptions. /li liManage change safely: model version upgrades and retirements, prompt and configuration changes, and dependency and security updates, validated against evaluation sets before release. /li liFollow run cost: monitor token consumption and cost per service, alert on drift, and propose optimizations. /li liMaintain Terraform, Helm, and Azure DevOps artifacts, ensuring consistency across DEV, TST, ACC, and PROD environments. /li liApply data protection rules: manage trace retention, erasure, and access restrictions as defined in each service’s data protection impact assessment. /li liMaintain runbooks and documentation, and report service health to service owners and the AI CoE regularly. /li liDeliver small enhancements and fixes; structural changes and new features will be handled by the Build team. /li /ul h3Why Join Keystone Solutions? /h3 pAs a consultant with Keystone Solutions, you will

experience:
/p ul libConsultancy Nature: /b Engage in on-site work, embodying the consultant identity. /li libDynamic Projects: /b Tackle diverse challenges across various client environments. /li libTurbo-Charged Learning: /b Benefit from professional development and broad learning experiences. /li libSkyrocketing Ambition: /b Enjoy a commitment to career growth within our framework. /li libValues: /b Being a "K-Stone" means bringing our core values to every engagement. /li /ul h3Duration: /h3 p04/01/2027
- 31/12/ months (full time) /p h3Skills required: /h3 ul libAzure DevOps /b
- Level: Intermediate
- Most recent: This year /li libAzure Kubernetes Service
- AKS /b
- Level: Intermediate
- Most recent: This year /li libAzure OpenAI Service /b
- Level: Intermediate
- Most recent: This year /li libLangChain /b
- Level: Intermediate
- Most recent: Any time /li libLLM / Large Language Models /b
- Level: Intermediate
- Most recent: This year /li libPython /b
- Level: Intermediate
- Most recent: This year /li libRAG (Retrieval-Augmented Generation) /b
- Level: Intermediate
- Most recent: This year /li libTerraform /b
- Level: Intermediate
- Most recent: This year /li libVector databases /b
- Level: Intermediate
- Most recent: This year /li /ul h3Language

requirements:
/h3 ul liDutch or French Level Native or bilingual proficiency /li liEnglish Level Full professional proficiency /li /ul