LLM Evaluation Engineer

Job Title: LLM Evaluation Engineer Company: UMICHAT (SINGAPORE) PTE. LTD. Job Category: Data Science Roles & Responsibilities: Job Description * Build and maintain automated LLM evaluation pipelines covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role\-playing capabilities. Support one\-click evaluation, historical result comparison, and regression testing. * Conduct general capability evaluations using established benchmarks such as MMLU, C\-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis. * Conduct Agent capability evaluations, including setting up evaluation environments and tracking performance metrics for benchmarks such as BFCL, τ\-bench, and GAIA. * Design and execute persona and role\-playing evaluation frameworks, covering metrics such as identity recognition, role consistency, multi\-turn stability, and style consistency. * Record and analyse evaluation results across model training runs, perform comparisons and anomaly detection, and prepare evaluation reports for training checkpoints. * Conduct regular intermediate evaluations during the pre\-training stage to track the evolution and improvement of model capabilities. * Additional leave * Food provided * Health insurance * Professional development Work Location: In person Work Location: Geylang, S00, Singapore Availability: Full time

Job Requirements * Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related discipline. * Familiar with mainstream LLM evaluation benchmarks and frameworks, such as lm\-eval\-harness, OpenCompass, and EvalPlus. * Proficient in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis. * Familiar with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation. * Experience in evaluation data analysis and visualisation. * Detail\-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results. Benefits: Key Skills: Python Salary: SGD 4000 - 10000 per monthly