Ifm Us

<div><span>About the Institute of Foundation Models</span></div>
<div>We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.</div>
<div> </div>
<div><strong>As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. </strong>You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.</div>
<div> </div>
<div> </div><div><br /></div><div>
<h2>The Role</h2>
<p>We’re looking for an RL infrastructure engineer to help extend and scale our end-to-end RL training systems. You’ll work side-by-side with world-class researchers and engineers to:</p>
<ul>
<li>Extend distributed training frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod)</li>
<li>Integrate rollout generation, reward computation, trajectory processing, policy updates, and weight synchronization</li>
<li>Build robust config + launch systems across multi-node, multi-GPU clusters</li>
<li>Own experiment tracking, metrics logging, and job monitoring for external visibility</li>
<li>Improve training system reliability, maintainability, and performance</li>
</ul>
<p>Hands-on experience developing end-to-end RL infrastructure is required. Strong infrastructure and systems experience is what we value most.</p>
</div>
<h2>Key Responsibilities</h2><div>
<ul>
<li>Distributed Framework Ownership – Extend or modify training frameworks (e.g., DeepSpeed, FSDP) to support new use cases and architectures.</li>
<li>Training &amp; Inference Integration – Connect training workers with rollout generation engines (e.g., vLLM, SGLang), including trajectory exchange and policy-weight synchronization.</li>
<li>Launch Config &amp; Debugging – Create and debug multi-node launch scripts with flexible batch sizes, parallelism strategies, and hardware targets.</li>
<li>Metrics &amp; Monitoring – Build systems for experiment tracking, job monitoring, and logging usable by collaborators and researchers.</li>
<li>Infra Engineering – Write production-quality code and tests for ML infra in PyTorch or JAX; ensure reliability and maintainability at scale.</li>
<li>RL Pipeline Development – Implement reward/verifier integration and trajectory processing, and validate log probabilities, token masks, and loss inputs with researchers.</li>
<li>RL Execution &amp; Recovery – Coordinate rollout and training workers, including checkpoint/restart and failure recovery; track policy versions and sample staleness when using asynchronous execution.</li>

</ul></div>
<h2>Qualifications</h2><div>
<h3>Must-Haves:</h3>
<ul>
<li>5+ years of experience in ML systems, infra, or distributed training</li>
<li>Experience modifying distributed ML frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod)</li>
<li>Strong software engineering fundamentals (Python, systems design, testing)</li>
<li>Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and debugging skills (e.g., NCCL/GLOO)</li>
<li>Ability to implement algorithms across GPUs/nodes based on mathematical specs</li>
<li>Experience working on an ML platform/ infrastructure, and/or distributed inference optimization team</li>
<li>Experience with large-scale machine learning workloads (strong ML fundamentals)</li>
<li>Hands-on experience developing an LLM RL pipeline, with ownership across rollout/inference and distributed training integration.</li>
<li>Working knowledge of policy optimization methods such as PPO or GRPO, sufficient to implement and debug sampling, log probabilities, and policy updates.</li>

<h3>Nice-to-Haves:</h3>
<ul>
<li>Exposure to mixed-precision training (e.g., bf16, fp8) with accuracy validation</li>
<li>Familiarity with performance profiling, kernel fusion, or memory optimization</li>
<li>Open-source contributions or published research (MLSys, ICML, NeurIPS)</li>
<li>CUDA or Triton kernel experience</li>
<li>Experience with large-scale pre-training</li>
<li>Experience building custom training pipelines at scale and modifying them for custom needs</li>
<li>Deep familiarity with training infrastructure and performance tuning</li>
<li>Experience with asynchronous RL, multi-turn or tool-using rollout environments, or custom reward/verifier systems.</li>

</ul></ul></div>

Machine Learning Engineer — Reinforcement Learning

<div><span>About the Institute of Foundation Models</span></div>
<div>We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.</div>
<div> </div>
<div><strong>As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. </strong>You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.</div>
<div> </div>
<div> </div><div><br /></div><div>
<h2>The Role</h2>
<p>We’re looking for a distributed ML infrastructure engineer to help extend and scale our training systems. You’ll work side-by-side with world-class researchers and engineers to:</p>
<ul>
<li>Extend distributed training frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod)</li>
<li>Implement distributed optimizers from mathematical specs</li>
<li>Build robust config + launch systems across multi-node, multi-GPU clusters</li>
<li>Own experiment tracking, metrics logging, and job monitoring for external visibility</li>
<li>Improve training system reliability, maintainability, and performance</li>
</ul>
<p>Much of the work will support large-scale pre-training, and pre-training experience is required. Strong infrastructure and systems experience is what we value most.</p>
</div>
<h2>Key Responsibilities</h2><div>
<ul>
<li>Distributed Framework Ownership – Extend or modify training frameworks (e.g., DeepSpeed, FSDP) to support new use cases and architectures.</li>
<li>Optimizer Implementation – Translate mathematical optimizer specs into distributed implementations.</li>
<li>Launch Config &amp; Debugging – Create and debug multi-node launch scripts with flexible batch sizes, parallelism strategies, and hardware targets. Select and validate data, tensor, pipeline, expert, and context parallelism strategies as appropriate for the model and cluster.</li>
<li>Metrics &amp; Monitoring – Build systems for experiment tracking, job monitoring, and logging usable by collaborators and researchers.</li>
<li>Infra Engineering – Write production-quality code and tests for ML infra in PyTorch or JAX; ensure reliability and maintainability at scale.</li>
<li>Data Loading &amp; Checkpointing – Build and maintain data-loading and checkpoint/restart workflows, restoring model, optimizer, RNG, and data progress after interruptions.</li>

</ul></div>
<h2>Qualifications</h2><div>
<h3>Must-Haves:</h3>
<ul>
<li>5+ years of experience in ML systems, infra, or distributed training</li>
<li>Experience modifying distributed ML frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod)</li>
<li>Strong software engineering fundamentals (Python, systems design, testing)</li>
<li>Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and debugging skills (e.g., NCCL/GLOO)</li>
<li>Ability to implement algorithms across GPUs/nodes based on mathematical specs</li>
<li>Experience working on an ML platform/ infrastructure, and/or distributed inference optimization team</li>
<li>Experience with large-scale machine learning workloads (strong ML fundamentals)</li>
<li>Experience with large-scale pre-training</li>

<h3>Nice-to-Haves:</h3>
<ul>
<li>Exposure to mixed-precision training (e.g., bf16, fp8) with accuracy validation</li>
<li>Familiarity with performance profiling, kernel fusion, or memory optimization</li>
<li>Open-source contributions or published research (MLSys, ICML, NeurIPS)</li>
<li>CUDA or Triton kernel experience</li>
<li>Experience building custom training pipelines at scale and modifying them for custom needs</li>
<li>Deep familiarity with training infrastructure and performance tuning</li>

</ul></ul></div>

Machine Learning Engineer — Pre-training (LLM)

<div><span>About the Institute of Foundation Models</span></div>
<div>We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.</div>
<div> </div>
<div><strong>As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. </strong>You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.</div>
<div> </div>
<div> </div><div><br /></div><div>
<h2>The Role</h2>
<p>The GPU Kernel Engineer will play a role at the forefront of optimizing performance for the machine learning software stacks, especially at training and inference, and support the team to develop new and cutting-edge systems. The ideal candidate will have a strong background in parallel computing, and hands-on experience in system level coding, debug methodologies, and large-scale machine learning experience.</p>
<p>This role focuses on CUDA kernel development and optimization. Distributed training experience is a plus.</p>
</div>
<h2>Key Responsibilities</h2><div>
<ul>
<li>Understand, analyze, profile, optimize, and provide guidance to the team on deep learning workloads on state-of-the-art hardware and software platforms to improve their efficiency with different levels of optimization</li>
<li>Design and implement performance benchmarks and testing methodologies to evaluate application performance</li>
<li>Build tools to automate workload analysis, workload optimization, and other critical workflows</li>
<li>Triage system issues and identify bottleneck and inefficiencies by analyzing the sources of issues and the impact on hardware, network and propose solutions to enhance GPU utilization</li>
<li>Support the team to develop appropriate kernels and systems for new model architectures and algorithms</li>
<li>Participate in, or lead design reviews with peers and stakeholders to decide amongst available technologies.</li>
<li>Review code developed by other developers and provide feedback to ensure best practices (e.g., style guidelines, checking code in, accuracy, testability, and efficiency).</li>
<li>Contribute to existing documentation or educational content and adapt content based on product/program updates and user feedback.</li>
<li>Represent MBZUAI at industry conferences and events, showcasing the institution’s cutting-edge HPC and deep learning capabilities and establishing MBZUAI as a global leader in AI research and innovation.</li>
<li>Perform all other duties as reasonably directed by the line manager that are commensurate with these functional objectives.</li>
<li>Validate CUDA kernel outputs and gradients against reference implementations, and benchmark representative shapes, dtypes, and model workloads.</li>

</ul></div>
<h2>Technical Qualifications</h2><div>
<h3>Must-Haves:</h3>
<ul>
<li>Strong C++ skills and hands-on CUDA kernel development and optimization for deep-learning workloads.</li>
<li>Understanding of GPU memory hierarchy, warp/block execution, and compute-memory trade-offs, with demonstrated profiling-driven optimization.</li>
<li>Strong Python skills and experience integrating kernels with PyTorch or an equivalent framework, including numerical and gradient validation where needed.</li>

<h3>Nice-to-Haves:</h3>
<ul>
<li>Experience with Triton, CUTLASS, or PTX/SASS analysis.</li>
<li>Experience with multi-node distributed training or inference systems.</li>
<li>Experience validating mixed-precision computations, such as BF16 or FP8.</li>

</ul></ul></div>

Machine Learning Engineer — GPU Kernel

<div>
<div>
<p><strong><span>About the Institute of Foundation Models</span></strong></p>
</div>
<p>The Institute of Foundation Models is a dedicated research lab focused on building, understanding, using, and risk-managing foundation models. Our mission is to advance AI research, support the next generation of AI builders, and develop impactful systems that improve how frontier models are trained, evaluated, deployed, and governed.</p>
<p>As part of our team, you will work closely with researchers, machine learning engineers, data scientists, software engineers, and product teams on some of the most important challenges in AI development. You will contribute to systems that help measure model quality, identify failure modes, and improve the reliability, safety, and readiness of model releases.</p>
</div>
<h2>The Role</h2><div>
<p>We are looking for an <strong>Eval360 - Error Analysis Engineer</strong> to help build, improve, and operate Eval360, an evaluation service that serves as a quality gate for AI models. This person will focus specifically on <strong>error analysis</strong>: understanding where models fail, why they fail, how those failures should be categorized, and how evaluation systems can better detect, measure, and prevent these issues before models are released.</p>
<p>You will collaborate with researchers, machine learning engineers, product managers, data scientists, and platform teams to develop AI evaluation applications and internal tools based on next-generation AI research. You will be part of a cross-functional team responsible for the full software development lifecycle, from requirements gathering and system design to implementation, deployment, monitoring, debugging, documentation, and continuous improvement.</p>
<p>The ideal candidate is comfortable working across the stack, including front-end interfaces for reviewing errors, back-end evaluation pipelines, data analysis workflows, model evaluation infrastructure, databases, dashboards, and APIs. This person should have strong software engineering skills, excellent analytical judgment, and the ability to turn ambiguous model failures into structured insights that improve evaluation quality.</p>
</div>
<h2>Key Responsibilities</h2><div>
<p><span>•</span>  <span>Collaborate with researchers, machine learning engineers, data scientists, product managers, and internal stakeholders to implement innovative software solutions for Eval360 and related model evaluation workflows.</span></p>
<p><span>•</span>  <span>Build and improve Eval360 as an evaluation service that acts as a quality gate for model development, model comparison, and model release decisions.</span></p>
<p><span>•</span>  <span>Perform deep error analysis on model outputs, including identifying failure patterns, categorizing issues, tracing root causes, and proposing improvements to evaluation methodology.</span></p>
<p><span>•</span>  <span>Develop tools, workflows, and dashboards that make it easier for researchers and engineers to inspect model failures, compare model behavior, and understand quality regressions.</span></p>
<p><span>•</span>  <span>Design and implement client-side and server-side architecture for evaluation review systems, error analysis interfaces, reporting tools, and internal evaluation applications.</span></p>
<p><span>•</span>  <span>Develop responsive, usable interfaces that support error triage, annotation review, evaluation debugging, and model quality investigation.</span></p>
<p><span>•</span>  <span>Build and maintain back-end services, APIs, data pipelines, and integrations that support evaluation execution, results storage, analysis, and reporting.</span></p>
<p><span>•</span>  <span>Test software to ensure responsiveness, correctness, reliability, and efficiency across evaluation workflows.</span></p>
<p><span>•</span>  <span>Troubleshoot, debug, and upgrade evaluation systems, including identifying issues in data processing, evaluation metrics, model output handling, job orchestration, and user-facing analysis tools.</span></p>
<p><span>•</span>  <span>Create and maintain security, access control, and data protection settings for evaluation data, model outputs, annotations, and internal tooling.</span></p>
<p><span>•</span>  <span>Write clear technical documentation for Eval360 systems, error taxonomies, evaluation workflows, debugging procedures, and user-facing tools.</span></p>
<p><span>•</span>  <span>Work with researchers, data scientists, analysts, and machine learning engineers to improve evaluation quality, model diagnostics, and failure-mode visibility.</span></p>
<p><span>•</span>  <span>Keep track of new development tools, evaluation frameworks, model analysis methods, data quality techniques, and architectures relevant to AI evaluation systems.</span></p>
<p><span>•</span>  <span>Contribute to the design of error taxonomies, evaluation rubrics, quality thresholds, regression detection methods, and model readiness criteria.</span></p>
<p><span>•</span>  <span>Help ensure Eval360 produces reliable, interpretable, and actionable signals for model quality gates.</span></p>
<p><span>•</span>  <span>Contribute to research publications, technical reports, internal knowledge sharing, and external presentations where appropriate.</span></p>
<p><span>•</span>  <span>Contribute to intellectual property and thought leadership in AI evaluation, error analysis, model quality measurement, and evaluation infrastructure.</span></p>
<p><span>•</span>  <span>Perform all other duties as reasonably directed by the line manager that are aligned with these functional objectives.</span></p>
</div>
<h2>Academic Qualifications</h2><div>
<p><span>•</span>  <span>Bachelor's degree in Computer Science, Machine Learning, Data Science, Software Engineering, Statistics, or a related technical field required.</span></p>
<p><span>•</span>  <span>Master's or Ph.D. in Computer Science, Machine Learning, Artificial Intelligence, Data Science, or a related field preferred.</span></p>
</div>
<h2>Professional Experience</h2><div>
<p><span>•</span>  <span>Proven experience as a Software Engineer, Full Stack Developer, Machine Learning Evaluation Engineer, Data Scientist, AI Engineer, or similar role.</span></p>
<p><span>•</span>  <span>Experience building software systems for AI, machine learning, data analysis, evaluation, annotation, experimentation, or model monitoring.</span></p>
<p><span>•</span>  <span>Experience working with AI algorithms and the ability to develop systems that accommodate AI-related requirements.</span></p>
<p><span>•</span>  <span>Experience performing error analysis, model evaluation, data quality analysis, or failure-mode investigation for machine learning or language model systems.</span></p>
<p><span>•</span>  <span>Experience developing internal applications, dashboards, review tools, or web-based workflows for technical users.</span></p>
<p><span>•</span>  <span>Familiarity with common software stacks, including front-end frameworks, back-end services, databases, APIs, and cloud or internal infrastructure.</span></p>
<p><span>•</span>  <span>Familiarity with GitHub, Git, CI/CD workflows, and collaborative software development practices.</span></p>
<p><span>•</span>  <span>Knowledge of front-end languages and libraries such as HTML, CSS, JavaScript, TypeScript, React, Angular, or similar technologies.</span></p>
<p><span>•</span>  <span>Knowledge of back-end languages and frameworks such as Python, Java, C#, Node.js, FastAPI, Flask, Django, or similar technologies.</span></p>
<p><span>•</span>  <span>Familiarity with databases such as MySQL, PostgreSQL, MongoDB, or other structured and unstructured data stores.</span></p>
<p><span>•</span>  <span>Familiarity with evaluation frameworks, experiment tracking systems, data pipelines, or machine learning infrastructure is strongly preferred.</span></p>
<p><span>•</span>  <span>Ability to analyze complex model outputs and translate qualitative failures into structured, measurable categories.</span></p>
<p><span>•</span>  <span>Strong problem-solving and troubleshooting skills, especially for ambiguous technical issues involving models, data, metrics, and software systems.</span></p>
<p><span>•</span>  <span>Effective communication and collaboration skills, with the ability to work across research, engineering, data, and product teams.</span></p>
<p><span>•</span>  <span>Strong attention to detail and a high bar for evaluation quality, reliability, and interpretability.</span></p>
</div>
<h2>Preferred Qualifications</h2><div>
<p><span>•</span>  <span>Experience with large language models, foundation models, multimodal models, or model evaluation systems.</span></p>
<p><span>•</span>  <span>Experience designing or using error taxonomies, evaluation rubrics, benchmark datasets, human evaluation workflows, or automated grading systems.</span></p>
<p><span>•</span>  <span>Experience with Python-based data analysis tools such as pandas, NumPy, Jupyter, or similar.</span></p>
<p><span>•</span>  <span>Experience with visualization or dashboarding tools for model quality analysis.</span></p>
<p><span>•</span>  <span>Experience with distributed systems, job queues, workflow orchestration, or large-scale data processing.</span></p>
<p><span>•</span>  <span>Experience working in a research environment or with fast-moving AI product and model teams.</span></p>
</div>

Eval360 - Error Analysis Engineer

<div><span>About the Institute of Foundation Models</span></div>
<div>We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.</div>
<div> </div>
<div><strong>As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. </strong>You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers.</div>
<div> </div>
<div><strong>Key Responsibilities </strong></div>
<div>This <strong>intensive, short-term internship</strong> offers a unique opportunity for aspiring AI engineers to;</div>
<div>• Collect, prepare and processing training and validation datasets.</div>
<div>• Assist in designing and tweaking model architectures.</div>
<div>• Implement testing frameworks to assess model performance.</div>
<div>• Analyze model outputs for accuracy and bias.</div>
<div>• Participate in the review and refinement of AI algorithms.</div>
<div>• Document findings and procedures for team collaboration.</div>
<div> </div>
<div><strong>Academic Qualifications</strong></div>
<div>Currently pursuing or recently completed a Master's, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, Applied Mathematics, Statistics, or a related quantitative field.</div>

Ifm Us

Open jobs at Ifm Us

Machine Learning Engineer — Reinforcement Learning

Machine Learning Engineer — Pre-training (LLM)

Machine Learning Engineer — GPU Kernel

Eval360 - Error Analysis Engineer

AI Research Internship - WM

Working for Ifm Us?

Featured companies