About the Team
The AI Model Serving team powers every production Workday agent and machine learning use case. We run services that handle all production AI workloads, acting as the gateway to vendor-hosted LLMs on GCP and AWS Bedrock and operating the model deployment platform that hosts and scales Workday’s models. We host thousands of traditional ML models across sharded Ray Serve clusters, manage approximately 2,000 requests per second with peaks over 10,000 requests per second, and provide a uniform interface for accessing models on Bedrock and Gemini. In the coming year we’ll enhance our vendor interface, scale our architecture for 20+ agents in production, improve cost governance, explore hosting open‑source LLMs, evaluate a potential rewrite for performance gains, and keep pace with the rapidly evolving AI landscape.
About the Role
As a Principal Software Development Engineer on the AI Model Serving team, you will be a technical leader shaping the platform’s vision and direction alongside the engineering manager. You will make critical design decisions, drive outcomes across the team, and cultivate a positive, inclusive culture. In addition to the model serving platform, the team owns the production model registry; you will guide its evolution to meet the needs of ML teams across the organization.
Key Responsibilities
- Help set the product vision for the AI Model Serving platform in partnership with the engineering manager.
- Lead the team technically by making decisions that drive performance, reliability, and scalability.
- Design, implement, and maintain large‑scale systems to move ML models to production.
- Write design documents to build consensus for new components and enhancements.
- Evaluate and adopt new technologies from Workday and the broader industry.
- Troubleshoot the continuous integration software pipelines and improve them.
- Build relationships with software engineers, machine learning engineers, and data scientists on partner teams.
- Respond to alerts and debug production issues to maintain platform health.
- Review pull requests and enforce consistency, performance, readability, and security across code bases.
- Develop documentation to share knowledge with other engineers.
Basic Qualifications
- 8+ years of software development experience focused on large‑scale distributed systems.
- Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
Other Qualifications
- Deep experience designing, building, and scaling production‑grade distributed systems.
- Proficiency in Python with extensive production‑level experience.
- Familiarity with large language models and traditional machine learning models, including serving, scaling, and monitoring in production.
- Ability to design and maintain monitoring strategies for system health, performance, and cost.
- Excellent written and verbal communication skills with the ability to write clear design documents and articulate complex technical ideas.
- Collaborative spirit with mentoring experience and a commitment to an inclusive team environment.
Pay Transparency Statement
The annualized base salary ranges for the primary location and any additional locations are listed below. Workday pay ranges vary by work location. The role may also be eligible for a bonus plan, commission/bonus, and annual stock grants. Each candidate’s compensation will be based on geography, experience, skills, duties, and business need.
Primary Location: CAN.ON.Toronto
Primary CAN Base Pay Range: $168,000 – $252,000 CAD
Additional CAN Locations: $168,000 – $252,000 CAD
Our Approach to Flexible Work
We require spending at least half (50%) of each quarter in the office or in the field. Remote “home office” roles can still come together in our offices for important moments. Flexibility is designed to support your business, team, and personal needs while fostering intentional in‑person collaboration.
Equal Opportunity Employer
Workday is an Equal Opportunity Employer including individuals with disabilities and protected veterans. Pursuant to applicable Fair Chance law, we consider qualified applicants with arrest and conviction records. We are committed to an accessible and inclusive hiring experience; if you require accommodation, contact
#J-18808-Ljbffr
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
