Senior Product Quality Analyst
Technology, Data & Digital · Data, AI & Analytics · Data Science · Artificial Intelligence · Business Intelligence
In short
Spotify is seeking a Senior Product Quality Analyst to define and measure the quality of AI voice models. This role involves designing evaluation rubrics, conducting evaluation campaigns, and benchmarking against competitors to ensure Spotify's AI voices meet specific product needs and user expectations across various languages and regions.
Responsibilities
- Own the quality bar for Spotify’s AI voice models, defining what “good enough” means for different products, audiences, languages and use cases.
- Independently design evaluation rubrics and build an evaluation approach that measures qualities such as naturalness, tone, pacing, pronunciation, accent, audio quality and overall experience.
- Plan and carry out evaluation campaigns, turning qualitative and quantitative findings into clear recommendations for Voice Production, Speak and product teams.
- Establish an ongoing evaluation rhythm, including when and how we reassess model quality as our technology and products evolve.
- Benchmark Spotify’s AI voices against third-party models and competitors to understand where our experiences lead, where they differ and where there are opportunities to improve.
- Help determine appropriate quality trade-offs as Spotify expands into additional languages and regions, considering the product use case and needs of each market.
- Partner closely with voice experts, machine learning teams, product partners and vendors to maintain a consistent quality bar across demanding and sometimes competing priorities.
- Surface the most meaningful gaps in model performance and use evaluation insights to help influence future research and technical priorities.
Requirements
- Experience in product quality, evaluation, data curation or a closely related field where human judgment is used to assess model or product performance.
- Ability to independently define evaluation criteria and turn subjective qualities into clear, repeatable rubrics.
- Comfort designing and running both qualitative and MOS-style evaluations and synthesizing the results into actionable insights.
- Understanding of voice or audio quality, including qualities such as naturalness, pacing, tone, accent and pronunciation, or the ability to develop that expertise quickly.
- Strong judgment and ability to maintain a consistent quality bar while balancing different product requirements, markets and stakeholder perspectives.
- Comfort working autonomously, identifying what needs to be evaluated and establishing the processes and cadence to make that evaluation useful over time.
- Clear communication across creative, technical and product teams and ability to act as a reliable point of contact for internal partners and external vendors.
- Curiosity about generative AI, speech technology and the evolving AI voice landscape and ability to translate changes in the industry into meaningful benchmarks for Spotify.
Benefits
- The United States base range for this position is $114,486–$163,552 USD, plus equity.
- Health insurance
- Six-month paid parental leave
- 401(k) retirement plan
- Monthly meal allowance
- 23 paid days off
- Paid flexible holidays
- Paid sick leave
- Extensive learning opportunities, through our dedicated team, GreenHouse.
- Flexible share incentives letting you choose how you share in our success.
- Global parental leave, six months off - for all new parents.
- All The Feels, our employee assistance program and self-care hub.
- Flexible public holidays, swap days off according to your values and beliefs.
#AI Voice#Product Quality#Evaluation#Data Curation#Voice Production#AI DJ#Generative AI Ads#Personalized Podcasts#Agent Voice Chat
Company
Spotify
Job Posted
1 month ago
Employment Type
Full Time
WorkMode
Hybrid
Experience Level
Senior
Locations
New York, United States
Los Angeles, United States
Applicants
Be an early applicant