AI Benchmarking Lead, Performance Benchmarking Evaluation

Technology, Data & Digital · Data, AI & Analytics · Data Science · Machine Learning · Data Analysis

In short

As an AI Benchmarking Lead in Hyderabad, you will ensure the reliability and accuracy of AI model evaluations for Amazon's Seller Assistant. Your role involves quality checking audits, improving auditor consistency, and driving continuous improvement in audit processes and methodologies.

Responsibilities

Benchmark Seller Assistant AI models for relevancy, correctness, and completeness.
Evaluate audits performed by the core auditing team to increase confidence in evaluation metrics.
Improve audit reliability and consistency through systematic measurement of auditor accuracy.
Conduct targeted calibration to ensure quality standards across the auditing function.
Enforce quality standards by quality-checking audits and providing actionable feedback.
Drive continuous improvement in audit processes and methodologies.
Surface high-confidence product issues earlier by validating and categorizing model failures.
Serve as point of contact for annotation tasks, ensuring quality execution and delivery.
Understand dependencies across ML data workflows and articulate customer impact.
Modify existing annotation methods and update SOPs.
Document SOP changes, secure approval, and share knowledge.
Test new SOPs and tools, providing feedback.
Structure data collection, analyze results, and share inputs for SOP changes.
Collate, track, and report progress on key metrics.
Identify operational issues and recommend suggestions to improve productivity and quality.

Requirements

Bachelor's degree or equivalent in a related field.
Experience in natural language data labeling, data annotation, linguistic annotation, or other forms of data markup.
Proficiency in MS Excel.
Basic understanding of SQL and Python.
Experience with Microsoft Office products and applications.
Strong verbal and written communication skills in English.
Knowledge about SOA and processes that deal with sellers.

Desired Qualifications

1 to 3 years of equivalent experience.
Performed annotation related tasks across ML data process areas.
Strong knowledge of process documentation and analysis.
Technical proficiency in SQL querying and Python programming for data analysis.
Strong analytical and problem-solving skills.
Ability to work independently and as part of a team.

Benefits

Inclusive culture that empowers Amazonians.
Support for individuals with disabilities for workplace accommodations.

Skills

MS ExcelSQLPython

#AI#Benchmarking#Performance#Evaluation#Seller Assistant#Gen-AI#Copilot#Data Annotation#Linguistic Annotation#ML Data#Quality Assurance#Process Improvement#SQL#Python

About Amazon

Earth's most customer-centric company

Amazon is guided by four principles: customer obsession rather than competitor focus, passion for invention, commitment to operational excellence, and long-term thinking. We are driven by the excitement of building technologies, inventing products, and providing services that change lives. We embrace new ways of doing things, make decisions quickly, and are not afraid to fail. We have the scope and capabilities of a large company, and the spirit and heart of a small one.\n\nTogether, Amazonians research and develop new technologies from Amazon Web Services to Alexa on behalf of our customers:…

See all open jobs at Amazon Read more about Amazon Website