Job Opportunities

Job Description

Job Title: AI Model Evaluator (LLM & Agent Systems)

Job Type: Contract (Minimum 2 weeks, with potential extension)

Location: Remote

Job Summary:

Join our customer’s team as an AI Model Evaluator (LLM & Agent Systems) and play a pivotal role in shaping the future of generative AI and autonomous agents. You’ll help benchmark, analyze, and assess cutting-edge AI systems in real-world scenarios, providing structured insights that drive improvements. This position is ideal for analytical professionals passionate about AI quality and real-world impact.

Key Responsibilities:

  1. Evaluate outputs from large language models (LLMs) and autonomous agent systems against defined guidelines and rubrics
  2. Review multi-step agent actions, including screenshots and reasoning traces, to determine accuracy and quality
  3. Consistently apply evaluation standards, flagging edge cases and identifying recurring patterns or failure modes
  4. Provide detailed, structured feedback to inform benchmarking, product evolution, and model refinement
  5. Participate in calibration and alignment sessions to ensure consistent application of evaluation criteria
  6. Work collaboratively to adapt to evolving scenarios and ambiguous evaluation situations
  7. Document findings and communicate insights clearly both in writing and verbally to relevant stakeholders

Required Skills and Qualifications:

  1. Demonstrated experience with LLM evaluation, AI output analysis, QA/testing, UX research, or similar analytical roles
  2. Strong background in AI model evaluation, benchmarking, and applying rubric-based scoring frameworks
  3. Exceptional attention to detail and sound judgement in ambiguous or edge-case scenarios
  4. Proficiency in English (B2+ or equivalent) with excellent written and verbal communication skills
  5. Ability to adapt quickly to evolving guidelines and work independently
  6. Comfort with remote work and a commitment of at least 20 hours per week for the initial term
  7. Analytical mindset with a focus on actionable, qualitative feedback

Preferred Qualifications:

  1. Experience with RLHF, annotation workflows, or AI benchmarking frameworks
  2. Familiarity with autonomous agent systems or workflow automation tools
  3. Background in mobile apps or digital product evaluation processes

Apply Here

Job Description

Job Title: Product Validation Expert

Job Type: Contract (Full-time or Part-time)

Location: Remote

Job Summary:

Join our customer’s team as a Product Validation Expert and play a key role in ensuring that a widely used AI-powered consumer product deliver safe, accurate, and helpful experiences. You will design and implement evaluation processes to understand user perceptions.

Key Responsibilities:

  1. Design and develop robust rubrics for evaluating AI model outputs from a user experience (UX) perspective.
  2. Receive, analyze, and execute evaluation tasks focused on identifying potential issues that could lead to a poor user experience of this AI product.
  3. Collaborate closely with cross-functional teams to define and refine product validation criteria and safety standards.
  4. Translate user insights into actionable feedback for model improvement, focusing on accuracy, safety, and helpfulness.
  5. Document findings and recommendations with clarity, ensuring stakeholders understand the impact on end-user satisfaction.
  6. Continuously monitor evolving AI product usage and proactively identify emerging risks or concerns from a UX point of view.

Required Skills and Qualifications:

  1. Exceptional written and verbal communication skills, with a strong emphasis on clarity and precision.
  2. Ability to develop and apply evaluative frameworks that balance user experience, safety, and product performance.
  3. Strong analytical mindset and attention to detail when assessing user risks or usability issues.
  4. Comfort collaborating with diverse teams and adapting to fast-changing project requirements.
  5. Proficiency in documenting complex findings in a user-friendly and actionable manner.
  6. Ability to commit at least 15 hours per week to sharing experiences and tasks.

Preferred Qualifications:

  1. Familiarity with current widely used AI consumer products and their typical user journeys.

Apply Here

Job Description

Job Title: Hindi Bilingual Expert

Job Type: Contract (Full-time or Part-time)

Location: Remote

Job Summary

Join our customer’s team as a Hindi Bilingual Expert, where your linguistic expertise will drive impactful language solutions. You will play a critical role in transcribing, analyzing, and enhancing content, leveraging your advanced proficiency in both Hindi and English. Bring your passion for languages and attention to detail to an environment that values communication excellence and cultural nuance.

Key Responsibilities

  1. Transcribe video content accurately in Hindi, annotating and outlining precise timestamps.
  2. Describe and analyze speaker emotionality, tone, and intent in both written and spoken communications.
  3. Provide comprehensive grammatical and conceptual analysis of Hindi language usage.
  4. Define complex words and phrases, offering clear conceptual explanations for language tasks.
  5. Create detailed language reports and summaries to support project objectives and customer needs.
  6. Collaborate with a cross-functional team to ensure linguistic consistency and high-quality output.
  7. Commit to a minimum of 15 hours per week, delivering timely and insightful results.

Required Skills and Qualifications

  1. Native or near-native fluency in both Hindi and English, with a strong command of grammar and syntax.
  2. Exceptional written and verbal communication skills, with a keen eye for detail.
  3. Proven experience with transcription, annotation, or linguistic analysis tasks.
  4. Ability to identify and articulate emotional tone and intent in multimedia content.
  5. Advanced analytical skills, with the ability to deconstruct language at a conceptual level.
  6. Self-motivated, reliable, and able to manage time effectively in a remote work setting.
  7. Comfortable working with complex language tasks and adapting to evolving project requirements.

Preferred Qualifications

  1. Background in linguistics, translation, or language education.
  2. Experience with cross-linguistic analysis and cultural context evaluation.
  3. Familiarity with digital transcription and annotation tools.

Apply Here