AI Conversation Evaluation & Context Research Specialist

About the role

Gramian Consultancy, a boutique consultancy in IT professional services and engineering talent solutions, is looking for experienced Subject Matter Experts (SMEs) from psychology, social work, counseling, qualitative research, human-computer interaction, communication, conflict resolution, coaching and related advisory fields. You will review conversations between people and AI models, check whether the AI gathered enough context before giving advice, spot missed opportunities for clarification and give structured feedback to improve conversational quality.

Key responsibilities

• Review AI conversations covering interpersonal relationships, workplace situations, education, family matters, everyday planning, wellbeing and other real-world decisions.
• Evaluate whether the AI gathered sufficient context before offering advice, recommendations or proposed solutions.
• Identify missing information and suggest relevant follow-up questions that would improve the AI's response.
• Assess whether additional context would materially change the recommendation or decision, distinguishing load-bearing information from non-essential detail.
• Conduct qualitative reviews using structured evaluation guidelines and document findings consistently.
• Consider personal, psychological, social, financial, practical and organizational factors when evaluating conversations.
• Collaborate with research teams to keep evaluation standards consistent and complete tasks on time.

Required profile

• At least 3-5 years of direct professional experience in one of four pathways:
• Social work: MSW or equivalent, plus LCSW, LICSW, LMSW or equivalent, with experience in client assessment, case review, interviewing or counseling.
• Counseling, therapy or psychology: relevant master's degree with LMFT, LPC, LCPC or equivalent, or PhD/PsyD in clinical, counseling, social or I-O psychology, with client-facing experience.
• Qualitative research, HCI or communication: MA, PhD or equivalent in HCI, communication, anthropology, sociology, qualitative research or a close field, with experience in qualitative research, interviews, conversation analysis or thematic analysis.
• Mediation, coaching, student services, organizational consulting or similar: relevant qualification or demonstrated expertise, with at least 5 years of advisory or client-facing experience.
• Strong analytical and interviewing skills, and the ability to tell essential, decision-changing information from less relevant detail.