Posted Today
Position Summary:
Seeking a QA AI Engineer to perform testing and quality assurance for systems, applications, and AI-enabled products developed by Xpansiv. This role is responsible for ensuring the quality, reliability, accuracy, and safety of Xpansiv’s AI-driven products and proprietary LLM infrastructure. The QA AI Engineer will create test plans, document and execute test cases, build automated and semi-automated evaluation frameworks, and validate both deterministic software behavior and non-deterministic AI outputs. This role works closely with AI Engineering, Product, Operations, Engineering, business analysts, business owners, and subject matter experts to build quality into AI products from the start and provide the human-in-the-loop assurance required by Xpansiv’s AI governance standards.
Key Responsibilities:
Create test plans for AI-powered and traditional software applications, including manual testing, automated testing, performance testing, regression testing, security testing, and end-to-end testing
Formulate and document test cases based on product requirements, user stories, acceptance criteria, AI governance standards, and business workflow expectations
Execute test cases through targeted manual testing, automated testing, exploratory testing, and AI-specific evaluation methods
Design, build, and maintain evaluation pipelines for AI-powered applications across Xpansiv business lines
Develop evaluation datasets, golden sets, and scenario suites that measure accuracy, consistency, structured-output quality, policy adherence, and business-rule compliance of LLM outputs
Detect, document, and reproduce AI-specific failure modes, including hallucinations, prompt injection, inconsistent outputs, formatting errors, data leakage, bias, unsafe responses, and model or prompt-update regressions
Build automated and semi-automated AI evaluation frameworks, including model-graded assertions, regression harnesses, prompt test suites, and quality scorecards, alongside traditional QA automation
Validate microservices, APIs, data extraction workflows, document-processing pipelines, RAG-based systems, agents, and structured-output generation for business-critical use cases
Perform functional, end-to-end, cross-browser, regression, security, performance, API, and integration testing as needed for AI-enabled and non-AI system components
Own quality gates and go/no-go readiness criteria for pilots, beta launches, production go-lives, and post-release model or prompt updates
Establish and track quality KPIs, including test coverage, pass rates, defect density, escaped-defect rate, AI accuracy metrics, hallucination rate, evaluation-score trends, and release readiness
Partner with AI Engineering to embed testing, monitoring, observability, evaluation, and quality controls into the AI development lifecycle
Conduct safety, bias, adversarial, and red-team testing to support responsible and compliant AI behavior aligned to Xpansiv’s AI Usage Standard
Work with developers, product managers, business analysts, business owners, and subject matter experts to review identified defects, provide clarifications, validate fixes, discuss solutions, and continuously improve AI accuracy and reliability
Support human-in-the-loop review requirements for high-risk client, financial, regulatory, and operational workflows
Qualifications:
10+ years of experience in software quality assurance, including both manual and automated testing
Experience creating test plans, documenting test cases, executing test cases, validating defects, and supporting release-readiness decisions across multiple projects
Experience with functional testing, end-to-end testing, cross-browser testing, regression testing, security testing, performance testing, API testing, and integration testing
Experience testing or evaluating AI, ML, or LLM-powered systems, such as chatbots, copilots, agents, classifiers, RAG applications, document-processing workflows, or decision-support tools
Familiarity with generative AI evaluation methods, including golden datasets, model-graded evaluations, prompt regression testing, hallucination detection, red-teaming, and quality-drift monitoring
Working knowledge of prompt engineering, Retrieval Augmented Generation, agent-based systems, structured outputs, and AI guardrails sufficient to test these systems effectively
Experience with automated software testing tools such as Cypress, Playwright, Selenium, Rest Assured, or similar frameworks
Experience testing microservices, APIs, and web services using tools such as Postman, SoapUI, Rest Assured, or similar API testing tools
Proficiency in at least one scripting or programming language used for test automation, data validation, or evaluation tooling
Experience with source version control tools such as Git
Experience working in agile software development process models
Experience interacting with developers, product managers, business analysts, business owners, and subject matter experts to review and validate test plans, test cases, test results, and defects
Understanding of AI governance, responsible AI principles, data-handling guardrails, privacy considerations, and human-in-the-loop review practices
Skills / Abilities:
Strong communication and collaboration skills with clients, developers, business analysts, product managers, business owners, management, and cross-functional stakeholders
Strong analytical, troubleshooting, and failure-mode thinking, with the ability to translate ambiguous requirements and AI behavior into concrete test scenarios
Exceptional attention to detail, time management, and documentation quality
Proven ability to work independently as a self-starter and collaboratively as part of a cross-functional team
Ability to quickly absorb business and technical concepts, including unfamiliar AI workflows, data flows, and domain-specific business rules
Proven ability to work calmly under tight deadlines, production-readiness pressure, or critical quality situations
Curiosity and sound judgment when evaluating emerging AI capabilities, limitations, risks, and quality tradeoffs
Position Summary:
Seeking a QA AI Engineer to perform testing and quality assurance for systems, applications, and AI-enabled products developed by Xpansiv. This role is responsible for ensuring the quality, reliability, accuracy, and safety of Xpansiv’s AI-driven products and proprietary LLM infrastructure. The QA AI Engineer will create test plans, document and execute test cases, build automated and semi-automated evaluation frameworks, and validate both deterministic software behavior and non-deterministic AI outputs. This role works closely with AI Engineering, Product, Operations, Engineering, business analysts, business owners, and subject matter experts to build quality into AI products from the start and provide the human-in-the-loop assurance required by Xpansiv’s AI governance standards.
Key Responsibilities:
Create test plans for AI-powered and traditional software applications, including manual testing, automated testing, performance testing, regression testing, security testing, and end-to-end testing
Formulate and document test cases based on product requirements, user stories, acceptance criteria, AI governance standards, and business workflow expectations
Execute test cases through targeted manual testing, automated testing, exploratory testing, and AI-specific evaluation methods
Design, build, and maintain evaluation pipelines for AI-powered applications across Xpansiv business lines
Develop evaluation datasets, golden sets, and scenario suites that measure accuracy, consistency, structured-output quality, policy adherence, and business-rule compliance of LLM outputs
Detect, document, and reproduce AI-specific failure modes, including hallucinations, prompt injection, inconsistent outputs, formatting errors, data leakage, bias, unsafe responses, and model or prompt-update regressions
Build automated and semi-automated AI evaluation frameworks, including model-graded assertions, regression harnesses, prompt test suites, and quality scorecards, alongside traditional QA automation
Validate microservices, APIs, data extraction workflows, document-processing pipelines, RAG-based systems, agents, and structured-output generation for business-critical use cases
Perform functional, end-to-end, cross-browser, regression, security, performance, API, and integration testing as needed for AI-enabled and non-AI system components
Own quality gates and go/no-go readiness criteria for pilots, beta launches, production go-lives, and post-release model or prompt updates
Establish and track quality KPIs, including test coverage, pass rates, defect density, escaped-defect rate, AI accuracy metrics, hallucination rate, evaluation-score trends, and release readiness
Partner with AI Engineering to embed testing, monitoring, observability, evaluation, and quality controls into the AI development lifecycle
Conduct safety, bias, adversarial, and red-team testing to support responsible and compliant AI behavior aligned to Xpansiv’s AI Usage Standard
Work with developers, product managers, business analysts, business owners, and subject matter experts to review identified defects, provide clarifications, validate fixes, discuss solutions, and continuously improve AI accuracy and reliability
Support human-in-the-loop review requirements for high-risk client, financial, regulatory, and operational workflows
Qualifications:
10+ years of experience in software quality assurance, including both manual and automated testing
Experience creating test plans, documenting test cases, executing test cases, validating defects, and supporting release-readiness decisions across multiple projects
Experience with functional testing, end-to-end testing, cross-browser testing, regression testing, security testing, performance testing, API testing, and integration testing
Experience testing or evaluating AI, ML, or LLM-powered systems, such as chatbots, copilots, agents, classifiers, RAG applications, document-processing workflows, or decision-support tools
Familiarity with generative AI evaluation methods, including golden datasets, model-graded evaluations, prompt regression testing, hallucination detection, red-teaming, and quality-drift monitoring
Working knowledge of prompt engineering, Retrieval Augmented Generation, agent-based systems, structured outputs, and AI guardrails sufficient to test these systems effectively
Experience with automated software testing tools such as Cypress, Playwright, Selenium, Rest Assured, or similar frameworks
Experience testing microservices, APIs, and web services using tools such as Postman, SoapUI, Rest Assured, or similar API testing tools
Proficiency in at least one scripting or programming language used for test automation, data validation, or evaluation tooling
Experience with source version control tools such as Git
Experience working in agile software development process models
Experience interacting with developers, product managers, business analysts, business owners, and subject matter experts to review and validate test plans, test cases, test results, and defects
Understanding of AI governance, responsible AI principles, data-handling guardrails, privacy considerations, and human-in-the-loop review practices
Skills / Abilities:
Strong communication and collaboration skills with clients, developers, business analysts, product managers, business owners, management, and cross-functional stakeholders
Strong analytical, troubleshooting, and failure-mode thinking, with the ability to translate ambiguous requirements and AI behavior into concrete test scenarios
Exceptional attention to detail, time management, and documentation quality
Proven ability to work independently as a self-starter and collaboratively as part of a cross-functional team
Ability to quickly absorb business and technical concepts, including unfamiliar AI workflows, data flows, and domain-specific business rules
Proven ability to work calmly under tight deadlines, production-readiness pressure, or critical quality situations
Curiosity and sound judgment when evaluating emerging AI capabilities, limitations, risks, and quality tradeoffs
