Testlio - AI Solutions Architect (United States)

AI Developer is needed in Indiana, United States.

Client: Testlio

Location: Indiana

Contract: Freelance

Location: This is a 100% remote role open to those who reside in the United States.

Job Description

Testlio’s fully managed crowdsourced testing platform integrates expert, on-demand testers directly into your release process. We are hiring a visionary and hands-on AI Solutions Architect to own the definition, packaging, technical design, and delivery methodology for our AI testing offerings. This role combines deep practitioner expertise with a strong business perspective, designing testing approaches for complex AI-powered applications, advancing Testlio’s proprietary framework, and collaborating with sales and delivery to drive our AI business forward.

Requirements

Technical Skills

• Hands-on Data Science Background: Extensive practical experience in machine learning models, NLP structures, or data pipelines.
• AI Evaluation Expertise: Familiarity with modern LLM evaluation frameworks and metric-driven validation ecosystems.
• Agentic Framework Proficiency: Experience with autonomous agent architectures or multi-agent swarms.
• Strong Software Engineering Foundations: Proficiency in complex non-deterministic and deterministic systems.

Human Skills

• Proven Client-Facing Experience: Track record of leading discovery calls and managing stakeholder workshops.
• Commercial & Strategic Acumen: Experience in designing service offerings and understanding pricing dynamics.
• Consultative Communication: Ability to bridge technical data science outputs with strategic customer business value.
• Comfort with Ambiguity: Adaptability to navigate fast-evolving client layouts and AI application lifecycles.
• Mentorship Mindset: Passion for continuous learning and sharing best practices with internal teams.

Other Information

Why will you love this job?

• Evaluate at a Scale That Exists NowElse: Design evaluation systems with thousands of experts globally.
• Innovate at the Frontier of QA: Shape industry standards for validating AI.
• High-Impact Consultative Visibility: Collaborate with global market leaders.
• Decompose Complex Ecosystems: Turn unknowns into structured testing criteria.
• Drive company influence: Assist in pipeline and business decisions regarding AI Testing.

Why will you love being a part of Testlio?

• Great Culture: Inclusive, purpose-driven culture with a focus on collaboration.
• Remote Work: Fully remote environment with effective asynchronous working systems.
• Investment in You: Flexible paid time off and support for personal and professional development.
• Winning Business: Growing, profitable company with long-term client relationships.

What would your day look like?

Strategic & Commercial Responsibilities

• Define and Maintain the AI Service Catalog: Offerings, packaging, delivery methodology, and key metric tracking.
• Vision Activation & Marketing: Produce content and activate viewpoints on AI Testing.
• Sales Collaboration: Support pre-deal discovery and articulate Testlio’s AI testing value proposition.
• Offering Lifecycle Management: Refine offerings using market and competitive intelligence.

Collaboration on Hands-on Delivery

• System & Architectural Deconstruction: Execute system breakdowns for tailored AI testing approaches.
• Delivery Expertise: Serve as a subject-matter expert during complex engagements.
• Technical Problem Solving: Address technical roadblocks and work towards scalable solutions.
• Framework Advancement: Improve Testlio’s AI testing framework.
• Empower Internal Teams: Develop training datasets and documentation for quality test execution.

What is the application process?

The application process includes application submission, a recruiter interview, a TestGorilla assessment, approximately four team and stakeholder interviews, reference checks, and an offer with a background check.

Diversity and Inclusion

Testlio is an equal-opportunity employer committed to an inclusive environment. We are female-founded and have a diverse team. More details can be found on our website’s DEI section.

#LI-Remote