Expedia AI Chief: Evaluations Are the New Product Roadmap, Guardrails a 'Necessary Evil'

At the VB Transform 2026 conference in Menlo Park, Expedia Group's first chief AI and data officer, Xavi Amatriain, said that product requirements are now being replaced by evaluations. He explained that teams should define what a product should do through evaluations, including security tests, before writing code. With AI-generated code, the focus shifts entirely to designing those evaluations. Amatriain warned that too many guardrails can harm a system. They make it rigid and distort user feedback, leading to incorrect learning. He called guardrails a necessary evil that should be minimized over time. Instead, Expedia uses three layers of AI governance: broad principles, processes and tools to enforce them, and automation on top. Risk-based 'toll gates' tie evaluation rounds and security reviews to each agent's risk level. He advocated for specialized agents over a single AI model. Expedia's architecture breaks tasks into tools, skills, sub-agents, and orchestrators. This makes the system easier to secure. For travel bookings, the AI recommends but never books—the user must make the final click. This design eliminates the need for many guardrails. Security must be built in from the start, Amatriain said. He noted that future threats will come from other AI systems. Monitoring feedback loops help improve evaluations quickly. VentureBeat research shows 66% of companies plan to allow AI deployments without human review, but only 5% trust automated evaluations. Half have seen agents pass internal tests but fail with real customers.