AI Chatbot Discovery

Do users actually want an AI Chatbot?

My Role I led UX research for the discovery of AI Chatbot as part of a continuous discovery practice embedded in an agile cross-functional team. I owned research planning, method selection, participant recruitment, and synthesis across multiple research phases.

Background The team came in with a solution already named: we are adding an AI Chatbot. No one had asked whether users wanted one. No one had asked what the chatbot should do. Leadership and business pressure had framed the chatbot as the answer before the question had been defined. The risk of moving forward without validation was significant. Shipping an unvalidated feature onto an already unstable platform meant wasted engineering investment, eroded user trust, and a feature no one would use.

Research reframed the directive into a question: Do users think a chatbot would be helpful, and if so, what do they need it to accomplish?

Platform The tool at the center of this study manages app and group access for a large enterprise organization. Users rely on it for membership management, entitlement updates, and support across compliance-sensitive workflows. The experience had a known problem: basic tasks required submitting help tickets to complete. The process was slow, confusing, and frequently unstable.

Method Survey, The team needed breadth across user roles, not depth from a small sample. A survey was the fastest path to a defensible signal under the timeline and stakeholder pressure in play.

Recruitment Participants were drawn directly from activity logs from the platform, targeting users who had interacted with it in the past 90 days. Organizational data on job role and business unit was pulled in advance, keeping the survey focused on behavior and perception rather than demographics.

Response Rate 300 users were contacted. 34 responded, a roughly 10% response rate, which is a strong result for a targeted enterprise recruitment.

Insights Users were not asking for innovation. They were asking for restoration. The prior system had capabilities the current platform removed. The chatbot concept was well received precisely because users hoped it would fill those gaps. An AI Chatbot that returns undifferentiated support article links does not solve that problem. It repackages it. This reframed what a successful AI chatbot would actually need to do.

Impact Research changed what is being built. The AI chatbot was scoped for contextual assistance rather than generic link-surfacing. Stability issues surfaced in the study were documented and added to the product backlog as tracked risks. The launch timeline was set to FY27 to allow for a properly validated, user-informed implementation. Proof of concept usability testing was scoped as the next phase of research before any launch decision.

The project did not end with a shipped feature. It ended with a more deliberate product decision, which is the right outcome for a study that started with a directive and ended with a question answered.

If you’re interested in learning more about my research on the AI Chatbot, please contact zmarrich@gmail.com to schedule a detailed review of the case study.