AI Chat Logs Research

Large Language Model-based AI is transforming how users interact with assistive systems. Logs of user interactions from new chat and copilot AI systems provide more extensive signals of user satisfaction, success, and enjoyment than conventional search and recommendation logs due to the rich use of natural language in the interactions. Moreover, the LLMs themselves can be leveraged to extract the signals effectively from the natural language in the logs. This combination provides a unique opportunity to utilize the chat logs to quickly understand user intent, assess satisfaction, and characterize usage to drive continuous improvements in the chat and copilot systems over time.