SMCalFlow

One central challenge in conversational AI is the design of a dialogue state representation that agents can use to reason about the information and actions available to them. We’ve developed a new representational framework for dialogue that enables efficient machine learning of complex conversations.

SMCalFlow is a large English-language dialogue dataset, featuring natural conversations about tasks involving calendars, weather, places, and people. Each turn is annotated with an executable dataflow program featuring API calls, function composition, and complex constraints built from strings, numbers, dates and times. For more information about dataflow-based dialogue agents and details about the dataset, check out our blog post and paper:

The code and links to trained models are available on github:

To participate in the leaderboard, download a copy of the SMCalFlow dataset:

• Data

The dataset is distributed under theCC BY-SA 4.0 license.

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. We will not access your code or model for any purpose other than to evaluate your model on SMCalFlow. Here's a tutorial walking you through official evaluation of your model:

Email us at sathomso@microsoft.com and subhrroy@microsoft.com.