CoQA

A Conversational Question Answering Challenge

What is CoQA?

CoQA is a large-scale dataset for building Conversational Question Answering systems. The goal of the CoQA challenge is to measure the ability of machines to understand a text passage and answer a series of interconnected questions that appear in a conversation. CoQA is pronounced as coca .

CoQA paper

CoQA contains 127,000+ questions with answers collected from 8000+ conversations. Each conversation is collected by pairing two crowdworkers to chat about a passage in the form of questions and answers. The unique features of CoQA include 1) the questions are conversational; 2) the answers can be free-form text; 3) each answer also comes with an evidence subsequence highlighted in the passage; and 4) the passages are collected from seven diverse domains. CoQA has a lot of challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning.

Download

Browse the examples in CoQA:

Download a copy of the dataset in json format:


Evaluation

To evaluate your models, use the official evaluation script. To run the evaluation, use python evaluate-v1.0.py --data-file <path_to_dev-v1.0.json> --pred-file <path_to_predictions>.

Once you are satisfied with your model performance on the dev set, you submit it to get the official scores on the test sets. We have two test sets, an in-domain set which constitutes the domains present in the training and the dev sets, and an out-of-domain set which constitutes unseen domains (see the paper for more details). To preserve the integrity of the test results, we do not release the test set to the public. Follow this tutorial on how to submit your model for an official evaluation:

Submission Tutorial

License

CoQA contains passages from seven domains. We make five of these public under the following licenses:

  • Literature and Wikipedia passages are shared under CC BY-SA 4.0 license.
  • Children's stories are collected from MCTest which comes with MSR-LA license.
  • Middle/High school exam passages are collected from RACE which comes with its own license.
  • News passages are collected from the DeepMind CNN dataset which comes with Apache license.

Questions?

Ask us questions at our google group or at sivar@cs.stanford.edu or danqi@cs.stanford.edu.

Acknowledgements

We thank the SQuAD team for allowing us to use their code and templates for generating this website.

Leaderboard

RankModelIn-domainOut-of-domainOverall
Human Performance

Stanford University

(Reddy & Chen et al. TACL '19)
89.487.488.8

1

Sep 05, 2019
RoBERTa + AT + KD (ensemble)

Zhuiyi Technology

https://arxiv.org/abs/1909.10772
91.489.290.7

2

Sep 05, 2019
RoBERTa + AT + KD (single model)

Zhuiyi Technology

https://arxiv.org/abs/1909.10772
90.989.290.4

3

Mar 30, 2019
Google SQuAD 2.0 + MMFT (ensemble)

MSRA + SDRG

89.988.089.4

4

Sep 13, 2019
XLNet + Augmentation (single model)

Xiaoming

https://github.com/stevezheng23/xlnet_extension_tf
89.986.989.0

5

Mar 30, 2019
ConvBERT (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

88.785.487.8

5

Mar 30, 2019
Google SQuAD 2.0 + MMFT (single model)

MSRA + SDRG

88.586.087.8

6

Jan 26, 2019
BERT + MMFT + ADA (ensemble)

Microsoft Research Asia

87.585.386.8

6

Mar 29, 2019
ConvBERT (single model)

Joint Laboratory of HIT and iFLYTEK Research

87.784.686.8

7

Jan 22, 2019
BERT + MMFT + ADA (single model)

Microsoft Research Asia

86.481.985.0

8

Aug 26, 2019
BERT + AttentionFusionNet (single model)

Beijing Kingsoft AI Lab

85.477.383.0

9

Jan 04, 2019
BERT + Answer Verification (single model)

Sogou Search AI Group

https://github.com/sogou/SMRCToolkit
83.880.282.8

10

Jan 07, 2019
BERT with History Augmented Query (single model)

Fudan University NLP Lab

82.778.681.5

11

Feb 01, 2019
BERT Large Finetuned Baseline (single model)

Anonymous

82.678.481.4

12

Jan 22, 2019
BERT Large Augmented (single model)

Microsoft Dynamics 365 AI Research

82.577.681.1

13

Dec 13, 2018
D-AoA + BERT (single model)

Joint Laboratory of HIT and iFLYTEK Research

81.477.380.2

14

Aug 01, 2019
BERT Augmented + AoA (single model)

Netease Games AI Lab

81.177.480.0

15

Mar 11, 2019
CNet (single model)

Anonymous

80.977.179.8

16

Nov 30, 2018
SDNet (ensemble)

Microsoft Speech and Dialogue Research Group

https://github.com/Microsoft/SDNet
80.775.979.3

17

Feb 22, 2019
CQANet (single model)

Nanjing University

80.276.579.1

18

May 10, 2019
CANet (single model)

Northwestern Polytechnical University

80.175.778.9

19

Jul 14, 2019
BERT Finetuned Baseline

single model

79.776.378.7

19

Apr 14, 2019
BERT w/ 2-context (single model)

NTT Media Intelligence Laboratories

https://arxiv.org/pdf/1905.12848
79.875.978.7

20

Dec 31, 2018
BERT-base finetune (single model)

Tsinghua University CoAI Lab

79.874.178.1

21

Apr 20, 2019
Bert-FlowDelta (single model)

National Taiwan University, MiuLab

https://arxiv.org/abs/1908.05117
79.274.177.7

22

Mar 01, 2019
GraphFlow (single model)

RPI and IBM Research

https://arxiv.org/pdf/1908.00059.pdf
78.474.577.3

23

Nov 27, 2018
SDNet (single model)

Microsoft Speech and Dialogue Research Group

https://github.com/Microsoft/SDNet
78.073.176.6

24

Aug 29, 2019
Flow Framework (single model)

SIAT NLP Group

77.073.175.8

25

Oct 07, 2018
FlowQA (single model)

Allen Institute for Artificial Intelligence

https://arxiv.org/abs/1810.06683
76.371.875.0

26

Jul 17, 2019
HisFurC + BERT

single model

76.070.474.4

27

Jan 15, 2019
RNet + PGNet + BERT (single model)

Nanjing University

74.770.073.3

28

Feb 01, 2019
XyzNet (single model)

Beijing Normal University

74.368.872.7

29

Dec 31, 2018
DrQA + marker features (single model)

Stanford University

71.665.169.7

30

Dec 11, 2018
BiDAF++ (single model)

Beijing University of Posts and Telecommunications

71.165.569.5

31

Sep 27, 2018
BiDAF++ (single model)

Allen Institute for Artificial Intelligence

https://arxiv.org/abs/1809.10735
69.463.867.8

32

Nov 23, 2018
Bert Base Augmented (single model)

Fudan University NLP Lab

68.461.866.5

33

Dec 18, 2018
RNet_DotAtt + seq2seq with copy attention (single model)

University of Science and Technology of China

68.162.366.4

34

Dec 31, 2018
Simplified BiDAF++ (single model)

Peking University

68.760.566.3

35

Aug 21, 2018
DrQA + seq2seq with copy attention (single model)

Stanford University

https://arxiv.org/abs/1808.07042
67.060.465.1

36

Aug 21, 2018
Vanilla DrQA (single model)

Stanford University

https://arxiv.org/abs/1808.07042
54.547.952.6