OLAC Record

Title:TDT5 Multilingual Text
Access Rights:Licensing Instructions for Subscription & Standard Members, and Non-Members: http://www.ldc.upenn.edu/language-resources/data/obtaining
Bibliographic Citation:Graff, David, et al. TDT5 Multilingual Text LDC2006T18. Web Download. Philadelphia: Linguistic Data Consortium, 2006
Contributor:Graff, David
Kong, Junbo
Maeda, Kazuaki
Strassel, Stephanie
Date (W3CDTF):2006
Date Issued (W3CDTF):2006-12-19
Description:*Introduction* The TDT5 Multilingual Text corpus was developed by the Linguistic Data Consortium (LDC) and contains the complete set of English, Arabic, and Chinese newswire text, totalling 407,503 stories, used in the 2004 Topic Detection and Tracking technology evaluations. The TDT5 corpora were created by LDC with support from the DARPA TIDES (Translingual Information Detection, Extraction and Summarization) Program. Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data such as newswire and broadcast news. The topic relevance annotations corresponding to this publication can be found in TDT5 Topics and Annotations (LDC2006T19). There were four TDT tasks defined for the 2004 evaluation: the tracking of known topics, the detection of unknown topics, the detection of initial stories on unknown topics, and the detection of pairs of stories on the same topic (links). Of these four tasks, the topic tracking task and the link detection task are considered to be "primary." Previous TDT evaluations also included a story segmentation task. This task applied only to broadcast news. Since TDT5 does not include broadcast news, there is no story segmentation task in the 2004 TDT Evaluation. *Data* The TDT5 corpus spans collections from April-September 2003 of English, Chinese, and Arabic news text. A total of 15 distinct news "sources" are included (where a "source" comprises data from a given news agency in a particular language; when an agency publishes in multiple languages, each language is considered a different "source"). In contrast to earlier TDT corpora, TDT5 has no broadcast/audio content, only printed news from wire and web sources. TDT5 is a subset of the previous LDC release HARD 2004 Text (LDC2005T28). In the HARD 2004 Text release, data is organized into one file per day per source. For the current TDT5 corpus, roughly half of the daily files are partitioned into chunks of some maximum size (around 30 K-words, on average), in order to support the "look-ahead" condition in the TDT5 evaluation plan and to keep sample file size relatively consistent. *Samples* The images below are samples of the text data contained in this corpus. * Arabic (TXT) * English (TXT) * Chinese (TXT) *Updates* None at this time.
Extent:Corpus size: 2474639 KB
ISBN: 1-58563-417-4
ISLRN: 492-763-586-162-2
DOI: 10.35111/azvq-yv94
Mandarin Chinese
Standard Arabic
Language (ISO639):eng
License:LDC User Agreement for Non-Members: https://catalog.ldc.upenn.edu/license/ldc-non-members-agreement.pdf
Medium:Distribution: Web Download
Publisher:Linguistic Data Consortium
Publisher (URI):https://www.ldc.upenn.edu
Relation (URI):https://catalog.ldc.upenn.edu/docs/LDC2006T18
Rights Holder:Portions © 2003 Agence France Presse, An-Nahar, The Associated Press, The Central News Agency, Los Angeles Times-Washington Post News Service, Inc., The New York Times, SPH AsiaOne Ltd, Ummah Press Service, Xinhua News Agency, © 2003, 2004, 2006 Trustees of the University of Pennsylvania
Type (DCMI):Text
Type (OLAC):primary_text


Archive:  The LDC Corpus Catalog
Description:  http://www.language-archives.org/archive/www.ldc.upenn.edu
GetRecord:  OAI-PMH request for OLAC format
GetRecord:  Pre-generated XML file

OAI Info

OaiIdentifier:  oai:www.ldc.upenn.edu:LDC2006T18
DateStamp:  2021-02-12
GetRecord:  OAI-PMH request for simple DC format

Search Info

Citation: Graff, David; Kong, Junbo; Maeda, Kazuaki; Strassel, Stephanie. 2006. Linguistic Data Consortium.
Terms: area_Asia area_Europe country_CN country_GB country_SA dcmi_Text iso639_arb iso639_cmn iso639_eng olac_primary_text

Up-to-date as of: Tue May 7 7:24:50 EDT 2024