OLAC Record

Bibliographic Citation:http://hdl.handle.net/11858/00-097C-0000-000D-F67B-7
Creator:Pomikálek, Jan
Date (W3CDTF):2013-02-01T16:34:32Z
Date Available:2013-02-01T16:34:32Z
Description:onion (ONe Instance ONly) is a tool for removing duplicate parts from large collections of texts. The tool has been implemented in Python, licensed under New BSD License and made an open source software (available for download including the source code at http://code.google.com/p/onion/). It is being successfuly used for cleaning large textual corpora at Natural language processing centre at Faculty of informatics, Masaryk university Brno and it's industry partners. The research leading to this piece of software was published in author's Ph.D. thesis "Removing Boilerplate and Duplicate Content from Web Corpora". The deduplication algorithm is based on comparing n-grams of words of text. The author's algorithm has been shown to be more suitable for textual corpora deduplication than competing algorithms (Broder, Charikar): in addition to detection of identical or very similar (95 %) duplicates, it is able to detect even partially similar duplicates (50 %) still achieving great performace (further described in author's Ph.D. thesis). The unique deduplication capabilities and scalability of the algorithm were been demonstrated while building corpora of American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting --- several TB of text documents were deduplicated resulting in corpora of 70 billions tokens altogether.
PRESEMT, Lexical Computing Ltd
Identifier (URI):http://hdl.handle.net/11858/00-097C-0000-000D-F67B-7
Language (ISO639):eng
Publisher:Masaryk University, NLP Centre
Rights:BSD 3-Clause "New" or "Revised" license
text deduplication
n-gram deduplication
n-gram model
Type (DCMI):Software


Archive:  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University
Description:  http://www.language-archives.org/archive/lindat.mff.cuni.cz
GetRecord:  OAI-PMH request for OLAC format
GetRecord:  Pre-generated XML file

OAI Info

OaiIdentifier:  oai:lindat.mff.cuni.cz:11858/00-097C-0000-000D-F67B-7
DateStamp:  2021-06-29
GetRecord:  OAI-PMH request for simple DC format

Search Info

Citation: Pomikálek, Jan. 2013. Masaryk University, NLP Centre.
Terms: area_Europe country_GB dcmi_Software iso639_eng

Up-to-date as of: Thu Oct 5 0:38:54 EDT 2023