Language resource management -- Corpus query lingua franca (CQLF) -- Part 1: Metamodel

This document describes the abstract metamodel designed to accommodate any corpus query language
(QL) and providing a basis for coarse-grained classification. The metamodel consists of several
components referred to as CQLF classes, levels, and modules, and is illustrated with examples from
the Single-stream class (where a single data stream is used to organize the relevant data structures).
Within this class, this document discusses three CQLF levels (Linear, Complex and Concurrent), as well
as their subdivisions into modules, dictated by functional and modelling criteria.
This document does not provide a way to specify further details beyond the above-mentioned divisions,
and neither does it contain within its scope QLs designed to query more than one concurrent data
stream, as in multimodal corpora or in parallel corpora (such QLs can still be classified according to the
criteria suggested here for less expressive QLs).

Gestion des ressources linguistiques -- Corpus query lingua franca (CQLF) -- Partie 1: Métamodèle

Upravljanje z jezikovnimi viri - Lingua franca za korpusne poizvedbe (CQLF) - 1. del: Metamodel

Ta dokument opisuje abstraktni metamodel, ki je prilagojen poljubnemu jeziku za korpusne poizvedbe (QL) in podaja podlago za grobozrnato razvrstitev. Metamodel je sestavljen iz več komponent, imenovanih razredi, stopnje in moduli lingue france za korpusne poizvedbe (CQLF), ter je opisan s primeri iz razreda enojnega podatkovnega toka (pri čemer so ustrezne podatkovne strukture urejene z uporabo enojnega podatkovnega toka).
V okviru tega razreda ta dokument obravnava tri stopnje lingue france za korpusne poizvedbe (linearna, kompleksna in hkratna) ter njihovo razdeljenost na module, odvisne od meril funkcionalnosti in modeliranja.
Ta dokument ne podaja načina za določanje dodatnih informacij, ki podrobneje opisujejo zgoraj omenjene razdelitve, in na njegovo področje uporabe ne spadajo jeziki za korpusne poizvedbe, namenjeni za poizvedovanje v več kot enem hkratnem podatkovnem toku, tako kot v večmodalnih ali vzporednih korpusih (takšne jezike za korpusne poizvedbe je še vedno mogoče razvrstiti v skladu s tukaj predlaganimi merili za manj izrazne jezike za korpusne poizvedbe).

General Information

Status
Published
Publication Date
06-Sep-2018
Current Stage
6060 - National Implementation/Publication (Adopted Project)
Start Date
30-Jul-2018
Due Date
04-Oct-2018
Completion Date
07-Sep-2018

Buy Standard

Standard
SIST ISO 24623-1:2018
English language
17 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day
Standard
ISO 24623-1:2018 - Language resource management -- Corpus query lingua franca (CQLF)
English language
12 pages
sale 15% off
Preview
sale 15% off
Preview
Standard
SIST ISO 24623-1:2018
English language
17 pages
sale 10% off
Preview
sale 10% off
Preview

e-Library read for
1 day

Standards Content (sample)

SLOVENSKI STANDARD
SIST ISO 24623-1:2018
01-oktober-2018
Upravljanje z jezikovnimi viri - Lingua franca za korpusne poizvedbe (CQLF) - 1.
del: Metamodel
Language resource management -- Corpus query lingua franca (CQLF) -- Part 1:
Metamodel

Gestion des ressources linguistiques -- Corpus query lingua franca (CQLF) -- Partie 1:

Métamodèle
Ta slovenski standard je istoveten z: ISO 24623-1:2018
ICS:
01.020 Terminologija (načela in Terminology (principles and
koordinacija) coordination)
01.140.20 Informacijske vede Information sciences
35.240.30 Uporabniške rešitve IT v IT applications in information,
informatiki, dokumentiranju in documentation and
založništvu publishing
SIST ISO 24623-1:2018 en,fr,de

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
SIST ISO 24623-1:2018
---------------------- Page: 2 ----------------------
SIST ISO 24623-1:2018
INTERNATIONAL ISO
STANDARD 24623-1
First edition
2018-04
Language resource management —
Corpus query lingua franca (CQLF) —
Part 1:
Metamodel
Gestion des ressources linguistiques — Corpus query lingua franca
(CQLF) —
Partie 1: Métamodèle
Reference number
ISO 24623-1:2018(E)
ISO 2018
---------------------- Page: 3 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2018

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO 2018 – All rights reserved
---------------------- Page: 4 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Aims ................................................................................................................................................................................................................................... 4

5 Assumptions .............................................................................................................................................................................................................. 4

6 CQLF Metamodel ................................................................................................................................................................................................... 4

7 Conformance ............................................................................................................................................................................................................. 7

Annex A (informative) Example CQLF conformance statements ............................................................................................. 8

Bibliography .............................................................................................................................................................................................................................12

© ISO 2018 – All rights reserved iii
---------------------- Page: 5 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www .iso .org/ patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following

URL: www .iso .org/ iso/ foreword .html.

This document was prepared by Technical Committee ISO/TC 37, Language and terminology,

Subcommittee SC 4, Language resource management.

A list of all parts in the ISO 24623 series can be found on the ISO website. Additional parts on single-

stream and multi-stream ontology architectures are planned to be developed in the future.

iv © ISO 2018 – All rights reserved
---------------------- Page: 6 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
Introduction

A range of standards relating to language resource management, with the Linguistic Annotation

Framework (ISO 24612) at the centre, have been developed. These standards are mostly designed to

regulate the representation aspect of language data – they look at the data from the point of view of

preparation and curation. This document complements this perspective by that of the end-user, that is

to say, from the point of view of processing and querying.

The corpus linguistic community has, by now, developed several corpus query languages (QLs), and there

is a particularly large number of them if “dialects” and forks are included. There are two main reasons

for this abundance. Firstly, there are socio-economic and organizational factors, with separate query

systems having been created by isolated projects with un-coordinated funding, many of them eventually

developing their own set of followers. Secondly, query systems are typically sensitive to the format of the

data and are often designed with a specific purpose in mind. For example, systems for querying parallel

audio and transcription streams with multiple speakers have different characteristics from systems

designed to query purely textual data with a single layer of morphosyntactic description. Dependency

and hierarchical annotations demand yet another set of solutions. All of this results in the richness of

alternatives or near-alternatives on the one hand, and in the lack of interoperability among the variants

on the other. As a consequence, a “wrong” choice at the beginning of a project can bury months of

research by exposing inadequacies in the initial decision after the project has become mature enough to

move to new extended functionality and towards addressing more complex information needs.

This document codifies, in a modular way, the best existing practices followed in the design of corpus

query languages. Its theoretical aim is to provide a basis for the investigation of the relationships

between language resource architecture and corpus query language properties. The practical aim of

the Corpus Query Lingua Franca (henceforth CQLF) is to provide linguists and language technology

practitioners with a clear and coherent basis for making informed choices concerning data architectures

and the query languages appropriate to them.
© ISO 2018 – All rights reserved v
---------------------- Page: 7 ----------------------
SIST ISO 24623-1:2018
---------------------- Page: 8 ----------------------
SIST ISO 24623-1:2018
INTERNATIONAL STANDARD ISO 24623-1:2018(E)
Language resource management — Corpus query lingua
franca (CQLF) —
Part 1:
Metamodel
1 Scope

This document describes the abstract metamodel designed to accommodate any corpus query language

(QL) and providing a basis for coarse-grained classification. The metamodel consists of several

components referred to as CQLF classes, levels, and modules, and is illustrated with examples from

the Single-stream class (where a single data stream is used to organize the relevant data structures).

Within this class, this document discusses three CQLF levels (Linear, Complex and Concurrent), as well

as their subdivisions into modules, dictated by functional and modelling criteria.

This document does not provide a way to specify further details beyond the above-mentioned divisions,

and neither does it contain within its scope QLs designed to query more than one concurrent data

stream, as in multimodal corpora or in parallel corpora (such QLs can still be classified according to the

criteria suggested here for less expressive QLs).
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 24611, Language resource management — Morpho-syntactic annotation framework (MAF)

ISO 24612, Language resource management — Linguistic annotation framework (LAF)

ISO 24615-1, Language resource management — Syntactic annotation framework (SynAF) — Part 1:

Syntactic model
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— IEC Electropedia: available at https:// www .electropedia .org/
— ISO Online browsing platform: available at https:// www .iso .org/ obp
3.1
annotation
information added to primary data (3.9), independent of its representation

[SOURCE: ISO 24612:2012, 2.3, modified — "linguistic" at the beginning of the definition was deleted.]

© ISO 2018 – All rights reserved 1
---------------------- Page: 9 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
3.1.1
concurrent annotations

multiple, potentially conflicting annotations (3.1) describing, entirely or partly, the same character span

(3.2) or an overlapping sequence of character spans

Note 1 to entry: Concurrent annotations may be expected to conflict in several ways: content-wise (with different

tags for the same character span), structure-wise (assuming different structural arrangements within the

targeted character spans), and also in terms of segment edges (which is typically due to structurally conflicting

claims concerning the encompassing character spans). Concurrent annotations typically come from different

sources (e.g. tools or human annotators) or result from different settings (e.g. different parsing models or

segmentation rules) within a single tool. When encoded in XML, concurrent annotations are typically expressed

by means of stand-off techniques.
3.1.2
dependency annotation

annotation (3.1) that encodes the dependency relations between character spans (3.2)

Note 1 to entry: An example of a dependency relation (see ISO 24615-1:2014, 3.5) is one between a verb and

its subject or direct object, between an attributive adjective and its head noun, or between a preposition and

the head of its dependent noun phrase. Dependency relations may be defined at the word-level alone, or may

involve higher-level syntactic constructs, in which case it is possible to speak of mixed hierarchical-dependency

annotations.
3.1.3
hierarchical annotation

annotation (3.1) that encodes the relationship of dominance (often also precedence) necessary to define

syntactic trees over character spans (3.2)

Note 1 to entry: Annotating hierarchical relationships requires only the relation of dominance to be indicated.

Precedence is typically implicit in the ordering of character spans.
3.1.4
segmentation annotation

annotation (3.1) that delimits linguistic elements that appear in the primary data (3.9)

Note 1 to entry: These elements include (1) continuous segments (appearing contiguously in the primary

data), (2) super- and sub-segments, where groups of segments will comprise the parts of a larger segment

(e.g. contiguous word segments typically comprise a sentence segment), (3) discontinuous segments (linking

continuous segments) and (4) landmarks (e.g. time stamps) that note a point in the primary data. In current

practice, segmental information may or may not appear in the document containing the primary data itself.

[SOURCE: ISO 24612:2012, 2.5]
3.1.5
simple annotation

annotation (3.1) that constitutes a single information package whose interpretation is not dependent on

other annotations

Note 1 to entry: This definition is intended to distinguish the simplest (“tabular”) kind of annotation from

more complex relational structures (providing hierarchical, dependency, or alignment information); simple

annotations are the only kind of annotations present at the linear level of complexity.

3.1.6
stand-off annotation

annotation (3.1) that can be layered over primary data (3.9) but is separated from the data stream that

it targets

Note 1 to entry: Stand-off annotations refer to specific locations in the primary data, by addressing the character

offsets, elements or coordinates to which the annotation applies. They can be serialized as separate documents,

but do not have to be. Multiple stand-off annotation documents for a given type of annotation can refer to the

same primary document (e.g. two different part of speech annotations for a given text). It is also possible to

construct hierarchies of stand-off annotation layers, where layer n can reference layers 0..n−1.

2 © ISO 2018 – All rights reserved
---------------------- Page: 10 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
[SOURCE: ISO 24612:2012, 2.7, modified — The definition and note were modified.]
3.2
character span

sequence of characters, identified by start and end offsets, to which an annotation may be applied

Note 1 to entry: This definition is a relaxed version of the definition in ISO 24615-1:2014, 3.16, the difference lying

in the use of “may be applied” over “is applied”. Compare also the definition of “region” in ISO 24612:2012, 2.10.

3.3
character span containment

relation obtaining between character spans (3.2) of primary data (3.9) in which character span A

contains character span B if the initial offset of span A is equal to or higher than that of span B, and the

final offset of span A is smaller than or equal to that of span B

Note 1 to entry: The relation of character span containment is used for stating a relationship between two or

more character spans or simple annotations, without the need to utilize tree-based concepts and mechanisms.

Instead of tree traversal, operators such as contains, in or within are typically used for character span containment

queries.
3.4
corpus query language

formal language designed to retrieve specific information from (large) language data collections, and

thereby incorporate certain abstractions over commonly shared data models that make it possible for

the user (or user agents) to address parts of those data models
3.5
CQLF class
top-level division in the CQLF data model

Note 1 to entry: The CQLF Metamodel distinguishes two classes: Single-stream (where the annotation structure

is built upon a single data stream, typically a character stream) and Multi-stream (corresponding to e.g. multi-

modal corpora or parallel corpora).
3.6
CQLF implementation

query language that has been analysed with respect to the criteria described by the CQLF Metamodel,

and thus has been “located” in the proposed feature matrix as “conformant with CQLF”

3.7
CQLF level

part of the matrix of QL properties, defined in terms of the general features of the assumed corpus data

models, and consequently the set of properties of a corpus query language that is used to address these

features

Note 1 to entry: The CQLF Metamodel distinguishes three levels of complexity within the Single-stream class:

Linear, Complex and Concurrent.
3.8
CQLF module

subcomponent of a CQLF level, defined with reference to a specified data-model characteristic

Note 1 to entry: CQLF Metamodel currently distinguishes three modules within CQLF Level 1, Linear (plain-

text, segmentation, and simple annotation), and three modules within CQLF Level 2, Complex (hierarchical,

dependency, and containment).
3.9
primary data
electronic representation of language data
© ISO 2018 – All rights reserved 3
---------------------- Page: 11 ----------------
...

INTERNATIONAL ISO
STANDARD 24623-1
First edition
2018-04
Language resource management —
Corpus query lingua franca (CQLF) —
Part 1:
Metamodel
Gestion des ressources linguistiques — Corpus query lingua franca
(CQLF) —
Partie 1: Métamodèle
Reference number
ISO 24623-1:2018(E)
ISO 2018
---------------------- Page: 1 ----------------------
ISO 24623-1:2018(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2018

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO 2018 – All rights reserved
---------------------- Page: 2 ----------------------
ISO 24623-1:2018(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Aims ................................................................................................................................................................................................................................... 4

5 Assumptions .............................................................................................................................................................................................................. 4

6 CQLF Metamodel ................................................................................................................................................................................................... 4

7 Conformance ............................................................................................................................................................................................................. 7

Annex A (informative) Example CQLF conformance statements ............................................................................................. 8

Bibliography .............................................................................................................................................................................................................................12

© ISO 2018 – All rights reserved iii
---------------------- Page: 3 ----------------------
ISO 24623-1:2018(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www .iso .org/ patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following

URL: www .iso .org/ iso/ foreword .html.

This document was prepared by Technical Committee ISO/TC 37, Language and terminology,

Subcommittee SC 4, Language resource management.

A list of all parts in the ISO 24623 series can be found on the ISO website. Additional parts on single-

stream and multi-stream ontology architectures are planned to be developed in the future.

iv © ISO 2018 – All rights reserved
---------------------- Page: 4 ----------------------
ISO 24623-1:2018(E)
Introduction

A range of standards relating to language resource management, with the Linguistic Annotation

Framework (ISO 24612) at the centre, have been developed. These standards are mostly designed to

regulate the representation aspect of language data – they look at the data from the point of view of

preparation and curation. This document complements this perspective by that of the end-user, that is

to say, from the point of view of processing and querying.

The corpus linguistic community has, by now, developed several corpus query languages (QLs), and there

is a particularly large number of them if “dialects” and forks are included. There are two main reasons

for this abundance. Firstly, there are socio-economic and organizational factors, with separate query

systems having been created by isolated projects with un-coordinated funding, many of them eventually

developing their own set of followers. Secondly, query systems are typically sensitive to the format of the

data and are often designed with a specific purpose in mind. For example, systems for querying parallel

audio and transcription streams with multiple speakers have different characteristics from systems

designed to query purely textual data with a single layer of morphosyntactic description. Dependency

and hierarchical annotations demand yet another set of solutions. All of this results in the richness of

alternatives or near-alternatives on the one hand, and in the lack of interoperability among the variants

on the other. As a consequence, a “wrong” choice at the beginning of a project can bury months of

research by exposing inadequacies in the initial decision after the project has become mature enough to

move to new extended functionality and towards addressing more complex information needs.

This document codifies, in a modular way, the best existing practices followed in the design of corpus

query languages. Its theoretical aim is to provide a basis for the investigation of the relationships

between language resource architecture and corpus query language properties. The practical aim of

the Corpus Query Lingua Franca (henceforth CQLF) is to provide linguists and language technology

practitioners with a clear and coherent basis for making informed choices concerning data architectures

and the query languages appropriate to them.
© ISO 2018 – All rights reserved v
---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO 24623-1:2018(E)
Language resource management — Corpus query lingua
franca (CQLF) —
Part 1:
Metamodel
1 Scope

This document describes the abstract metamodel designed to accommodate any corpus query language

(QL) and providing a basis for coarse-grained classification. The metamodel consists of several

components referred to as CQLF classes, levels, and modules, and is illustrated with examples from

the Single-stream class (where a single data stream is used to organize the relevant data structures).

Within this class, this document discusses three CQLF levels (Linear, Complex and Concurrent), as well

as their subdivisions into modules, dictated by functional and modelling criteria.

This document does not provide a way to specify further details beyond the above-mentioned divisions,

and neither does it contain within its scope QLs designed to query more than one concurrent data

stream, as in multimodal corpora or in parallel corpora (such QLs can still be classified according to the

criteria suggested here for less expressive QLs).
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 24611, Language resource management — Morpho-syntactic annotation framework (MAF)

ISO 24612, Language resource management — Linguistic annotation framework (LAF)

ISO 24615-1, Language resource management — Syntactic annotation framework (SynAF) — Part 1:

Syntactic model
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— IEC Electropedia: available at https:// www .electropedia .org/
— ISO Online browsing platform: available at https:// www .iso .org/ obp
3.1
annotation
information added to primary data (3.9), independent of its representation

[SOURCE: ISO 24612:2012, 2.3, modified — "linguistic" at the beginning of the definition was deleted.]

© ISO 2018 – All rights reserved 1
---------------------- Page: 6 ----------------------
ISO 24623-1:2018(E)
3.1.1
concurrent annotations

multiple, potentially conflicting annotations (3.1) describing, entirely or partly, the same character span

(3.2) or an overlapping sequence of character spans

Note 1 to entry: Concurrent annotations may be expected to conflict in several ways: content-wise (with different

tags for the same character span), structure-wise (assuming different structural arrangements within the

targeted character spans), and also in terms of segment edges (which is typically due to structurally conflicting

claims concerning the encompassing character spans). Concurrent annotations typically come from different

sources (e.g. tools or human annotators) or result from different settings (e.g. different parsing models or

segmentation rules) within a single tool. When encoded in XML, concurrent annotations are typically expressed

by means of stand-off techniques.
3.1.2
dependency annotation

annotation (3.1) that encodes the dependency relations between character spans (3.2)

Note 1 to entry: An example of a dependency relation (see ISO 24615-1:2014, 3.5) is one between a verb and

its subject or direct object, between an attributive adjective and its head noun, or between a preposition and

the head of its dependent noun phrase. Dependency relations may be defined at the word-level alone, or may

involve higher-level syntactic constructs, in which case it is possible to speak of mixed hierarchical-dependency

annotations.
3.1.3
hierarchical annotation

annotation (3.1) that encodes the relationship of dominance (often also precedence) necessary to define

syntactic trees over character spans (3.2)

Note 1 to entry: Annotating hierarchical relationships requires only the relation of dominance to be indicated.

Precedence is typically implicit in the ordering of character spans.
3.1.4
segmentation annotation

annotation (3.1) that delimits linguistic elements that appear in the primary data (3.9)

Note 1 to entry: These elements include (1) continuous segments (appearing contiguously in the primary

data), (2) super- and sub-segments, where groups of segments will comprise the parts of a larger segment

(e.g. contiguous word segments typically comprise a sentence segment), (3) discontinuous segments (linking

continuous segments) and (4) landmarks (e.g. time stamps) that note a point in the primary data. In current

practice, segmental information may or may not appear in the document containing the primary data itself.

[SOURCE: ISO 24612:2012, 2.5]
3.1.5
simple annotation

annotation (3.1) that constitutes a single information package whose interpretation is not dependent on

other annotations

Note 1 to entry: This definition is intended to distinguish the simplest (“tabular”) kind of annotation from

more complex relational structures (providing hierarchical, dependency, or alignment information); simple

annotations are the only kind of annotations present at the linear level of complexity.

3.1.6
stand-off annotation

annotation (3.1) that can be layered over primary data (3.9) but is separated from the data stream that

it targets

Note 1 to entry: Stand-off annotations refer to specific locations in the primary data, by addressing the character

offsets, elements or coordinates to which the annotation applies. They can be serialized as separate documents,

but do not have to be. Multiple stand-off annotation documents for a given type of annotation can refer to the

same primary document (e.g. two different part of speech annotations for a given text). It is also possible to

construct hierarchies of stand-off annotation layers, where layer n can reference layers 0..n−1.

2 © ISO 2018 – All rights reserved
---------------------- Page: 7 ----------------------
ISO 24623-1:2018(E)
[SOURCE: ISO 24612:2012, 2.7, modified — The definition and note were modified.]
3.2
character span

sequence of characters, identified by start and end offsets, to which an annotation may be applied

Note 1 to entry: This definition is a relaxed version of the definition in ISO 24615-1:2014, 3.16, the difference lying

in the use of “may be applied” over “is applied”. Compare also the definition of “region” in ISO 24612:2012, 2.10.

3.3
character span containment

relation obtaining between character spans (3.2) of primary data (3.9) in which character span A

contains character span B if the initial offset of span A is equal to or higher than that of span B, and the

final offset of span A is smaller than or equal to that of span B

Note 1 to entry: The relation of character span containment is used for stating a relationship between two or

more character spans or simple annotations, without the need to utilize tree-based concepts and mechanisms.

Instead of tree traversal, operators such as contains, in or within are typically used for character span containment

queries.
3.4
corpus query language

formal language designed to retrieve specific information from (large) language data collections, and

thereby incorporate certain abstractions over commonly shared data models that make it possible for

the user (or user agents) to address parts of those data models
3.5
CQLF class
top-level division in the CQLF data model

Note 1 to entry: The CQLF Metamodel distinguishes two classes: Single-stream (where the annotation structure

is built upon a single data stream, typically a character stream) and Multi-stream (corresponding to e.g. multi-

modal corpora or parallel corpora).
3.6
CQLF implementation

query language that has been analysed with respect to the criteria described by the CQLF Metamodel,

and thus has been “located” in the proposed feature matrix as “conformant with CQLF”

3.7
CQLF level

part of the matrix of QL properties, defined in terms of the general features of the assumed corpus data

models, and consequently the set of properties of a corpus query language that is used to address these

features

Note 1 to entry: The CQLF Metamodel distinguishes three levels of complexity within the Single-stream class:

Linear, Complex and Concurrent.
3.8
CQLF module

subcomponent of a CQLF level, defined with reference to a specified data-model characteristic

Note 1 to entry: CQLF Metamodel currently distinguishes three modules within CQLF Level 1, Linear (plain-

text, segmentation, and simple annotation), and three modules within CQLF Level 2, Complex (hierarchical,

dependency, and containment).
3.9
primary data
electronic representation of language data
© ISO 2018 – All rights reserved 3
---------------------- Page: 8 ----------------------
ISO 24623-1:2018(E)
3.10
token
non-empty contiguous sequence of graphemes or phonemes in a document
[SOURCE: ISO 24611:2012, 3.21, modified — The note was deleted.]
4 Aims

The CQLF Metamodel is intended to establish a frame and a basis for establishing the potential extent

and the limits of interoperability between different corpus query systems. It aims to provide a single

matrix of a few well-defined properties in which any corpus QL can be located for the purpose of coarse-

grained comparison with the others. Further parts of the standard elaborate on these prope

...

SLOVENSKI STANDARD
SIST ISO 24623-1:2018
01-oktober-2018
Upravljanje z jezikovnimi viri - Lingua franca za korpusne poizvedbe (CQLF) - 1.
del: Metamodel
Language resource management -- Corpus query lingua franca (CQLF) -- Part 1:
Metamodel

Gestion des ressources linguistiques -- Corpus query lingua franca (CQLF) -- Partie 1:

Métamodèle
Ta slovenski standard je istoveten z: ISO 24623-1:2018
ICS:
01.020 7HUPLQRORJLMD QDþHODLQ Terminology (principles and
NRRUGLQDFLMD coordination)
35.060 Jeziki, ki se uporabljajo v Languages used in
informacijski tehniki in information technology
tehnologiji
SIST ISO 24623-1:2018 en,fr,de

2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

---------------------- Page: 1 ----------------------
SIST ISO 24623-1:2018
---------------------- Page: 2 ----------------------
SIST ISO 24623-1:2018
INTERNATIONAL ISO
STANDARD 24623-1
First edition
2018-04
Language resource management —
Corpus query lingua franca (CQLF) —
Part 1:
Metamodel
Gestion des ressources linguistiques — Corpus query lingua franca
(CQLF) —
Partie 1: Métamodèle
Reference number
ISO 24623-1:2018(E)
ISO 2018
---------------------- Page: 3 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
COPYRIGHT PROTECTED DOCUMENT
© ISO 2018

All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may

be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting

on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address

below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO 2018 – All rights reserved
---------------------- Page: 4 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
Contents Page

Foreword ........................................................................................................................................................................................................................................iv

Introduction ..................................................................................................................................................................................................................................v

1 Scope ................................................................................................................................................................................................................................. 1

2 Normative references ...................................................................................................................................................................................... 1

3 Terms and definitions ..................................................................................................................................................................................... 1

4 Aims ................................................................................................................................................................................................................................... 4

5 Assumptions .............................................................................................................................................................................................................. 4

6 CQLF Metamodel ................................................................................................................................................................................................... 4

7 Conformance ............................................................................................................................................................................................................. 7

Annex A (informative) Example CQLF conformance statements ............................................................................................. 8

Bibliography .............................................................................................................................................................................................................................12

© ISO 2018 – All rights reserved iii
---------------------- Page: 5 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
Foreword

ISO (the International Organization for Standardization) is a worldwide federation of national standards

bodies (ISO member bodies). The work of preparing International Standards is normally carried out

through ISO technical committees. Each member body interested in a subject for which a technical

committee has been established has the right to be represented on that committee. International

organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.

ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of

electrotechnical standardization.

The procedures used to develop this document and those intended for its further maintenance are

described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the

different types of ISO documents should be noted. This document was drafted in accordance with the

editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).

Attention is drawn to the possibility that some of the elements of this document may be the subject of

patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of

any patent rights identified during the development of the document will be in the Introduction and/or

on the ISO list of patent declarations received (see www .iso .org/ patents).

Any trade name used in this document is information given for the convenience of users and does not

constitute an endorsement.

For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and

expressions related to conformity assessment, as well as information about ISO's adherence to the

World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following

URL: www .iso .org/ iso/ foreword .html.

This document was prepared by Technical Committee ISO/TC 37, Language and terminology,

Subcommittee SC 4, Language resource management.

A list of all parts in the ISO 24623 series can be found on the ISO website. Additional parts on single-

stream and multi-stream ontology architectures are planned to be developed in the future.

iv © ISO 2018 – All rights reserved
---------------------- Page: 6 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
Introduction

A range of standards relating to language resource management, with the Linguistic Annotation

Framework (ISO 24612) at the centre, have been developed. These standards are mostly designed to

regulate the representation aspect of language data – they look at the data from the point of view of

preparation and curation. This document complements this perspective by that of the end-user, that is

to say, from the point of view of processing and querying.

The corpus linguistic community has, by now, developed several corpus query languages (QLs), and there

is a particularly large number of them if “dialects” and forks are included. There are two main reasons

for this abundance. Firstly, there are socio-economic and organizational factors, with separate query

systems having been created by isolated projects with un-coordinated funding, many of them eventually

developing their own set of followers. Secondly, query systems are typically sensitive to the format of the

data and are often designed with a specific purpose in mind. For example, systems for querying parallel

audio and transcription streams with multiple speakers have different characteristics from systems

designed to query purely textual data with a single layer of morphosyntactic description. Dependency

and hierarchical annotations demand yet another set of solutions. All of this results in the richness of

alternatives or near-alternatives on the one hand, and in the lack of interoperability among the variants

on the other. As a consequence, a “wrong” choice at the beginning of a project can bury months of

research by exposing inadequacies in the initial decision after the project has become mature enough to

move to new extended functionality and towards addressing more complex information needs.

This document codifies, in a modular way, the best existing practices followed in the design of corpus

query languages. Its theoretical aim is to provide a basis for the investigation of the relationships

between language resource architecture and corpus query language properties. The practical aim of

the Corpus Query Lingua Franca (henceforth CQLF) is to provide linguists and language technology

practitioners with a clear and coherent basis for making informed choices concerning data architectures

and the query languages appropriate to them.
© ISO 2018 – All rights reserved v
---------------------- Page: 7 ----------------------
SIST ISO 24623-1:2018
---------------------- Page: 8 ----------------------
SIST ISO 24623-1:2018
INTERNATIONAL STANDARD ISO 24623-1:2018(E)
Language resource management — Corpus query lingua
franca (CQLF) —
Part 1:
Metamodel
1 Scope

This document describes the abstract metamodel designed to accommodate any corpus query language

(QL) and providing a basis for coarse-grained classification. The metamodel consists of several

components referred to as CQLF classes, levels, and modules, and is illustrated with examples from

the Single-stream class (where a single data stream is used to organize the relevant data structures).

Within this class, this document discusses three CQLF levels (Linear, Complex and Concurrent), as well

as their subdivisions into modules, dictated by functional and modelling criteria.

This document does not provide a way to specify further details beyond the above-mentioned divisions,

and neither does it contain within its scope QLs designed to query more than one concurrent data

stream, as in multimodal corpora or in parallel corpora (such QLs can still be classified according to the

criteria suggested here for less expressive QLs).
2 Normative references

The following documents are referred to in the text in such a way that some or all of their content

constitutes requirements of this document. For dated references, only the edition cited applies. For

undated references, the latest edition of the referenced document (including any amendments) applies.

ISO 24611, Language resource management — Morpho-syntactic annotation framework (MAF)

ISO 24612, Language resource management — Linguistic annotation framework (LAF)

ISO 24615-1, Language resource management — Syntactic annotation framework (SynAF) — Part 1:

Syntactic model
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.

ISO and IEC maintain terminological databases for use in standardization at the following addresses:

— IEC Electropedia: available at https:// www .electropedia .org/
— ISO Online browsing platform: available at https:// www .iso .org/ obp
3.1
annotation
information added to primary data (3.9), independent of its representation

[SOURCE: ISO 24612:2012, 2.3, modified — "linguistic" at the beginning of the definition was deleted.]

© ISO 2018 – All rights reserved 1
---------------------- Page: 9 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
3.1.1
concurrent annotations

multiple, potentially conflicting annotations (3.1) describing, entirely or partly, the same character span

(3.2) or an overlapping sequence of character spans

Note 1 to entry: Concurrent annotations may be expected to conflict in several ways: content-wise (with different

tags for the same character span), structure-wise (assuming different structural arrangements within the

targeted character spans), and also in terms of segment edges (which is typically due to structurally conflicting

claims concerning the encompassing character spans). Concurrent annotations typically come from different

sources (e.g. tools or human annotators) or result from different settings (e.g. different parsing models or

segmentation rules) within a single tool. When encoded in XML, concurrent annotations are typically expressed

by means of stand-off techniques.
3.1.2
dependency annotation

annotation (3.1) that encodes the dependency relations between character spans (3.2)

Note 1 to entry: An example of a dependency relation (see ISO 24615-1:2014, 3.5) is one between a verb and

its subject or direct object, between an attributive adjective and its head noun, or between a preposition and

the head of its dependent noun phrase. Dependency relations may be defined at the word-level alone, or may

involve higher-level syntactic constructs, in which case it is possible to speak of mixed hierarchical-dependency

annotations.
3.1.3
hierarchical annotation

annotation (3.1) that encodes the relationship of dominance (often also precedence) necessary to define

syntactic trees over character spans (3.2)

Note 1 to entry: Annotating hierarchical relationships requires only the relation of dominance to be indicated.

Precedence is typically implicit in the ordering of character spans.
3.1.4
segmentation annotation

annotation (3.1) that delimits linguistic elements that appear in the primary data (3.9)

Note 1 to entry: These elements include (1) continuous segments (appearing contiguously in the primary

data), (2) super- and sub-segments, where groups of segments will comprise the parts of a larger segment

(e.g. contiguous word segments typically comprise a sentence segment), (3) discontinuous segments (linking

continuous segments) and (4) landmarks (e.g. time stamps) that note a point in the primary data. In current

practice, segmental information may or may not appear in the document containing the primary data itself.

[SOURCE: ISO 24612:2012, 2.5]
3.1.5
simple annotation

annotation (3.1) that constitutes a single information package whose interpretation is not dependent on

other annotations

Note 1 to entry: This definition is intended to distinguish the simplest (“tabular”) kind of annotation from

more complex relational structures (providing hierarchical, dependency, or alignment information); simple

annotations are the only kind of annotations present at the linear level of complexity.

3.1.6
stand-off annotation

annotation (3.1) that can be layered over primary data (3.9) but is separated from the data stream that

it targets

Note 1 to entry: Stand-off annotations refer to specific locations in the primary data, by addressing the character

offsets, elements or coordinates to which the annotation applies. They can be serialized as separate documents,

but do not have to be. Multiple stand-off annotation documents for a given type of annotation can refer to the

same primary document (e.g. two different part of speech annotations for a given text). It is also possible to

construct hierarchies of stand-off annotation layers, where layer n can reference layers 0..n−1.

2 © ISO 2018 – All rights reserved
---------------------- Page: 10 ----------------------
SIST ISO 24623-1:2018
ISO 24623-1:2018(E)
[SOURCE: ISO 24612:2012, 2.7, modified — The definition and note were modified.]
3.2
character span

sequence of characters, identified by start and end offsets, to which an annotation may be applied

Note 1 to entry: This definition is a relaxed version of the definition in ISO 24615-1:2014, 3.16, the difference lying

in the use of “may be applied” over “is applied”. Compare also the definition of “region” in ISO 24612:2012, 2.10.

3.3
character span containment

relation obtaining between character spans (3.2) of primary data (3.9) in which character span A

contains character span B if the initial offset of span A is equal to or higher than that of span B, and the

final offset of span A is smaller than or equal to that of span B

Note 1 to entry: The relation of character span containment is used for stating a relationship between two or

more character spans or simple annotations, without the need to utilize tree-based concepts and mechanisms.

Instead of tree traversal, operators such as contains, in or within are typically used for character span containment

queries.
3.4
corpus query language

formal language designed to retrieve specific information from (large) language data collections, and

thereby incorporate certain abstractions over commonly shared data models that make it possible for

the user (or user agents) to address parts of those data models
3.5
CQLF class
top-level division in the CQLF data model

Note 1 to entry: The CQLF Metamodel distinguishes two classes: Single-stream (where the annotation structure

is built upon a single data stream, typically a character stream) and Multi-stream (corresponding to e.g. multi-

modal corpora or parallel corpora).
3.6
CQLF implementation

query language that has been analysed with respect to the criteria described by the CQLF Metamodel,

and thus has been “located” in the proposed feature matrix as “conformant with CQLF”

3.7
CQLF level

part of the matrix of QL properties, defined in terms of the general features of the assumed corpus data

models, and consequently the set of properties of a corpus query language that is used to address these

features

Note 1 to entry: The CQLF Metamodel distinguishes three levels of complexity within the Single-stream class:

Linear, Complex and Concurrent.
3.8
CQLF module

subcomponent of a CQLF level, defined with reference to a specified data-model characteristic

Note 1 to entry: CQLF Metamodel currently distinguishes three modules within CQLF Level 1, Linear (plain-

text, segmentation, and simple annotation), and three modules within CQLF Level 2, Complex (hierarchical,

dependency, and containment).
3.9
primary data
electronic representation of language data
© ISO 2018 – All rights reserved 3
---------------------- Page: 11 ----------------------
SIST ISO 24623-1:2018
ISO 246
...

Questions, Comments and Discussion

Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.