This chapter deals with general considerations useful for the formal encoding of linguistic annotation data.

1. Markup

• the phenomena investigated: Some researchers will describe the words and their properties, some will go for the relations of chunks of words, and others investigate their sound structures;
• the theory: This is how the linguistic phenomena are labelled, they may be called words and the theory might claim that these can be put into part of speech categories like A, B or C, or another theory might say that the POS values are NN, OO, PP and QQ. And perhaps these categories are not called POS but wc (word categories);
• the markup: This is the kind of characters, the grammar, the formalism used to represent the labels used by a given theory for the description of linguistic material. So if a piece of the corpus is:

"I saw it cross the street"


The individual words could be marked up in these two ways:
 

I saw it cross the street	normal orthographic markup
<w>I</w><w>saw</w><w>it</w><w>cross</w><w>the</w><w>street</w>	XML

2. Data files

The following figure represents these types of information and relations among them. The ellipse in the middle symbolize annotation levels. Square boxes are non-level information. Arrows indicate referring relations: Arrows between levels denote the reference between levels, e.g. reference from the word level to the phone-level; arrows from the level files to square boxes symbolize reference to rules that have been used during the annotation; arrows pointing from square boxes to ellipses indicate (meta-)annotation of the level label files. The shading of boxes or ellipses denotes that their contents is annotation of linguistic objects. Note that the direction of the arrows is also a reflex of the process of annotation, i.e. in most cases those objects pointed to are produced earlier.
 
 

The data produced during the annotation of a dialogue will be kept in separate files, but it should also be ensured that their mutual relationship and the fact that they belong to the same annotation project is transparent. There are two strategies to specify this membership information:

• list: A project resource file is established that keeps the file names of the data used for the annotation project (annotation process information)
• links: Relations are established between elements of different levels of description; the elements refer to each other and are located in separate files (level annotation information)

The following figure represents the different types of files and explicit references. It is an adapted version of a figure in D1.2 which has been modified in order to also reflect the direction of reference. Bold lines represent reference by file name, dotted lines represent reference (href-attributes) between annotation elements of different level annotation files.

3 Markup Conventions

XML has been chosen to be used for the representation of annotation data in the MATE project. Besides the actual good support of software for this standard, there are the following reasons for XML as the representation model:

XML uses

• one general description model containing
• elements
• their attributes
• part-whole relations betwen elements
• other relations between elements
• one encoding format to represent, relate, and distinguish these kinds of information

On the formal side this model is reflected by a uniform syntax described above: XML-elements are enclosed by angular brackets

inside which the values (val) of attributes (att) are specified: In XML Elements are those entities that conceptually group together descriptions of linguistic items. In XML element names are put after the <, such as in <word>. Attributes are those entities that allow further description of entities by specifying property dispositions. Attribute names denote the property dimension, its actual status for a given element is described by an attribute value. In <word pos="NN"> pos is the attribute, the exact value is NN. PCDATA are any characters which are not included in a pair of one opening (<) and one closing (>)angular bracket (see below).

Despite this general approach, XML does not provide the following:

typed/grammar specification of attribute values for the distinction of floats, characters or for the definition of attribute values by regular expressions or BNF grammars,

inference models for element values that allow for centralized specification of properties that are shared by more than one element,

applicability restriction of attributes that are mutual exclusive e.g. words that are nouns cannot have tempus information and case information is not applicable to verbs.

3.1 Minimal redundancy

is only sensible if the theory or the application assumes that there are vowels which can be voiceless (e.g. in the case of whispering).

In general, elements used for tagging should not carry the theory itself but that part of information that cannot be predicted only.
 

3.2 Maximal consistency

3.3 Universal parsability

• First, a general grammar should be used, supporting a uniform representation of different entity types. One example of this is XML which guarantees that any linguistic information can be parsed by one type of parser once it is encoded in XML.
• The second level of parsability refers to the actions or the meaning of the markup. This is the behaviour of a piece of software after having parsed the file. Examples of this behaviour may be the representation, display, and reactions to user input. This second level of parsability cannot be encoded my XML, it has to be defined elsewhere.
• A third level of parsability is the processing of information (stored in XML, processed and displayed by the computer) by human annotators: The information displayed must be accessible for them, too. Thus, additional information is needed to represent the actual meaning and theory behind the XML annotation applied to linguistic data.

3.4 Optimal maintainability

3.5 Naming conventions

3.6 Linking information

Adding annotation of the sentence level would either

• a) produce a (new) document which is a copy of the first one (or a new version of the first document) plus sentence tags (<s>) added

OR

• b) be a second file with <s> elements that hyperlink to the first one

As stated above, the second approach is recommended  which - depending on the DTD - treats the <w> elements as children of <s> elements like in variant a), but in a  non-invasive fashion [see [1]].
 
 

3.7 PCDATA

In this case, the elements are filled by PCDATA which we perceive as orthographical words. Sentence annotation building upon this would add <s> and </s> around the text before:
 

In this case, the <s> element is filled, too.

Empty elements are those which do not include neither other elements nore PCDATA, e.g. in the case of sentence annotation that refers to other elements by an href attribute:
 

It is recommended to use PCDATA only if these PCDATA are textual information (marked up text from source data). The use of PCDATA inside an element for specification of values is only preferable if these are very long and explicit.

In the following two examples there are examples where location information of a situation is provided. The second example is a case for choosing PCDATA.
 

All other information should be coded by attribute values, links and embedded elements.
 

3.8 Elements vs. Attributes

Or, see the following options for representing prosody and phrase types:
 

Phrases:
 

ToBI labels:
 

As a matter of convention, one should choose that level of abstraction that allows to segment a series of entities, name each of these entities by that one term which can be applied to all of them and encode their differences by attributes and values respectively. Note, that there is an interrelation between mutual exclusive attribute values and the choice of level of abstraction: If the description level and element type chosen is <w> a part-of-speech value of "noun" theoretically blocks the application of tense. If the element types chosen were <noun>, <verb>, <adj>, etc., this would certainly not happen, but the very information that all of these elements belong to one group of phenomena will be lost and is not exploitable for query access to the data. One further solution is the additional encoding of abstraction level information.
 

Attribute values should be chosen in a way that allows as much conceptualization as possible. Consider phonetic sounds: In the case where speech has to be segmented into sounds, one could think of a set of attributes specifying articulatory or auditive properties of these vowels as attributes, the values of which are set to plus (+) or minus (-). The alternative would obviously be to use the conventional IPA or SAM-PA symbols as values of a single attribute, e.g. "type". The first alternative has two disadvantages: If one assumes that the set of possible combinations of articulatory or auditive properties is known and the sounds to be segmented are limited to a small subset of all possible combinations, it seems an effort too hard to choose this option. The other reason is that there might be mutual dependencies of the values e.g. no plosive can be rounded. These dependencies cannot be constrained automatically  by the grammar of a DTD or so. Thus it is highly recommended to choose that set of attributes that guarantees maximal mutual independence of the attributes, i.e. find entity descriptions which are used to encode typical attribute-value constellations, in this case sound symbols. It might not always be possible to find a set attributes that are not mutually dependent, cf. the word attribute CASE which is not applicable to verbs as discussed above.
 

3.10  Time information

There are various options for the encoding of this information:

• First of all, one has to decide between specifying time in samples (signal measurement points) or seconds/miliseconds: The advantage of using samples is that samples are the finest grains available relative to the sample frequency used for a given segment of recorded speech. The disadvantage of using samples is that it is more difficult to compare time relations across documents: In order to access time, the sample frequency would have to be available for its calculation.
• Second, the properties - and attributes respectively - to use have to be chosen. In principle one could either employ start and end or duration. As duration can be calculated by the other two properties, but not the other way around, start and end seem more appropriate and are recommended as standard attributes. Yet, for some elements this concept seems difficult to apply, e.g. in the case of f0 values, door slams or other events which conceptually do not seem to have an extension in time. However, to be consistent, these elements should have the same attributes with the specialty that the values of the start and the end attribute are equal.
• Third, one could argue to leave one of the attributes out because many times, the start time of an element equals the end time of the element before. Unfortunately, this is not always the case, e.g., if there are pauses, utterances of other speakers, or simply because the element is located at the beginning or the end of the event chain. Some of these problems could only be solved if one would have all physical events listed and annotated in one document, what would require very complex DTDs and massive efforts in the handling of information. As it seems easier to keep elements of different conceptual levels in separate files, and it would be more complicated for the user or the software to decide when to include this information and when not, start and end attribute should always appear ensemble in an element tag.
• The fourth option touches the completeness of time information for every element on every level or whether time information could be inherited. For time information, one would want to provide start and end on only one layer and make elements from higher levels of description (e.g., words) that are conceptually related to these units (e.g., phones) point to that information or inherit it.  Basically two options can be considered:

• There is a special kind of attribute that allows to inherit values of other attributes, such that one can say that the value of the startinh attribute of YZ elements inherits that information from the start attributes of XY elements:

 

yz.xml

<xy id="xy001" start="000" end="002"/> <xy id="xy002" start="002" end="005"/> <xy id="xy003" start="005" end="009"/>


 

xy.xml

<yz startinh="xy001.start"/>

• There is no explicit start or end attribute in higher level elements at all but for the sake of getting this information, a processor has to go down in the element hierarchy and check all first children (of first children)* until this information is found specified somewhere. Exactly the same procedure is used for the end value, with the only difference that one uses the last children (of children)* in that case.

 

xy.xml

<xy id="xy001" start="000" end="002"/> <xy id="xy002" start="002" end="005"/> <xy id="xy003" start="005" end="009"/>

yz.xml

<yz href="xy.xml#id(xy001)..id(xy003)"/>

The second variant is recommended: If two levels of description are situated in the same hierarchy and the elements of one of them are parent elements of the other, then it is proposed to note start and end information for every element of the lower or lowest level in that conceptual hierarchy. The attributes start and end of the higher level elements can be calculated by evaluating the start value of the first embedded element and the end value of the last embedded element [2]. This in accordance with the two principles stated above, namely minimal redundancy and maximal consistency. If the time information is changed and all time information were kept separately, i.e. put to every level individually, then information of each element on each level would have to be changed. If there is only one basic representation all other tags refer to, and time information of these tags is inferred form the basic tag, then the time information has only to be changed once and changes applied to the time values on the lowest level will automatically affect the time specification of the associated higher levels.

In the following example, the start value of the sentence (<s>) would be 0.01 and the end value would be 0.62.
 

 

time.xml

<s id="s_001">   <w id="w_001" start="0.01" end="0.20>It</w>   <w id="w_002" start="0.20" end="0.37>was</w>   <w id="w_003" start="0.37" end="0.42>time</w>   <w id="w_004" start="0.42" end="0.62>again</w> </s>

To exploit this principle most effectively, time information and thus the initial transcription of spoken material should be applied to the level with the smallest units under investigation (cf. the section on cross-level).

Time is relative and if an element has the start value of 4.34 which is to be compared to the start value of other elements in another hierarchy, this may cause problems as this time information will in both cases refer to the time elapsed relative to the beginning of the document they reside in only but may be incompatible for many cases. It is recommended that time is either specified relative to midnight, January 1, 1970 UTC (universal time code) or relative to the beginning of the recording. For the latter case it is useful to apply the special attribute rectime that states the time distance of the beginning of the recording relative to midnight, January 1, 1970 UTC. Even if a software is not able to calculate and compare the time information of elements of different documents, it will be easy to check if the documents are compatible, and thus if a query that compares start and end information of these documents is sensible.
 
 

References

[1]Isard, A., McKelvie, D. and Thompson, H.S.: Towards a Minimal Standard for Dialogue Transcripts: A New Sgml Architecture for the HCRC Map Task Corpus. Proceedings of the 5th International Conference on Spoken Language Processing, ICSLP98, Sydney. http://www.cogsci.ed.ac.uk/~dmck/Papers/icslp98.ps

[2] The SGML/XML Web Page: SGML/XML Elements versus Attributes. http://www.oasis-open.org/cover/elementsAndAttrs.html