TOWARDS AN EVENT ANNOTATED CORPUS OF POLISH

The paper presents a typology of events built on the basis of TimeML speciﬁcation adapted to Polish language. Some changes were introduced to the deﬁnition of the event categories and a motivation for event categorization was formulated. The event annotation task is presented on two levels — ontology level (language independent) and text mentions (language dependant). The various types of event mentions in Polish text are discussed. A procedure for annotation of event mentions in Polish texts is presented and evaluated. In the evaluation a randomly selected set of documents from the Corpus of Wrocław University of Technology (called KPWr) was annotated by two linguists and the annotator agreement was calculated. The evaluation was done in two iterations. After the ﬁrst evaluation we revised and improved the annotation procedure. The second evaluation showed a signiﬁcant improvement of the agreement between annotators. The current work was focused on annotation and categorisation of event mentions in text. The future work will be focused on description of event with a set of attributes, arguments and relations.


Introduction
Event recognition is a subtask of information extraction task.The goal of information extraction is to understand the meaning of a text at some level on which one can catch given type of information and present it in a structured manner.Event recognition focuses on finding in text references to some situations and extracting their descriptions.Event recognition has practical applications in many tasks from the field of natural language processing, like text summarization (Maybury, 1995), discourse analysis, events aggregation and reporting (Vossen et al., 2014;van Erp, Fokkens, & Vossen, 2014;Agerri et al., 2014).Within the Clarin-PL project 1 we plan to develop methods and tools for event recognition for Polish.We want to identify event mentions in Polish texts, categorise them on a coarse-grained level and identify event attributes, arguments and relations in order to enable deeper text understanding.In order to create and evaluate such tools we need a practical guideline for event annotation dedicated to Polish and a corpora annotated with events.According to our best knowledge, the most popular and widely used specification for event annotation is TimeML Annotation Guidelines Version 1.2.1 (Saurí, Littman, Knippen, Gaizauskas, Setzer, & Pustejovsky, 2006) (henceforth, TimeML).The specification has been already adopted to several languages, including Spanish (Saurí, Batiukova, & Pustejovsky, n.d.), Catalan (Saurí, Batiukova, & Pustejovsky, n.d.), French (Bittar, 2010) and Italian (Caselli, Bartalesi Lenzi, Sprugnoli, Pianta, & Prodanof, 2011).Applying an existing guideline for another language requires a careful study of that language phenomena and might need some adjustments concerning language-specific issues.In the following sections we present results of our work on adaptation the TimeML specification to Polish language and evaluation of the specification on Polish texts.In Section 2 we present a definition of event concept and what we understand as an event on the ontology level.Section 3 contains a typology of event categories and motivation for event categorization.In Section 4 we define the event mentions for Polish as a text-level representation of events.In Section 5 we present results of two evaluations of the guidelines for Polish performed on the KPWr corpora (Broda, Marcińczuk, Maziarz, Radziszewski, & Wardyński, 2012).In Section 6 we present a detailed procedure for event mention annotation which was created as a result of first evaluation.Section 7 presents a summary of our current work and future plans including event description with attributes, arguments and relations.

What is an event?
Event is one of the primary concepts in almost any upper-level ontology.According to the Oxford Dictionary event "is a thing that happens or takes place". 2In the Suggested Upper Merged Ontology ontology (Pease, 2011) (henceforth, SUMO) an event is represented as a concept called Process, 3 which is defined as following: "The class of things that happen and have temporal parts or stages.Examples include extended events like a football match or a race, actions like pursuing and reading, and biological processes.The formal definition is: anything that occurs in time but is not an object.Note that a process may have participants 'inside' it which are objects, such as the players in a football match" In other words event is anything that takes place in time (date, time and/or duration) and space (has a location), may involve agents (executor or participants), may contain or be part of other events and may produce some outcome (object).In our work we will consider as event all situations which can be mapped onto the Process concept or any concept which is a subclass of Process in the SUMO ontology.The relations of event with other concepts from the SUMO ontology are illustraded on the Figure 1.The states are also treaded as events (Saurí et al., 2006) but they have a specific ontological status.It isn't simple to map the words denotating a state to Process in the SUMO ontology.As Vendler said suggestively, states are "that puzzling category in which the role of verb melts into that of predicate, and actions fade into qualities and relations" (Vendler, 1957, p. 109).This feature brought us to individual treatment of the mentions of the states.

Event Categories
We used seven coarse-grained categories of events, i.e. action, state, reporting, perception, aspectual, intensional action and intensional state.The categorisation was based on the TimeML guideline with some modifications.Instead of the occurrence term we used action.The occurrence category from TimeML refers only to specific temporaly located events.Generics -actions which refer to some general rules (for example, a boil event in sentence "Water boils in 100 • C") are not tagged.We noticed that the distinction between specific and generic events can be applied to any category of events what indicates that the event generality should be defined as an event attribute rather than its category.Taking into account Polish terminological tradition (e.g.Laskowski, 1998), we've decided to use the term action instead occurrence, 4 to accent its generality and to make visible the key opposition between the two core categories: action and state.In addition this change emphasizes the distinction between the state/action and intensional state/action.The remaining categories can be treated as auxiliary categories, as they refer to another events and introduce some additional information about the event.
TimeML specification doesn't introduce the higher level classification.Still the categories of events can be divided into four groups in respect to two factors: dynamicity (course in time) and event argument (see Table 1).The course in time factor divides events into static and dynamic events.The static events endure or persists over some period of time and though they may provide the potential change, they do not constitute a change (Mourelatos, 1978, p. 192).The event argument factor indicates if the event have (or might have) an argument that is an event.For example the start event indicates the beginning of some other event.
Table 1: Groups of event categories.

Without an event argument
With an event argument Static state intensional state Dynamic action perception reporting aspectual intensional action In addition we've noticed that actions or states which connect with an event argument could be divided into two groups.Occurring some of the events in the text (i.e., perception, reporting and aspectual ) signals that an event which is an argument occurred (or should have occurred) in the real world and occurring the other (i.e.intensional state and intensional action) doesn't gave such certainty.It wasn't the statement of TimeML specification authors but we treat it as the important remark for the future processing of extracted events.
Furthermore, we've decided to introduce a separate category for synsemantic verbs that occur with nominalizations (light predicates).Since they have specific grammatical function, they are described in the section on event mentions.
Figure 2 shows the final classification of events.

Action
Action represents a dynamic situation which occurrs in time and space.The event could have some type of outcome that can be a product, achievement or change from one state to another.Examples: build, dance, jump, running

State
State represents a static situation.It refers to object attributes (Apresjan, 2000, pp. 47-48) or situations which are stable and does not change over given period of time (Laskowski, 1998, p. 153).

Reporting
Reporting refers to a dynamic situation where an agent inform about an event or narrate an event.If the reporting refers to an action or a state then it is a strong indication that the action or state took place or was true.

Perception
Perception refers to a physical perception of an event by an agent.This class indicates that the agent was an observer of the event.The perception event is a strong indicator that the observed event took place or was true.

Aspectual
Aspectual refers to a dynamic situation which indicates a change of a phase of another event.The change can be (following TimeML): 1. Initiation -an event was started, 2. Reinitiation -an event was stopped and started again, 3. Termination -an event was stopped before it was completed, 4. Culmination -an event was completed, 5. Continuation -an event is continued.
If the aspectual event refers to an action or a state and it is not referred by any intensional action or state, then it is a strong indication that the action or state was true for some period of time.

Intensional action
Intensional action is a situation where an agent declare his or her will to perform an action or give a command to another agent to perform an action.We cannot infer if the action was or will be performed in the future.The possible groups of intensional actions (following TimeML) are: 1. Attempt -the agent tried to do X but failed to accomplish it.2. Delay -the agent postpone some action in time.
3. Avoid -the agent prevent same action which may happen.4. Ask -the agent asks somebody to do something.5. Promise -the agent promises to do something.6. Propose -the agent propose to do something.
Comparing to TimeML we removed two groups of events from this category, i.e.: investigation (investigate, delve) and naming (name, nominate, appoint, etc.).Those two groups does not require any other event as an argument, thus they can be treated as an action.Examples: try, delay, promise, ordering 3.7.Intensional state Intensional state is a state which refers to some possible actions or states.It indicates, than an agent refers to some possible event, which may or may not occur in the future.Most of the intensional states are connected with mental activities, emotions and needs.The possible groups of intensional states (following TimeML) are: a great impact on the future classifications including Polish tradition.For Laskowski (1998, p. 152-153), situation is a denotate of sentence constituted by verb.
It was important to introduce a method of annotating periphrastic predication.Ewa Jędrzejko points out several types of complex predicates (Jędrzejko, 2011, pp. 34-37): • Standard nominal predicates [VCOP + NKONKR // Nabstr // Adj // Adv], • The so-called modal predicates [VMOD + VINF] + ..., • The phase-aspectual complex predicates [VFAZ + V//NA] + ..., • The most common type of the VNA with basic 'generic' verbs [VGENER + NA//NE//Nabstr] + ..., • Periphrastic predicates in the strict sense of the term [VMETAF// METAPRED + Nabstr//NA//NE ] + ...,6 • So-called phraseological predicates, i.e. 'typical' idioms functioning as verbs [Vmetafor + N + 3 We've decided to exclude some verbs from annotation.Tagged elements should introduce enough information to classify the situation.Our assumption was that copulae and other auxiliary verbs (e.g.components of analytic future tense) are semantically (referentially) empty so they are not very useful for event extraction.Verbal part of modal predicates, phase-aspectual predicates and predicates with generic verbs may be called light predicate or light verb (Jespersen, 1965).According to Zolotova, Onipenko, and Sidorova (1999) they are modifiers (phase and modal) and compensators (accompanied by deverbal noun) There is no agreement concerning the definition and the semantics of light verbs (Kotsyba, 2014) but it was valid to include them to annotation as they carry a grammatical and very general but sufficient lexical meaning.We tag both elements of such predicates because they are relevant to different kind of event information -after that two tagged events will be linked as identical.
We have made an exception for one category.At the first stage of annotation we don't mark the nominalizations for the states due to the specific ontological status of this situations and the features of their nominalizations (Mourelatos, 1978, pp. 204-210).It is an open question if we need to recognize them in the future.

Other mentions
Taking into account the scope of our task it was important to consider all predicative expressions.Jodłowski (1976, pp. 31-33) introduced one of the first and basic classification.It includes many types of nonverbal predicates such as accent, intonation, context, pause or adverbs.Still, identification of these mentions would require context analysis or some additional data (e.g., conversational), so we decided to exclude them from annotation.
As Saurí et al. (2006) stated events may be expressed by adjectives.Some of them are nominalizations so it is valid to annotate them.We have decided that the mentions that introduce other situations (i.e. that have an event argument) are the most important.

Annotators Agreement
The inter-annotator agreement was measured on randomly selected documents from the Corpus of Wrocław University of Technology called KPWr (Broda et al., 2012).We used the positive specific agreement (psa) (Hripcsak & Rothschild, 2005) as there are no negative decisions to count to measure the agreement between two linguists.The documents were annotated using the Annotator perspective from the Inforex system8 (see Figure 4) (Marcińczuk, Kocoń, & Broda, 2012).In the first iteration we randomly selected 100 documents.The results are presented in Table 2.The agreement for event mentions without categorisation was ca.85% and with categorisation it drops to 68%.The results show, that the most confusing categories were state (36.98%) and light predicate (39.60%).The best agreement was achieved for aspectual (86.79%) and action (77.89%).We have carefully analyzed the discrepancies between annotators and introduce some clarifications in the guideline.The major changes were: • Categories which require another event as an argument (aspectual, perception, reporting, intensional action and intensional state) can be annotated regard- less the argument is directly stated in the sentence or not (for example the event argument might be omitted or referenced by a pronoun).• We have formulated a procedure for recognition state mentions.We have defined the following criteria: • If the mention as a lexical unit is present in the plWordNet9 (Piasecki, Szpakowicz, & Broda, 2009) then the synset containing the lexical unit must be a direct or indirect hyponym of an artificial synset call "state verb".• State has no dynamic.
• State does not change in time.
• Verb representing state has imperfect aspect.
• Verb representing state does not have an perfective form.
• Passive construction does not indicate a state.Event category results from the semantic of the verb, not the grammatical construction.For example sentences "John was killed" and "Tom killed John" represent the same action of killing a person named John.• We defined a procedure to determine if given mention should be annotated and with what category.The procedure consists of a set of yes-no questions.
The procedure is presented in Section 6.
Next, we have performed a second evaluation to check, if the clarifications improved the agreement between annotators.In the second iteration we randomly selected another set of 50 documents.The documents were annotated by the same two linguists.The results for second evaluation are presented in Table 3.The agreement for mention annotation without categorisation raised from 85% to 93%.There were noticeable improvement in annotation of the categories which were annotated with unacceptable agreement (state and light predicate) in the first iteration (form 36.98% to 74.26% for state; from 39.60% to 50% for light predicate).The overall agreement in the second evaluation was significantly higher.However, there are

Conclusions and future work
The evaluation showed that the annotation of event mentions is relatively simplethe agreement after second iteration was 93%.The categorisation of event mentions causes more problems and the agreement drops to 82%.This shows that the task is not trivial and if we want to obtain a good quality of data with high agreement the final annotation of whole KPWr corpus will require the "2+1" approach.This means that each document in the corpus will be annotated separately by two linguists and the differences will be evaluated by a third linguist -supervisor.The future plan is to prepare guidelines for event description with attributes, arguments and relations.The attributes, we are considering, are: • generality -is the event specific or general, • polarity -is the form of the mention affirmative or negative, • modality -is it assertoric, optative, imperative or interrogative, • tense and aspect Most of them (in particular generality) have to be annotated manually.Although there are tools10 that could be used for automatic annotation of tense and aspect.
The events will be linked with their generic arguments, i.e. agent, time and location.In the last step we will mark the relations between the events.The categories of relations include identity and references between reporting, perception, aspectual, intensional action and intensional state and their event arguments.

Figure 1 :
Figure 1: Event relations with other concepts from SUMO ontology.

Table 2 :
Agreement between two linguists (A and B) after first iteration.

Table 3 :
Agreement between two linguists (A and B) after second iteration.