DOI: https://doi.org/10.11649/cs.1640

Aspects of gestural alignment in task-oriented dialogues

Maciej Karpiński, Agnieszka Czoska, Ewa Jarmołowicz-Nowikow, Konrad Juszczyk, Katarzyna Klessa

Abstract


Aspects of gestural alignment in task-oriented dialogues

Interlocutors in a conversation influence each other in a number of dimensions. This process may lead to observable changes in their communicative behaviour. The directions and profiles of these changes are often correlated with the quality of interaction and may predict its success. In the present study, the gestural component of communication is scrutinised for changes that may reflect the process of alignment. Two types of task-oriented dialogues between teenagers are recorded and annotated for gestures and their features. We hypothesize that the dialogue task type (collaborative vs. competitive), as well as certain culture-specific properties of alignment that differ between German and Polish pairs, may significantly influence the process of communication. In order to explore the data and detect tendencies in gestural behaviour, automatised annotation mining and statistical exploration have been used, including a moving frame approach aimed at the investigation of co-occurring strokes as well as re-occurring strokes and their features. Significant differences between German and Polish speakers, as well as between the two dialogue types, have been found in the number of gestures, stroke duration and amplitude.

 

Aspekty wzajemnego dopasowania gestykulacji w dialogach zadaniowych

Uczestnicy konwersacji wpływają na siebie wzajemnie w wielu różnych wymiarach. Proces ten może prowadzić do obserwowalnych zmian w ich zachowaniach komunikacyjnych. Kierunek i charakter tych zmian często wiążą się z jakością interakcji i mogą pozwolić przewidzieć jej sukces. Celem niniejszego badania jest eksploracja gestowego składnika komunikacji pod kątem zjawisk, które mogą odzwierciedlać zjawisko wzajemnego dopasowania uczestników. Zarejestrowano dwa typy dialogów zadaniowych z udziałem nastolatków. Nagrania anotowano pod kątem gestów i ich cech. W badaniu postawiono hipotezę, iż typ zadania dialogowego (ukierunkowane na współpracę vs. na konkurencję), jak również pewne specyficzne cechy kulturowe dopasowania komunikacyjnego, które różnicują polskie i niemieckie pary rozmówców, mogą oddziaływać na proces komunikacji. W celu eksploracji danych oraz wykrycia tendencji cechujących zachowania gestowe, wykorzystano zautomatyzowane metody analizy anotacji oraz techniki statystyczne. Uwzględniono podejście oparte na ruchomych oknach czasowych, umożliwiające analizę współwystępowania jak i powtarzania gestów właściwych i ich właściwości. Wykazano istotne różnice między rozmówcami polskimi i niemieckimi oraz między typami zadań dialogowych w zakresie liczby gestów, czasu trwania oraz amplitudy gestu właściwego.


Keywords


communicative alignment; intercultural communication; multimodal interactions; gesture function; gesture re-occurrence; gesture co-occurrence

Full Text:

PDF (in English)

References


Beňuš, Š., Gravano, A., & Hirschberg, J. (2011). Pragmatic aspects of temporal accommodation in turn-taking. Journal of Pragmatics, 43(12), 3001-3027. https://doi.org/10.1016/j.pragma.2011.05.011

Bonacchi, S., & Karpiński, M. (2014). Remarks about the use of the term "multimodality". Journal of Multimodal Communication Studies, 1, 1-7.

Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6), 1482-1493. https://doi.org/10.1037/0278-7393.22.6.1482

Campbell, N., & Scherer, S. (2010). Comparing measures of synchrony and alignment in dialogue speech timing with respect to turn-taking activity. In Proceedings of the 11th Annual Conference of the International Speech Communication Association. Retrieved from https://www.isca-speech.org/archive/interspeech_2010/index.html

Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception-behavior link and social interaction. Journal of Personality and Social Psychology, 76(6), 893-910. https://doi.org/10.1037/0022-3514.76.6.893

Chartrand, T. L., Maddux, W. W., & Lakin, J. L. (2006). Beyond the perception-behavior link: The ubiquitous utility and motivational moderators of nonconscious mimicry. In R. R. Hassin, J. S. Uleman, & J. A. Bargh (Eds.), The new unconscious (pp. 334-361). Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195307696.003.0014

Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127-149). Washington, DC: American Psychological Association. https://doi.org/10.1037/10096-006

Cleland, A. A., & Pickering, M. J. (2003). The use of lexical and syntactic information in language production: Evidence from the priming of noun-phrase structure. Journal of Memory and Language, 49(2), 214-230.

Czoska, A., Klessa, K., Karpiński, M., & Nowikow-Jarmołowicz, E. (2015). Prosody and gesture in dialogue: Cross-modal interactions. In Proceedings of 4th Gesture and Speech in Interaction (GESPIN) Conference, Nantes, France (pp. 83-88). Retrieved from https://hal.archives-ouvertes.fr/hal-01195646/document

Fusaroli, R., Rączaszek-Leonardi, J., & Tylén, K. (2014). Dialog as interpersonal synergy. New Ideas in Psychology, 32(January-April), 147-157. https://doi.org/10.1016/j.newideapsych.2013.03.005

Garrod, S., & Pickering, M. J. (2013). Interactive alignment and prediction in dialogue. In I. Wachsmuth, J. de Ruiter, P. Jaecks, & S. Kopp (Eds.), Alignment in communication: Towards a new theory of communication (pp. 193-204). Amsterdam: John Benjamins Publishing Company. (Advances in Interaction Studies, 6). https://doi.org/10.1075/ais.6.10gar

Giles, H., & Smith, P. (1979). Accommodation theory: Optimal levels of convergence. In H. Giles & R. N. St. Clair (Eds.), Language and social psychology (pp. 45-65). Baltimore, MD: University Park Press.

Giles, H., Taylor, D. M., & Bourhis, R. (1973). Towards a theory of interpersonal accommodation through language: Some Canadian data. Language in Society, 2(2), 177-192. https://doi.org/10.1017/S0047404500000701

Gravano, A., Levitan, R., Willson, L., Beňuš, Š., Hirschberg, J., & Nenkova, A. (2011). Acoustic and prosodic correlates of social behavior. In 12th Annual Conference of the International Speech Communication Association 2011 (Interspeech 2011). Retrieved from http://www.cs.columbia.edu/nlp/papers/2011/p95485.pdf

Guitar, B., & Marchinkoski, L. (2001). Influence of mothers' slower speech on their children's speech rate. Journal of Speech, Language, and Hearing Research, 44(4), 853-861. https://doi.org/10.1044/1092-4388(2001/067)

Healey, P. G. T., Purver, M., & Howes, C. (2014). Divergence in dialogue. PLoS ONE, 9(6). https://doi.org/10.1371/journal.pone.0098598

Holle, H., & Rein, R. (2015). EasyDIAg: A tool for easy determination of interrater agreement. Behavior Research Methods, 47(3), 837-847. https://doi.org/10.3758/s13428-014-0506-7

Jaecks, P., Damm, O., Hielscher-Fastabend, M., Malchus, K., Stenneken, P., & Wrede, B. (2013). What is the link between emotional and communicative alignment in interaction?: Towards a new theory of communication. In I. Wachsmuth, J. de Ruiter, P. Jaecks, & S. Kopp (Eds.), Alignment in communication: Towards a new theory of communication (pp. 205-224). Amsterdam: John Benjamins Publishing Company. (Advances in Interaction Studies, 6). https://doi.org/10.1075/ais.6.11jae

Karpiński, M. (2014). New challenges in psycholinguistics: Interactivity and alignment in interpersonal communication. Lingua Posnaniensis, 54(1), 97-106.

Karpiński, M., Jarmołowicz-Nowikow, E., & Czoska, A. (2015). Gesture annotation scheme development and application for entrainment analysis in task-oriented dialogues in diverse cultures. In Proceedings of 4th Gesture and Speech in Interaction (GESPIN) Conference, Nantes, France (pp. 161-166). Retrieved from https://www.academia.edu/15517265/Karpi%C5%84ski_M._Jarmo%C5%82owicz-Nowikow_E._Czoska_A._2015._Gesture_annotation_scheme_development_and_application_for_entrainment_analysis_in_task-oriented_dialogues_in_diverse_cultures._Proceedings_of_GESPIN_2015_Conference_Nantes_France_161-166

Karpiński, M., Klessa, K., & Czoska, A. (2014). Local and global convergence in the temporal domain in Polish task-oriented dialogue. In N. Campbell, D. Gibbon, & D. Hirst (Eds.), Proceedings of the 7th International Conference on Speech Prosody (pp. 743-747). Retrieved from https://www.academia.edu/7151184/Karpinski_M._Klessa_K._Czoska_A._2014._Local_and_global_convergence_in_the_temporal_domain_in_Polish_task_oriented_dialogue._In_N._Campbell_D._Gibbon_and_D._Hirst_Eds._Speech_Prosody_7_2014_Dublin_pp._743-747

Karpiński, M., & Klessa, K. (2018). Methods, tools and techniques for multimodal analysis of accommodation in intercultural communication. Computational Methods in Science and Technology, 24(1), 29-41. https://doi.org/10.12921/cmst.2018.0000006

Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511807572

Klessa, K., Karpiński, M., & Wagner, A. (2013). Annotation Pro - a new software tool for annotation of linguistic and paralinguistic features. In B. Bigi & D. Hirst (Eds.), Proceedings of TRASP (Tools and Resources for the Analysis of Speech Prosody) (pp. 51-54). Aix-en-Provence: Aix-Marseille Université.

Kousidis, S. (2010). A study of accommodation of prosodic and temporal features in spoken dialogues in view of speech technology applications (Doctoral thesis). Dublin Institute of Technology.

Kousidis, S., Dorran, D., Wang, Y., Vaughan, B., Cullen, C., Campbell, D., McDonnell, C., & Coyle, E. (2008). Towards measuring continuous acoustic feature convergence in unconstrained spoken dialogues. In Proceedings of Interspeech 2008 (pp. 1692-1695).

Kulesza, W. (2016). (Nie)świadomy kameleon: Analiza związku między stosowaniem niewerbalnej mimikry, uległością wobec tego procesu a (nie)świadomością. Psychologia Społeczna, 11(2(37)), 183-195. https://doi.org/10.7366/1896180020163705

Lausberg, H. (2013). NEUROGES - A coding system for the empirical analysis of hand movement behaviour as a reflection of cognitive, emotional, and interactive processes. In C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeill, & S. Tessendorf (Eds.), Body-language-communication: An international handbook on multimodality in human interaction (Vol. 1, pp. 1022-1036). Berlin: De Gruyter Mouton. https://doi.org/10.1515/9783110261318.1022

McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago, IL: University of Chicago Press.

Mol, L., Krahmer, E., Maes, A., & Swerts, M. (2012). Adaptation in gesture: Converging hands or converging minds? Journal of Memory and Language, 66(1), 249-264. https://doi.org/10.1016/j.jml.2011.07.004

Müller, C. (1998). Redebegleitende Gesten: Kulturgeschichte, Theorie, Sprachvergleich. Berlin: Berlin-Verl. Spitz. (Körper, Zeichen, Kultur, 1).

Pardo, J. S. (2006). On phonetic convergence during conversational interaction. The Journal of the Acoustical Society of America, 119(4), 2382-2393. https://doi.org/10.1121/1.2178720

Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. The Behavioral and Brain Sciences, 27(2), 169-190. https://doi.org/10.1017/S0140525X04000056

Ramseyer, F., & Tschacher, W. (2011). Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome. Journal of Consulting and Clinical Psychology, 79(3), 284-295. https://doi.org/10.1037/a0023419

Richardson, D. C., Dale, R., & Tomlinson, J. M. (2009). Conversation, gaze coordination, and beliefs about visual context. Cognitive Science, 33(8), 1468-1482. https://doi.org/10.1111/j.1551-6709.2009.01057.x

Sloetjes, H., & Wittenburg, P. (2008). Annotation by category: ELAN and ISO DCR. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). Retrieved from http://pubman.mpdl.mpg.de/pubman/item/escidoc:60774:3/component/escidoc:60775/Sloetjes_2008_annotation.pdf

Truong, K. P., & Heylen, D. (2012). Measuring prosodic alignment in cooperative task-based conversations. In INTERSPEECH-2012 (pp. 843-846), Portland, OR. Retrieved January 31, 2018, from https://www.isca-speech.org/archive/archive_papers/interspeech_2012/i12_0843.pdf

Ward, A., & Litman, D. J. (2007). Automatically measuring lexical and acoustic/prosodic convergence in tutorial dialog corpora. In Workshop on Speech and Language Technology in Education. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.137.6046&rep=rep1&type=pdf

Ward, A., & Litman, D. J. (2007). Measuring convergence and priming in tutorial dialog. University of Pittsburgh.

Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006). ELAN: A professional framework for multimodality research. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006) (pp. 1556-1559). Retrieved from http://www.lrec-conf.org/proceedings/lrec2006




Copyright (c) 2018 Maciej Karpiński, Agnieszka Czoska, Ewa Jarmołowicz-Nowikow, Konrad Juszczyk, Katarzyna Klessa

License URL: http://creativecommons.org/licenses/by/3.0/pl/