nle24-11.doc - A generic architecture and dialogue model...

Info icon This preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
A generic architecture and dialogue model for multimodal interaction D. Hofs, R. op den Akker & A. Nijholt Parlevink Research Group Centre of Telematics and Information Technology University of Twente, Enschede, the Netherlands {infrieks,anijholt}@cs.utwente.nl Abstract This paper presents a generic architecture and a dialogue model for multimodal interaction. Architecture and model are transparent and have been used for different task domains. In this paper the emphasis is on their use for the navigation task in a virtual environment. The dialogue model is based on the information state approach and the recognition of dialogue acts. We explain how pairs of backward and forward looking tags and the preference rules of the dialogue act determiner together determine the structure of the dialogues that can be handled by the system. The system action selection mechanism and the problem of reference resolution are discussed in detail. 1 Introduction To help users find their way in a virtual theatre we developed a navigation agent. In natural language dialogues in Dutch the agent can assist users, when they are looking for the location of an object or room, and he can show them the shortest route between any two locations in the theatre. The speech-based dialogue system allows users to ask questions such as ‘Where is the coffee bar?’ and ‘How do I get to the great hall?’ In addition to that the navigation agent has a map, on which he can mark locations and routes. Users can click on locations in the map and ask questions about them like ‘What is this?’ We introduce a rather generic architecture and dialogue model allowing speech recognition and multimodal interactions, including reference resolution modelling and subdialogue modelling. Backward and forward looking tags determine the structure of the dialogues. With different grammars for the speech recogniser and different implementations of the interpreters and clients for dialogue input and output, the same architecture was used for two different dialogue systems. In addition to the navigation application we used the speech architecture for Jacob, a tutor for the Towers of Hanoi, that earlier did not allow speech recognition (Evers and Nijholt 2000). In section 2 we describe the architecture of the system. In section 3 we describe the dialogue manager and related components in the navigation agent. In section 4 we have a short discussion about the information state approach versus the dialogue state approach in dialogue modelling and we present some conclusions.
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
2 A multimodal dialogue system We present a multimodal dialogue system that the user can communicate with through speech and mouse input. The system can communicate with the user through speech and images. Initially the architecture can be conceived as a box containing a dialogue manager and world knowledge, which can be connected with input processors and output generators.
Image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern