Multimodal Knowledge acquisition
What are the common uses of objects? Which perceptual attributes are characteristic of them? Which object(s) can one use to achieve certain goals? Which steps should one follow to attain the goals? This kind of knowledge comprises a vital part of our prior knowledge of the world, which we acquire developmentally, through multimodal exploration of the world and interaction with others. Acquiring such knowledge is a pre-requisite for the development of intelligent systems, especially ones that engage into interaction with humans or ones that involve multimodal information processing. However, hardly is such knowledge explicit; it is considered “common knowledge”, “easily implied information” for humans. What about machines though? In human-computer/robot interaction, this is a core problem, that hinders communication. Humans imply information that the machines need to somehow acquire (as prior knowledge) to understand their intentions, instructions, goals. The lack of explicit/verbal expression of such prior knowledge of the world is one of the biggest challenges in modern Generative AI systems. Multimodal Knowledge Acquisition is inherently related to experimental and computational research in learning.
Our ultimate objective in this research line is the development of large scale multimodal knowledge acquisition algorithms for human-computer/robot interaction. An important step towards this direction is the development of a large-scale, human-elicited, multimodal knowledge graph based on the POETICON project Lithic Tool Experiments. We designed and run a series of cognitive experiments through which we elicited object affordance and attribute information on a large scale. In the experiments we used both visual and tactile stimuli comprising lithic tools, i.e., novel objects to the modern man, made for particular functions that formed part of the everyday activities of another age; as such, they proved to be ideal for eliciting rich information on everyday activities of the modern man in a completely unguided/unbiased way. The participants used free speech (verbal reports) for answering a simple question: which objects were they presented with and what they could use them for. The experiment had various conditions including visual presentation of the stimuli in a computer screen, versus active manipulation of a lithic tool collection. All sessions were recorded with two cameras (en face and profile). The resulting data comprise more than 95 hours of verbal reports, from more than 120 participants. It includes both verbally expressed semantic information comprising object attributes, affordances and related argumentation, as well as a rich set of pantomimes, gestures and exploratory acts.
This work has been published in Nature-Scientific Data; the rich set of multimodal data has been further annotated in several aspects and now forms part of the open source, POETICON Lithic Tool Experiments (PLT) Dataset Series, comprising 5 datasets. Furthermore, a stand-alone tool has been developed for the in-context exploration of the embodied concepts and relations of the knowledge graph (PLT Embodied Concepts and Relations Exploration and Clustering Tool). The knowledge graph is currently being translated from Greek to English and French.