Keynote Speakers
-
Prof. Charles L. A. Clarke
University of WaterlooEvaluating Generative Information Retrieval Systems
Traditionally, information retrieval systems have been evaluated with a combination of implicit feedback from user interactions, explicit user feedback, and paid assessment by human labellers. Over the past year, both academic researchers and commercial services have begun to employ large language models to replace and augment human assessment for IR evaluation. Over the same period, generative information retrieval (Gen-IR) systems and retrieval augmented generation (RAG) systems have emerged as alternatives to traditional IR systems. The role of IR evaluation has always been to answer questions like, “Is this system better than that one?” and “Does this search result satisfy the searcher’s information need?” In a world where large language models can outperform human labellers on basic relevance assessment tasks, and where the results of a search are used only as input to a generative system, how do we answer these questions, and are these even the right questions to ask? While we can now build systems we could only dream about a few years ago, we still need to know if changes to a system reflect genuine benefits to its users. In this talk, I will survey the landscape as it exists in March 2024, and talk about directions forward.
-
Prof. Josiane Mothe
Institut de Recherche en Informatique de Toulouse; INSPE, Université de ToulouseShaping the Future of Endangered and Low-Resource Languages: Our Role in the Age of LLMs
Isidore of Seville is credited with the adage that it is language that gives birth to a people, and not the other way around, underlining the profound role played by language in the formation of cultural and social identity. Today, of the more than 7100 languages listed, a significant number are endangered. Since the 1970s, linguists, information seekers and enthusiasts have helped develop digital resources and automatic tools to support a wide range of languages, including endangered ones. The advent of Large Language Model (LLM) technologies holds both promise and peril. They offer unprecedented possibilities for the translation and generation of content and resources, key elements in the preservation and revitalisation of languages. They also present threat of homogenisation, cultural oversimplification and the further marginalisation of already vulnerable languages. This talk proposes an initiatory journey, exploring the potential paths and partnerships between technology and tradition, with a particular focus on the Occitan language. Occitan is a language from Southern France, parts of Spain and Italy that played a major cultural and economic role, particularly in the Middle Ages. It is now endangered according to UNESCO. This talk critically examines how human expertise and artificial intelligence can work together to offer hope for preserving the linguistic diversity that forms the foundation of our global and especially our European heritage while addressing some of the ethical and practical challenges that accompany the use of these powerful technologies.
-
Prof. Carlos Castillo
Universitat Pompeu FabraHuman Factors and Algorithmic Fairness
In this talk, we present ongoing research on human factors of decision support systems that has consequences from the perspective of algorithmic fairness. We study two different settings: a game and a high-stakes scenario. The game is an exploratory “oil drilling” game, while the high-stakes scenario is the prediction of criminal recidivism. In both cases, a decision support system helps a human make a decision. We observe that in general users of such systems must thread a fine line between algorithmic aversion (completely disregarding the algorithmic support) and automation bias (completely disregarding their own judgment). The talk presents joint work led by David Solans and Manuel Portela.