On August 20, the 37th Tsinghua Software Forum invited Giuseppe Carenini, Professor at the University of British Columbia and Director of the Master in Data Science, to deliver an academic talk entitled “Gaze-Guided Attention and Self-Generated Visualizations in Large Vision-Language Models.” The forum was held in a hybrid format, with faculty members and students from the School of Software, as well as researchers in artificial intelligence and information visualization from other universities, attending in person or online.

Prof. Carenini gave an academic talk

Carenini began by providing an overview of research at the intersection of natural language processing and information visualization. He reviewed recent advances in large vision-language models in “visualization-to-language” tasks, such as chart question answering, and “language-to-visualization” tasks, such as visualization generation. Building on this overview, Carenini presented two recent research directions. The first explored how human eye-gaze data could be used to guide model attention. The research team developed ChartGaze, an eye-tracking dataset, and proposed a gaze-guided attention refinement method that directed models toward chart regions relevant to the questions, thereby improving chart question-answering accuracy and interpretability. Subsequent work further applied the method to misleading charts to investigate whether human gaze supervision could enhance model robustness across different visual designs. The second research direction examined whether large vision-language models could support their own reasoning by generating visualizations. Carenini introduced VAMPS, a benchmark for visual-assisted mathematical problem solving. The experiments showed that, even for problems naturally suited to graphical solutions, direct analytical reasoning still outperformed reasoning with self-generated visualizations for most models, revealing continuing limitations in their ability to generate, understand, and use visual information. Finally, Carenini noted that visualization does not inherently improve the reasoning capabilities of large vision-language models. To fully realize the benefits of integrating natural language and information visualization, models must learn to attend to relevant visual information and to generate and interpret visualizations reliably.

Scene of the meeting

During the subsequent discussion session, participants raised questions about the role of eye-gaze data in model training, the interpretability of model attention, and the limitations of visualization tools in supporting reasoning. Carenini addressed each question in detail.