KTH
Interaction and
Robotics Lab

SAInt is the primary driver of the humanoid robot fleet expansion. The project develops robots that understand context, learn from interaction, and provide proactive physical and verbal assistance — targeting independent living for older adults and people with special needs. The Humanoid Robot Lab is where hardware development, manipulation learning, and physical collaboration research take place across WP2 and WP3.

Legged and wheeled humanoid platforms from leading vendors run in parallel, enabling simultaneous development across multimodal scene representation (Kragic, WP2) and safe compliant physical interaction (Jaquier, WP3).

Led by Noémie Jaquier (WASP Assistant Professor, RPL), the GeoRob Lab develops data-efficient robot learning, optimization, and control algorithms with sound theoretical guarantees. Research treats differential geometry and physics as core inductive biases — enabling robots to generalise from fewer demonstrations and operate reliably under real-world constraints. The lab was recently awarded a Swedish Research Council Starting Grant (2025).

Funded by Knut och Alice Wallenbergs Stiftelse, this decade-long programme develops robotic systems that interpret their surroundings like human senses and learn through interaction with people and the environment. The goal is robots that can perform a wide range of tasks in homes and complex real-world settings — handling unpredictable, non-repetitive situations that are impossible to pre-program. Key advances include multimodal AI models processing sound, images, and force data simultaneously; learning from human demonstration and VR teleoperation; and manipulation of soft and deformable objects such as textiles, clothing, and groceries.

"Now we can train robots instead of programming robots, which makes it much easier to find solutions to environments where things change."

A ten-year Swedish Research Council Distinguished Professor grant developing new self-supervised and meta-learning methodologies with causal reasoning for perception, control, and reasoning in robotics. The project targets robots capable of complex interactions with both rigid and deformable objects and humans in unstructured, real-world environments — moving beyond carefully structured lab settings to handle the open challenges of limited data, unknown unknowns, and transfer across tasks.

A five-year ERC Advanced Grant (€2.4M) addressing one of the core open challenges in robotics: enabling machines to interact with deformable objects as naturally as humans do. BIRD created new informative and compact representations of deformable objects that combine analytical and learning-based approaches, encoding geometric, topological, and physical properties of the robot, object, and environment. Research focused on multimodal, bimanual interaction tasks, combining theoretical methods with rigorous experimental evaluation to model skilled sensorimotor behaviour in dual-arm robot systems.

The IA-Lab kitchen is the primary human-subject testing environment for SAInt. All five interaction scenarios — from passive observation to full robot-human collaboration — are designed around realistic cooking and domestic tasks performed here. The lab's sensor array (overhead cameras, Meta Aria glasses, embedded displays, smart appliances) provides the rich multimodal data streams needed for each work package.

Five scenarios of increasing complexity run in this kitchen: Observer → Apprentice → Instructor → Teacher → Collaborator, building from passive task observation to full shared autonomy between human and robot.

PerCorSo designs the most appropriate ways for robots to behave in human-crowded environments. Its novelty lies in integrating spatial and social context understanding, multimodal communication, and autonomous motion strategies to advance real-world social robots. The project addresses trust in autonomous robots in two complementary senses: verifiability (provably safe systems) and social acceptability (perceived as safe and trustworthy by humans). The IA-Lab kitchen is central to the project's real-world validation — building from controlled lab scenarios toward deployment in environments such as elderly care, autonomous driving, and human-robot coworking.

A multimodal dataset for referential expression grounding collected in the IA-Lab kitchen. Participants wore Meta Aria smart glasses for first-person gaze-tracked video while a GoPro captured the exocentric view — synchronising eye tracking, speech, and spatial grounding during live cooking tasks.

Each recording pairs egocentric Aria video (30 fps) with exocentric GoPro footage, real-time gaze coordinates, 48kHz audio, and word-level transcription via WhisperX. Designed for referential expression grounding, gaze–speech synchronisation, and embodied dialogue research. Created by Anna Deichler (KTH); presented at NeurIPS 2025 SpaVLE Workshop.

A Vinnova-funded project (Swedish: Prata mat) developing a proactive conversational AI cooking assistant for elderly users. Wizard-of-Oz experiments in the IA-Lab smart kitchen engaged 6 senior participants (ages 63–66) in comparing two AI chef personas: an Instructional variant and a Chatty variant. The chatty AI Chef was perceived as more situationally aware and intelligent. The project addressed food waste reduction and independent living for older adults.

A Digital Futures postdoctoral fellowship (June 2022–June 2024) developing robots capable of lifelong personalised dialogue — learning and recalling a person’s attributes, preferences, and shared history over long time horizons. Research addressed how foundation models and LLMs can enable open-domain conversation that adapts to individual elderly users, supporting daily reminders, collaborative tasks, and social engagement.

Deep learning models for predicting turn-taking in spoken interaction — identifying when speakers will yield or hold the floor. The project produced two key models: TurnGPT (language-model-based, trained on transcripts with speaker-shift markers) and Voice Activity Projection (VAP) (audio-based, trained on ~2,000 hours of telephone conversations, preserving acoustic features like intonation and pausing). HRI experiments used a Furhat social robot — a KTH spin-off company co-founded by PI Gabriel Skantze and in business since 2014 — enabling the models to drive fewer interruptions and faster response times in face-to-face dialogue. Research showed turn-taking cues are largely language-specific across three language families.

A Vinnova-funded project coordinated by Scania developing scene perception capabilities for safe autonomous driving. The KTH RPL team contributes two work packages: WP4 builds comprehensive local scene models by fusing data from multiple sensors, algorithms, and HD maps; WP5 uses these models for proactive situation interpretation and prediction — detecting objects that individual sensors would miss in occluded or complex traffic scenarios (e.g. vehicles emerging from behind stopped buses, or at intersections with limited sightlines).

A Digital Futures postdoctoral project (Jan 2025–Dec 2026) building a collaborative spatial perception framework for city-scale digital twinning. Multiple autonomous agents jointly construct multi-level abstract representations of urban environments from LiDAR point clouds, RGB-D imagery, and remote sensing data — enabling robots to autonomously create and continuously update a complete digital mirror of the physical world with minimal human involvement.