Recently, CAHMP faculty, Dr. Craig Yu, is involved in a group researchers that present a novel interactive augmented reality (AR) storytelling approach guided by indoor scene semantics.
Their approach automatically populates virtual contents in real-world environments to deliver AR stories, which match both the story plots and scene semantics. During the storytelling process, a player can participate as a character in the story. Meanwhile, the behaviors of the virtual characters and the placement of the virtual items adapt to the player’s actions. An input raw story is represented as a sequence of events, which contain high-level descriptions of the characters’ states, and is converted into a graph representation with automatically supplemented low-level spatial details.
Their hierarchical story sampling approach samples realistic character behaviors that fit the story contexts through optimizations; and an animator, which estimates and prioritizes the player’s actions, animates the virtual characters to tell the story in AR. Through experiments and a user study, they validated the effectiveness of the approach for AR storytelling in different environments.
For more information about this project, please click here.
Also see Craig’s work in assistive robotics here.