Pen-based Interaction with Spreadsheets in Mobile Virtual Reality

  • Travis Gesslein ,
  • Verena Biener ,
  • Philipp Gagel ,
  • Daniel Schneider ,
  • Per Ola Kristensson ,
  • Eyal Ofek ,
  • ,
  • Jens Grubert

IEEE ISMAR 2020 |

Organized by IEEE

Download BibTex

Virtual Reality (VR) can enhance the display and interaction of mobile knowledge work and in particular, spreadsheet applications. While spreadsheets are widely used yet are challenging to interact with, especially on mobile devices, using them in VR has not been explored in depth. A special uniqueness of the domain is the contrast between the immersive and large display space afforded by VR, contrasted by the very limited interaction space that may be afforded for the information worker on the go, such as an airplane seat or a small work-space. To close this gap, we present a tool-set for enhancing spreadsheet interaction on tablets using immersive VR headsets and pen-based input. This combination opens up many possibilities for enhancing the productivity for spreadsheet interaction. We propose to use the space around and in front of the tablet for enhanced visualization of spreadsheet data and meta-data. For example, extending sheet display beyond the bounds of the physical screen, or easier debugging by uncovering hidden dependencies between sheet’s cells. Combining the precise on-screen input of a pen with spatial sensing around the tablet, we propose tools for the efficient creation and editing of spreadsheets functions such as off-the-screen layered menus, visualization of sheets dependencies, and gaze-and-touch-based switching between spreadsheet tabs. We study the feasibility of the proposed tool-set using a video-based online survey and an expert-based assessment of indicative human performance potential.

Pen-based Interaction with Spreadsheets in Mobile Virtual Reality

Virtual Reality (VR) can enhance the display and interaction of mobile knowledge work and in particular, spreadsheet applications. While spreadsheets are widely used yet are challenging to interact with, especially on mobile devices, using them in VR has not been explored in depth. A special uniqueness of the domain is the contrast between the immersive and large display space afforded by VR, contrasted by the very limited interaction space that may be afforded for the information worker on the go, such as an airplane seat or a small work-space. To close this gap, we present a tool-set for enhancing spreadsheet interaction on tablets using immersive VR headsets and pen-based input. This combination opens up many possibilities for enhancing the productivity for spreadsheet interaction. We propose to use the space around and in front of the tablet for enhanced visualization of spreadsheet data and meta-data. For example, extending sheet display beyond the bounds of the physical screen, or easier debugging by uncovering hidden dependencies between sheet’s cells. Combining the precise on-screen input of a pen with spatial sensing around the tablet, we propose tools for the efficient creation and editing of spreadsheets functions such as off-the-screen layered menus, visualization of sheets dependencies, and gaze-and-touch-based switching between spreadsheet tabs. We study the feasibility of the proposed tool-set using a video-based online survey and an expert-based assessment of indicative human performance potential.

Enhancing mobile work and productivity with virtual reality

As people work from home, new opportunities and challenges arise around mobile office work. On one hand, people may have flexible work hours and may not need to deal with traffic or long commutes. On the other hand, they may need to work at makeshift spaces, with less-than-optimal working conditions while physically separated from co-workers. 

Virtual reality (VR) has the potential to change the way we work, whether from home or at the office, and help address some of these new challenges. We envision the future office worker to be able to work productively everywhere, solely using portable standard input devices and immersive head-mounted displays. VR has the potential to enable this by allowing users to create working environments of their choice and by relieving them of physical limitations, such as constrained space or noisy environments. 

In this webinar, Microsoft Researcher Eyal Ofek presents a summary of research investigating opportunities and challenges for realizing a mobile VR office environment. In particular, you’ll learn how VR can be mixed with standard off-the-shelf equipment (such as tablets, laptops, or desktops) to enable effective, efficient, and ergonomic mobile knowledge work. 

Together, we’ll explore: 

  • The unique needs of developing immersive applications for productivity. 
  • Ways in which non-physical virtual workspaces can make work more efficient. 
  • How to adapt virtual workspaces to the physical environment. 

Resource list:

*This on-demand webinar features a previously recorded Q&A session and open captioning. 

Explore more Microsoft Research webinars: https://aka.ms/msrwebinars (opens in new tab)