The result of the collaboration is a tool which combines the most helpful elements of virtual assistants and screen readers (tools which convey elements displayed on a screen into non-visual cues, such as speech or Braille). The tool is called Voice Exploration, Retrieval, and Search (VERSE).
"People with visual impairments often rely on screen readers, and increasingly voice-based virtual assistants, when interacting with computer systems," said University of Waterloo mathematics student Alexandra Vtyurina, who conducted the study during an internship with Microsoft Research. "Virtual assistants are convenient and accessible, but lack the ability to deeply engage with content, such as read beyond the first few sentences of an article, list alternative search results and suggestions."
"In contrast, screen readers allow for deep engagement with accessible content and provide fine-grained navigation and control, but at the cost of reduced walk-up-and-use convenience."
According to Vtyurina, VERSE adds screen reader-like capabilities to virtual assistants, while allowing other connected devices (such as smart watches) to serve as inputs to the connected speakers which typically 'embody' virtual assistants. Users command VERSE using simple vocal commands, such as 'next', 'go back', and 'go forward'.
VERSE can also be paired with an app running on a smartphone or smartwatch. This allows for handy devices to be used as 'input accelerators' similar to keyboard shortcuts; for instance, rotating the crown of a smartwatch causes VERSE to jump to the next search result, section or paragraph.
The researchers surveyed 53 people with visual impairments, more than half of which use voice assistants frequently in addition to other connected devices. Information taken from this survey was used to inform the design of a VERSE prototype.
"At the outset, VERSE resembles other virtual assistants, as the tool allows people to ask a question and have it answered verbally with a word, phrase or passage," said Vtyurina. "VERSE is differentiated by what happens next."
"If people need more information, they can use VERSE to access other search verticals, for example, news, facts and related searches, and can visit any article that appears as a search result."
In recent weeks, the world's largest tech companies - including Amazon, Apple, and Google - have been in hot water over reports revealing that they use human workers to listen to recordings of people speaking to their virtual assistants in order to improve their natural language processing capabilities.