Study: Video → Audio Translation

As part of the redesign of the San Francisco Chronicle I began experimenting with a feature that we called “The Rundown,” which essentially repurposed the UI of Instagram stories as a news briefing experience.

The process required thinking through a content distillation mechanism that would allow a story to be presented as a single screen’s worth of information. I realized that this distillation process was similar to what it would take to create an audio-based briefing experience.

Attempting to solve for both media at once I assembled a short presentation that examined what it might take to translate a visual interface to an audio experience.

Progress / Duration indicator

Requires translation: Numbers written as fractions have a different meaning when read audibly.

Navigation

Requires translation: In the mobile interface navigating to a subsequent story is accomplished gesturally. This ability is suggested by the progress indicators at the top of the screen. An audio interface requires a direct verbal command, and that ability should be announced to the user at the beginning of the experience.

Exit Opportunity

Requires translation: the ubiquitous “x” sign needs a natural language equivalent in an audio interface: “stop” should do the trick.

Media

May require translation: If the media element is a video containing sound, a portion of that sound may be parsed out as an additional piece of story information. If a static image, a description would likely be redundant when placed alongside other elements of the experience.

Section Header

Requires translation: If ingested straight from an article headline this reads awkwardly. Punctuation should be substituted with conversational language in order to read like natural language when presented as audio.

Information Body

Some translation required: Like the headline, the body text may not feel natural when read aloud. Certain syntaxes should be reconstructed to sound more conversational for a better audio experience.

Action Item

Translation required: A user understands a button as option: an action they can take or not take. In an audio experience, the optional nature of the decision needs to be clarified. Calls to action should be phrased as questions, not as commands.

Conclusion

It’s clear that some UI elements translate better into audio experiences than others. Some of these can be solved with simple substitutions. Others, like headlines and story summaries, require more sophisticated natural language processing. As a shortcut, the CMS should adopt fields for audio-friendly headlines and abstracts.