Stanford CS547 HCI Seminar Spring 2026: Text, Visualization, and Their Combination

Marti Hearst argues that text is not a peripheral add‑on but a core component of information visualizations, and that the right amount and type of text can improve takeaways, though predictions and bias perception follow different patterns.

Defining Information Visualization and the Role of Language

Information visualization provides insight through spatial relations that are invisible in mere statistics, and language—especially titles and labels—plays a crucial role in what viewers notice and remember. Visualizations are external visual representations that are systematically spatially related to the information they represent, as illustrated by Anscombe’s quartet, where identical sufficient statistics yield radically different plots. Historically, the visualization community gave little attention to text; for example, Apple’s chart design guidelines omitted titles and tick annotations. However, eye‑tracking work showed that titles received the longest fixations during encoding and were most likely to be mentioned during recall, followed by labels and paragraphs, indicating that language is a key component of visualization.

Deciding What to Express in Text versus Visualization

Designers must first choose where to place text, what semantic level it should carry, and how much detail to include, because these choices shape viewer takeaways. A study that computed visually salient regions (red, green, blue) found that when captions aligned with the most salient features, participants’ takeaways matched the visual prominence; when captions highlighted less salient features, the caption’s content dominated the takeaways; and when there was no alignment, the visual features prevailed. This interplay demonstrates that text can override of the two modalities, whichever is better aligned with the viewer’s focus tends to win.

Types of Text and Their Impact on Different Audiences

High‑level textual descriptions that bring in external context are preferred by sighted readers, while blind and low‑vision users favor low‑level descriptions that mention concrete values, suggesting that text should be treated as coequal with visual elements. In an accessibility study, participants labeled captions into four categories: high‑level (external information, e.g., "the big drop in prices was caused by the financial crisis of 2008)), trend‑level (e.g., "prices fluctuate, but nevertheless increase over time)), lower‑level (mentioning a single point’s value), and another unspecified level. Blind and low‑vision participants opposed high‑level descriptions but favored low‑level ones; sighted participants showed the opposite pattern. The authors concluded that natural language should be seen as coequal with visualizations in design.

How Much Text Should Appear on a Chart

In experiments ranging from no text to title‑annotation‑subtitle‑heavy layouts, participants overwhelmingly preferred more relevant text, although a significant minority rejected charts altogether and a small group wanted no text at all. When varying text from none to title only, then title plus annotation, subtitle, and lots of text, the results showed a preference for more text; the original paper title had been "under and over texting" but was changed to "striking a balance" because over‑texting never emerged. A significant minority of participants did not want the chart, a small minority wanted no title or any text, and a fraction liked a little but not too much. These findings challenge Tufte’s minimalism and suggest that relevant annotation improves comprehension.

Text Annotations, Predictions, and Perceived Bias

Adding textual annotations to ambiguous charts did not shift viewers’ predictions, but the same annotations strongly signaled the author’s presumed bias toward the highlighted entity. In a prediction task with a chart showing two technology companies’ market shares and an upcoming event, textual annotations such as "green market share increased steadily" did not change the proportion of participants who thought blue or green would be higher. However, when the same label was present, participants were more likely to attribute the chart to a green‑favoring author, indicating that text serves as a strong signal of perceived bias even when it does not alter visual judgments.

Comparing Language‑Based and Visual‑Based Comparisons

Language can naturally compare only a few items (typically two to five), whereas visual encodings enable simultaneous comparison of many data points, and combining the two works best when visual context is provided. Linguistic comparisons are limited to small sets unless words like "respectively" are used, while visualizations can juxtapose, superimpose, or explicitly encode many points. In a collaboration with Tableau Research, a query for "show me the tallest buildings" used cognitive‑linguistics‑derived thresholds to decide which bars to highlight. In a chat‑interface study comparing event counts for weightlifting and taekwondo, 41% of participants did not want to see any visualization; among those who did, more bars were preferred because they provided context, whereas text‑only responses were praised for precision and simplicity. Some participants switched preferences when they realized the visual context gave them information they lacked in the text‑only version.

When Text Alone Is Preferred

Across multiple studies, sizable minorities of participants chose text‑only presentations—41% in a conversational‑bot scenario and 14% in a rich‑data visualization—indicating that text alone remains a viable option for information delivery. In a Bayesian reasoning task with doctors, Ottley et al. found no significant effect on accuracy between text and visualization conditions, but when both were presented together participants tended to focus on one modality and ignore the other. In the New York Times “You Draw It” study, text presentations helped recall specific numeric values, while visualizations helped recall trends; seeing one’s prediction against the observed data was helpful only with visualizations. These results suggest that offering a text‑only alternative can accommodate users who prefer or benefit from verbal information.

Cognitive Models of Text‑Visual Interaction

Existing cognitive frameworks—dual coding, dual channel, cognitive load, and dual‑process (System 1/System 2) theories—offer partial explanations for why text and visuals sometimes cooperate and sometimes compete, but no single model fully accounts for the observed complexities. Dual coding theory posits independent visual and verbal subsystems that combine to enhance memory; the dual channel model adds limited capacity per channel and the need to build connections between them. Cognitive load theory argues that simultaneous visual and verbal inputs can overload working memory and reduce comprehension. A dual‑process perspective (System 1 automatic, System 2 effortful) was applied to visualization: in a task asking whether the average of A and B is closer to 2 or 2.2, System 1 would lead to the incorrect answer 2, while System 2 requires effortful calculation to reach 2.2. This may explain why text annotations failed to shift predictions in the ambiguous market‑share chart—System 1 visual processing dominated the gut judgment. Additionally, inserting non‑alphanumeric visuals such as icons or hyperlinks into paragraphs can impede fluent reading because they attract the eye and disrupt the parafoveal preview that supports word recognition, a point motivated by research on reading fluency (e.g., Proust and the Squid).

Implications for Multimodal Language Models

Current multimodal large language models process vision and language through either unified embedding concatenation or cross‑attention layers, mirroring some aspects of human dual‑channel processing, though it remains unclear how these mechanisms inform the design of human‑focused text‑visualization combinations. In the unified embedding architecture, visual and textual tokens are each embedded and then concatenated before feeding into a standard transformer decoder. In the cross‑modality embedding architecture, vision is processed first, then cross‑attention layers allow the language model to attend to visual tokens while preserving pretrained language weights. The Flamingo model uses masked cross‑attention so that each text token attends only to the corresponding image, preventing earlier images from influencing later text. Work on Lava models shows that bottom layers propagate general visual information into linguistic representations, middle layers transfer task‑relevant visual features, and top layers produce the final answer. While these models demonstrate strong vision‑language reasoning, the talk notes that it is uncertain how these internal mechanisms map onto human cognition for text‑visualization tasks.

Open Directions

Future work should explore spoken language, accessibility linkages, document‑visualization integration, misinformation, multilingual aspects, and visualizing text itself, building on the findings that text and visuals are deeply intertwined yet context‑dependent. The speaker mentioned additional avenues such as linking visualizations to their references in text, studying the effect of text on visualization trust, and examining how different modalities affect reasoning in various domains, but did not elaborate due to time constraints.