Central to human cognition are the abilities to collect and accumulate information, to use it to guide thoughts and behavior, and to make good decisions. Many cognitive abilities such as learning, communication, reasoning, and development depend on these abilities.  Deficiency in them can lead to severe cognitive impairments. The approach that I have undertaken to understand these human mental abilities is to examine the neural mechanisms behind visual information collection: most of the information that we glean from the world comes through vision. Visual information collection, however, is not a simple process. As shown in the figure, given the same visual input, we can attend to the material properties of the tabletop to decide whether to place a hot pan on it, attend to the whole table to avoid walking into it, attend to both the table and the stool to find a place to sit down and eat, or attend to the entire ensemble of objects to recognize that we are in a kitchen. This example highlights two important facets of visual information processing. One is that vision is a multi-level process, in which we can extract texture/material properties, identities of single or multiple objects, or the gist of a scene. The other is that vision is highly dynamic, allowing us to effortlessly navigate through these different levels of visual hierarchy to extract the appropriate information to guide us through the task at hand.


Using cutting edge functional magnetic imaging (fMRI) methods such as adaptation, multi-voxel pattern analysis (MVPA) and pattern averaging, representation similarity analysis, correlation between behavioral and neural representations, and topographic mapping, the main focus of my search has been on understanding the dynamic and online aspect of visual information processing critical to decision making and behavior. I have made several discoveries regarding the involvement of the human posterior parietal cortex (PPC) in goal-directed visual processing: (1) I have documented the role of the human superior intra-parietal sulcus (IPS) in tracking visual working memory (VWM) capacity and the role of human inferior IPS in individuating and selecting a fixed number of four items among multiple competing items (Xu & Chun, 2006 & 2007; Xu, 2007, 2008, 2009, 2010; Jeong & Xu, 2013). This has helped resolve a critical debate in VWM regarding whether storage capacity is fixed or variable. My results show that both can be true, depending on the processing stage. These results have led me to propose the neural object file theory, a framework that could account for capacity-limited processing in vision and cognition beyond VWM (Xu & Chun, 2009). (2) I have documented the direct representation of VWM content in superior IPS. Critically, VWM representations in superior IPS, but not those in early visual areas, track behavioral performance and are unaffected by the presence and predictability of distractors (Bettencourt & Xu, 2016). These results, together with a thorough review of the literature, have prompted me to critically reexamine the sensory account of VWM storage, a view currently dominating the VWM literature. I argue instead that the posterior parietal cortex (PPC), rather than sensory regions, plays a more primary role in VWM storage (Xu, 2017). (3) Across a variety of different experimental paradigms, I have documented the representations of a diverse array of visual information in the human PPC. They include low-, mid-, and high-level visual information such as orientation, shape, viewpoint invariant object identity, and object category (Xu & Jeong, 2015; Bettencourt & Xu, 2016; Jeong & Xu, 2016 & 2017; Vaziri-Pashkam & Xu, 2017). These findings significantly challenge the “ventral-what and dorsal-where/how" two-pathway view of visual information processing and a purely attentional account of PPC function. Yet, by understanding visual representation in PPC and its interaction with spatial, attention and action-related processing, I argue that we can bring convergence among disparate lines of research and form a unified understanding of PPC’s role in vision, cognition and action (Xu, in press). (4) I have identified both the similarities and the differences in visual presentation between higher ventral and dorsal regions. Although both regions are capable of representing a diverse array of visual information and show similar tolerance to identity-irrelevant image transformations, dorsal regions show an overall greater resistance to distraction and are under greater attention and task control than ventral regions (Bettencourt & Xu, 2016; Jeong & Xu, 2017; Vaziri-Pashkam & Xu, 2017 & in press; see also Xu & Chun, 2016; Xu, 2010; Jeong & Xu, 2013). Thus a separation between ventral and dorsal pathways of visual information processing still exists, but instead of indicating the original “what vs where/how” distinction, it seems to more closely reflect the invariable vs dynamic aspect of visual information processing (Xu, in press). My current research focuses on understanding the precise neural coding schemes used by parietal regions in visual representation and how the binding of visual features is represented in occipito-temporal and parietal regions.


To fully understand the dynamic nature of vision, one also needs to understand the multi-level aspect of vision. To that end, I have made several discoveries regarding the neural underpinnings involved at distinctive levels of visual information processing: (1) At the single object level, I study the part-whole relationship critical for object recognition in ventral visual cortex. I have unveiled a previously unknown neural impairment on face configural processing in individuals with developmental prosopagnosia (DPs) (Zhang et al., 2015). This impairment is directly linked to DPs’ behavioral deficit in face processing and helps solve a long-standing mystery in face research. I have also examined how contextual association between a pair of objects may be represented (Wang & Xu, in preparation). (2) At the multiple object level, my work in VWM has led me to uncover the importance of PPC mechanisms in selecting and representing multiple visual objects in visual cognition (Xu & Chun, 2006, 2007, 2009; Xu, 2007, 2008, 2009). (3) At the object ensemble level, I documented the role of the anterior-medial ventral visual cortex in object ensemble representation, an important but vastly understudied aspect of visual processing, and highlighted the greater role this brain region may play in texture, ensemble and scene representation (Cant & Xu, 2012, 2015, 2017; Cant et al., 2015).


Overall, my empirical findings and theoretical contributions have brought fundamental insights regarding the mechanisms and algorithms our brain uses to select and represent visual information to guide thoughts, decisions and behavior. My work on human PPC directly bridges and complements existing monkey neurophysiology work on PPC. By examining the dynamic aspect of vision and the information needed for decision making, the neural mechanisms revealed in my work can lead to a better understanding of the neural underpinnings underlying more complex forms of decision making, such as in social and economic decision makings. Understanding the neural coding schemes used in ventral visual cortex and PPC can also provide significant insights and facilitate the development of better computational models that are capable of implementing vital human visual abilities in machine vision and artificial intelligence (AI).


Visual information representation in the mind and brain