Visual information processing needs to fulfill two important goals. On the one hand, there is a need to comprehend the entire visual input at multiple distinctive levels to have an informed and stable representation of our environment. On the other hand, visual processing needs be selective and adaptive, as only a fraction of the visual input may be useful to guide our thoughts and actions at a given moment (Xu, 2018b). My search has focused on these two key aspects of visual information processing using cutting edge functional magnetic imaging (fMRI) methods on humans. I have made several significant empirical and theoretical contributions regarding the involvement of the human posterior parietal cortex (PPC) in supporting the adaptive and online aspect of visual information processing critical to visual working memory (VWM), decision making and behavior. My work has also significantly advanced our understanding of visual processing at multiple levels of processing in the human occipito-temporal cortex (OTC) such as the representation of parts, objects and ensembles of objects. Together, my empirical results and theoretical analyses have revised the “what vs where/how” two-pathway distinction previously proposed to characterize visual processing in the OTC and PPC. I argue instead that such a two-pathway distinction better reflects the invariant vs adaptive aspects of visual processing in the primate brain. The presence of these two complementary visual processing systems captures the demands of visual processing, providing us with both a stable representation of the visual environment and allowing us to interact flexibly and efficiently with the world.


The PPC in adaptive visual processing

To study the neural mechanisms mediating the adaptive aspect of visual information processing, I focus on the neural mechanisms supporting VWM, as VWM provides the critical buffer needed to retain information for the task at hand. In earlier behavioral studies, I documented the impact of object-based encoding and feature binding on VWM capacity (Xu, 2002a, 2002b & 2006). Applying similar manipulations to examine the human brain and measuring fMRI response amplitudes, I discovered dissociating roles of the human inferior intra-parietal sulcus (IPS) and superior IPS in determining VWM capacity (Xu & Chun, 2006 & 2007; Xu, 2007, 2008, 2009, 2010; Jeong & Xu, 2013). Specifically, while the inferior IPS is involved in individuating and selecting a fixed number of four items among multiple competing items, the superior IPS participates in the encoding and storage of the selected items into VWM. This discovery has helped resolve a critical debate in VWM regarding whether storage capacity is fixed or variable. My results show that both can be true, depending on the processing stage. These results have led me to propose the neural object file theory (Xu & Chun, 2009), which argues that the same two-stage processing may occur in the brain whenever multiple items are competing for representation, with the inferior IPS involved in selecting objects based on spatial information and the superior IPS involved in encoding and retaining detailed object information. This neural object-file theory can explain various forms of capacity-limited processing in visual cognition beyond VWM and can account for a large number of existing human behavioral and patient findings.


Having linked the PPC to adaptive visual processing through its role in VWM, in more recent work using fMRI multi-voxel pattern analysis (MVPA), I have documented the direct representation of VWM content in the human superior IPS (Bettencourt & Xu, 2016). The PPC is best known for its role in space, attention and action-related processing. The direct representation of visual information in the PPC poses a significant challenge to these existing views on the PPC. Further work shows that VWM representations in the superior IPS, but not those in early visual areas, track behavioral performance and are unaffected by the presence and predictability of distractors (Bettencourt & Xu, 2016). These results, together with a thorough review of the literature, have prompted me to critically reexamine the sensory account of VWM storage, a view currently dominating the VWM literature. I show that the available evidence in human behavioral, fMRI, and transcranial magnetic stimulation studies as well as monkey neurophysiological studies does not support the sensory regions as playing central roles in VWM storage. Instead, existing evidence supports the PPC and the prefrontal cortex as the primary regions involved in VWM storage (Xu, 2017 & 2018a).


The PPC’s ability to directly represent visual information in VWM tasks during adaptive visual processing prompts me to thoroughly document the range of visual information that can be directed represented in the PPC. Across a variety of different experimental paradigms, I observe the representations of a diverse array of visual information in the human PPC using fMRI MVPA measures. They include low-, mid-, and high-level visual information such as orientation, shape, viewpoint invariant object identity, and object category (Xu & Jeong, 2015; Bettencourt & Xu, 2016; Jeong & Xu, 2016 & 2017; Vaziri-Pashkam & Xu, 2017 & in press; Vaziri-Pashkam, Taylor & Xu, in press). Moreover, I show that the representational similarity of the visual information held in the PPC is closely correlated with perception and behavioral performance (Bettencourt & Xu, 2016; Jeong & Xu, 2016). Visual representations in the PPC also exhibit tolerance to low-level image transformations such as position, size and spatial frequency, much like those in higher-level regions in the OTC (Vaziri-Pashkam & Xu, in press; Vaziri-Pashkam, Taylor & Xu, in press). Incorporating these findings as well as a survey of the broader human imaging and monkey neurophysiology literature, in a recent review I provide a detailed and comprehensive documentation of the range of visual information that can be directly represented in the PPC and their tolerance to image transformations (Xu, 2018b).


Despite the existence of visual representations in both the OTC and the PPC, as described earlier, my research on VWM shows that visual representations in the PPC exhibits an overall greater resistance to distraction than those in the OTC (Bettencourt & Xu, 2016). By directly manipulating attention and task, I further show that visual representations in the PPC are under greater attention and task control than those in the OTC (Vaziri-Pashkam & Xu, 2017; Jeong & Xu, 2017; see also Xu, 2010; Jeong & Xu, 2013). Together, these results highlight the adaptive nature of visual representation in the PPC and support a two-pathway separation between the OTC and the PPC based on the invariant vs adaptive aspect of visual information processing, rather than the “what” vs “where/how” distinction previously proposed (Xu, 2018b).


The direct representation of a diverse array of visual information in the PPC is incompatible with the three major functions previously ascribed to the PPC focusing on spatial, attentional, or action-related processing. Rather than further contributing to an already fragmented literature regarding the precise function of the PPC, in another recent review, by examining functional and anatomical evidence, I show that visual representation is both distinct and at the same time closely interacting with these three major PPC functions. I argue that we can bring convergence among these distinct PPC functions through PPC’s role in adaptive visual processing and form a structured, integrated and unified understanding of PPC’s role in vision, cognition and action (Xu, 2018c). Such a framework is not only capable of accommodating a large body of existing PPC findings and bridging disparate lines of PPC research, but it can also provide critical guidance to future studies on the PPC and better connect them to the existing PPC literature.


The processing of visual information in the OTC

To fully understand the adaptive nature of visual processing in the PPC, one also needs to understand the processing of the different types of visual information in the OTC, as the PPC receives a large amount of input from the OTC during adaptive visual processing (see Xu, 2018b for a review of the supporting evidence). To that end, I have made several discoveries regarding the neural underpinnings involved at distinctive levels of visual information processing in the OTC.


Single objects. At the single object level, I have studied the neural mechanisms specialized for face processing in the OTC. By examining the part-whole relationship critical for object recognition in the OTC using fMRI MVPA, I have unveiled a previously unknown neural impairment on face configural processing in individuals with developmental prosopagnosia (DPs) (Zhang et al., 2015). This impairment is directly linked to DPs’ behavioral deficit in face processing and helps solve a long-standing mystery in face research. Using fMRI response amplitude measures and MVPA as well as magnetoencephalography (MEG) measures, I have also documented the interaction between visual expertise and the neural mechanisms mediating face processing in the OTC (Xu, 2015; Xu, et al., 2015; Ross et al., 2018).


Multiple objects. In real world perception and action, the selection and representation of multiple visual objects are often critical. At the multiple object level, besides my VWM work unveiling the importance of PPC mechanisms in selecting and representing multiple visual objects in visual cognition (Xu & Chun, 2006, 2007, 2009; Xu, 2007, 2008, 2009), using fMRI MVPA, I have also examined how multiple objects may be represented in the OTC. Specifically, I describe how contextual association between a pair of objects and the layout of several objects may be directly represented in the OTC (Wang & Xu, in preparation-a & in preparation-b).


Object ensembles. In everyday vision, we can simultaneously represent a large number of objects (object ensembles), such as the people in a crowd and the trees in a forest, by extracting summary statistics from such object ensembles without forming detailed representations of the individual objects comprising the ensemble. Object ensemble representation thus complements the capacity-limited but detailed representation of single and multiple objects. Despite its importance in visual perception, the neural mechanisms mediating object ensemble representation is vastly understudied. Using fMRI adaptation, I have documented the unique role of the anterior-medial ventral visual cortex in object ensemble representation and the ensemble features that this brain region may encode (Cant & Xu, 2012, 2015 & 2017; Cant et al., 2015). Given this brain region’s additional involvement in texture and scene representation, I argue that the anterior-medial ventral visual cortex may play a general role in summary statistics representation that can account for its role in texture, ensemble and scene representation (Cant & Xu, 2012, 2015 & 2017).


My current and future research aims to further our understanding of both the invariant and the adaptive nature of visual information processing in the human brain. Overall, my empirical findings and theoretical contributions have brought fundamental insights regarding the mechanisms and algorithms our brain uses to select and represent visual information to guide thoughts, decisions and behavior. This work directly bridges and complements existing monkey neurophysiology work on the PPC. My continuous effort to understand the neural coding schemes used by the OTC and the PPC for visual information processing will not only enrich our understanding of the human brain, but can also facilitate the development of better computational models that are capable of implementing vital human visual abilities in artificial intelligence.


References Cited

(To read the papers listed here, please go to my publication page.)


Bettencourt, K.C. & Xu, Y. (2016) Decoding the content of visual short-term memory under distraction in occipital and parietal areas. Nature Neuroscience, 19, 150-157.


Cant, J.S. & Xu, Y. (2012) Object ensemble processing in human anterior-medial ventral visual cortex. Journal of Neuroscience, 32, 7685-7700.


Cant, J.S. & Xu, Y. (2015) The impact of density and ratio on object-ensemble representation in human anterior-medial ventral visual cortex. Cerebral Cortex, 25, 4226-4239.


Cant, J. S. & Xu, Y. (2017). The contribution of object shape and surface properties to object-ensemble representation in anterior-medial ventral visual cortex. Journal of Cognitive Neuroscience, 29, 398-412.


Cant, J.S., Sun, S. & Xu, Y. (2015) Distinct cognitive mechanisms involved in the processing of single objects and object ensembles. Journal of Vision, 15:12, 1–21.


Jeong, S. & Xu, Y. (2013) Neural representation of targets and distractors during visual object individuation and identification. Journal of Cognitive Neuroscience, 25, 117-126.


Jeong, S.K. & Xu, Y. (2016) Behaviorally relevant abstract object identity representation in the human parietal cortex. Journal of Neuroscience, 36, 1607-1619.


Jeong, S.K. & Xu, Y. (2017). Task-context dependent linear representation of multiple visual objects in human parietal cortex. Journal of Cognitive Neuroscience 29, 1778–1789.


Ross, D.A., Tamber-Rosenau, B.J., Palmeri, T.J., Zhang, J., Xu, Y., Gauthier, I. (2018). High resolution fMRI reveals configural processing of cars in right anterior fusiform face area of car experts. Journal of Cognitive Neuroscience 30, 973–984.


Vaziri Pashkam, M. & Xu, Y. (2017). Goal-directed visual processing differentially impacts human ventral and dorsal visual representations. Journal of Neuroscience 37, 8767– 8782.


Vaziri-Pashkam, M., Taylor, J. & Xu, Y. (in press). Spatial frequency tolerant visual object representations in the human ventral and dorsal visual processing pathways. Journal of Cognitive Neuroscience.


Vaziri-Pashkam, M. & Xu, Y. (in press). An information-driven two-pathway characterization of occipito-temporal and posterior parietal visual object representations. Cerebral Cortex.


Wang, R. & Xu, Y. (in preparation-a) Neural representation of contextual information for object pairs in human ventral visual cortex.


Wang, R. & Xu, Y. (in preparation-b). The neural representation of object layout.


Xu, Y. (2002a). Encoding color and shape from different parts of an object in visual short-term memory. Perception & Psychophysics, 64, 1260–1280.


Xu, Y. (2002b). Limitations of object-based feature encoding in visual short-term memory. Journal of Experimental Psychology: Human Perception and Performance, 28, 458–468.


Xu, Y. (2005). Revisiting the role of the fusiform face area in visual expertise. Cerebral Cortex, 15, 1234–1242.


Xu, Y. (2006). Understanding the object benefit in visual short-term memory: The roles of feature proximity and connectedness. Perception & Psychophysics, 68, 815–828.


Xu, Y. (2007) The role of the superior intraparietal sulcus in supporting visual short-term memory for multifeature objects. Journal of Neuroscience, 27, 11676-11686.


Xu, Y. (2008) Representing connected and disconnected shapes in human inferior intra-parietal sulcus. Neuroimage, 40, 1849-1856.


Xu, Y. (2009) Distinctive neural mechanisms supporting visual object individuation and identification. Journal of Cognitive Neuroscience, 21, 511-519.


Xu, Y. (2010) The neural fate of task-irrelevant features in object-based processing. Journal of Neuroscience, 30, 14020-14028.


Xu, Y. (2017). Reevaluating the sensory account of visual working memory storage. Trends in Cognitive Science. In press.


Xu, Y. (2018a). Sensory cortex is nonessential in working memory storage. (A reply to two commentaries). Trends in Cognitive Science 22, 192-193.


Xu, Y. (2018b). A tale of two visual systems: Invariant and adaptive visual information representations in the primate brain. Annual Review in Vision Science, 4, 311–336.


Xu, Y. (2018c). The posterior parietal cortex in adaptive visual processing. Trends in Neuroscience 41, 806-822.


Xu, Y. & Chun, M. M. (2006) Dissociable neural mechanisms supporting visual short-term memory for objects. Nature, 440, 91-95.


Xu, Y. & Chun, M. M. (2007) Visual grouping in human parietal cortex. Proceedings of the National Academy of Sciences, USA, 104, 18766-18771.


Xu, Y. & Chun, M. M. (2009) Selecting and perceiving multiple visual objects. Trends in Cognitive Sciences, 13, 167-174.


Xu, Y. & Jeong, S. (2015) The contribution of human superior intra-parietal sulcus to visual short-term memory and perception. In: Mechanisms of Sensory Working Memory: Attention and Performance XXV, Jolicoeur, P. and Martinez-Trujillo, J. (eds.), 33-42.


Xu, Y., Liu, J., & Kanwisher, N. (2005). The M170 is selective for faces, not for expertise. Neuropsychologia, 43, 588–597.


Zhang, J., Liu, Y., & Xu, Y. (2015) Neural decoding reveals impaired face configural processing in the right fusiform face area of individuals with developmental prosopagnosia. Journal of Neuroscience, 35, 1539-1548.






Visual information representation in the mind and brain