Computational mechanisms for neural representation of words

A critical way for humans to acquire, represent and communicate information is through language, yet the underlying computation mechanisms through which language contributes to our word meaning representations are poorly understood. We compared three major types of word computation mechanisms from large language corpus (simple co-occurrence, graph-space relations and neural-network-vector-embedding relations) in terms of the association of words’ brain activity patterns, measured by two functional magnetic resonance imaging (fMRI) experiments. Word relations derived from a graph-space representation, and not neural-network-vector-embedding, had unique explanatory power for the neural activity patterns in brain regions that have been shown to be particularly sensitive to language processes, including the anterior temporal lobe (capturing graph-common-neighbors), inferior frontal gyrus, and posterior middle/inferior temporal gyrus (capturing graph-shortest-path). These results were robust across different window sizes and graph sizes and were relatively specific to language inputs. These findings highlight the role of cumulative language inputs in organizing word meaning neural representations and provide a mathematical model to explain how different brain regions capture different types of language-derived information.


Introduction 1 2
A typical adult human brain stores the forms and meanings of approximately tens of thousands 3 of words (42,000 for typical English-speaking Americans) (Brysbaert,Stevens,Mandera,& Keuleers,4 2016), which allows naming objects and actions, understanding and producing sentences, and 5 contributing to various kinds of reasoning. Decades of neuroimaging and neuropsychological 6 literature have studied the cognitive neural representations of word meaning and revealed that they 7 are (at least partly) derived from sensory experiences distributed across multiple sensory association 8 cortices (Fernandino et al., 2016;A. Martin, 2016;Miceli et al., 2001), with those sharing physical 9 properties (sensory/motor experiences) represented more closely in corresponding brain regions 10 (Binder et al., 2016;Clarke & Tyler, 2014, 2015. The roles of the language system in representing  Does the brain capture word meaning representation from cumulative language experience, and 21 if yes, how? It has recently been shown that word relations constructed from various natural 22 language computation models derived from a large text corpus correlate with the fMRI whole-brain 23 response patterns to corresponding words (count models (Huth, De Heer, Griffiths, Theunissen, & 24 Gallant, 2016;Mitchell et al., 2008); GloVe model (Anderson et al., 2019;Pereira et al., 2018)) and 25 with the brain activity patterns of more specialized language processing regions/networks (word2vec 26 and LSA models (Carota, Kriegeskorte, Nili, & Pulvermüller, 2017;Carota, Nili, Pulvermüller, & 27 Kriegeskorte, 2021; Xiaosha Wang et al., 2018)). These observed correlations between language 28 computation models and brain activity patterns suggest that word representation in the brain may 29 follow certain types of computation from language inputs, but their interpretation requires caution. 30 There are significant correlations between word-relational structures constructed from different 31 language computation models, as well as with relational structures derived from the sensory systems 32 (e.g., "cat" and "dog" are closely related across all of these different types of measures) (Kumar, 2021;33 Lenci, 2018). Understanding the underlying computational architecture (and potentially algorithms) 34 of the language-derived knowledge system requires carefully examining the differences in these 35 models and testing which models better explain the brain activity patterns associated with word 36 meanings.

37
Ample evidence has shown that humans are sensitive to various types/scales of statistical 38 patterns from language-like inputs (Lynn & Bassett, 2020). Statistical learning studies show that 39 humans can detect both variations in the probabilities of transitions between items and in non-local 40 network structures (e.g. non-adjacent dependency, community structures) (Garvert,Dolan,& 41 Behrens, 2017; Saffran, Aslin, & Newport, 1996;Schapiro, Rogers, Cordova, Turk-Browne, & Botvinick, 42 2013). Accordingly, for the statistical patterns derived from the long-term representation of the 43 cumulative language inputs, we consider three major types of computation mechanisms: 1) Simple 44 co-occurrence, which involves the collective counts or probabilities of two words co-occurring in a 45 context window. Simple co-occurrence measures, reflecting the first-order proximity between two 46 words, have been used to model word relatedness (Church & Hanks, 1990;Recchia & Jones, 2009; 47 1 (Borge-Holthoefer & Arenas, 2010;Jackson & Bolger, 2014), with different types of 2 relationships/distances computed from the graph-related structure (e.g. common neighbors, 3 shortest path). It has been shown that different types of word relations (similarity and association) 4 can be measured using different graph measures implemented in a unified word co-occurrence graph 5 space, providing a relatively transparent, dissociable representation of word statistical relation 6 patterns (Jackson & Bolger, 2014). 3) Vector representation based on a set of fixed dimensions 7 obtained through matrix factorization methods (e.g. LSA) (Landauer & Dumais, 1997) or model-based 8 embedding methods (e.g. word2vec, GloVe) (Mikolov, Chen, Corrado, & Dean, 2013;Pennington, 9 Socher, & Manning, 2014). These vector models, especially embedding models (word2vec, GloVe), 10 have also been shown to achieve state-of-the-art performance in a variety of semantic-related 11 evaluation tasks (Baroni, Dinu, & Kruszewski, 2014;Pereira, Gershman, Ritter, & Botvinick, 2016). 12 While these vector models are computationally efficient and widely used in modeling behavioral and 13 neural semantic representations, the statistical information embedded in the obtained vector space 14 (e.g., the neural network embedding one) has remained elusive due to explicit/implicit dimension 15 reduction and hyperparameter tuning processes (Levy & Goldberg, 2014;Levy, Goldberg, & Dagan, 16 2015). 17 Here, we compare these three major types of computation mechanisms that capture different 18 aspects of statistical patterns of the language corpus in modeling neural activity patterns of word 19 processing: simple co-occurrence, graph-space relations and neural-network based vector-space 20 relations (i.e., cosine distance in a learnt word2vec space) (Fig. 1). We conducted a word production 21 fMRI experiment (oral picture naming) and a word recognition fMRI experiment (word familiarity 22 judgment) on the same set of 95 words to obtain their neural activity patterns across different 23 input/output modalities. Distances calculated from simple co-occurrence, graph-related measures 24 (graph-common-neighbors and graph-shortest-path) from a large-scale language corpus (Chinese computation models were fit with the neural RDMs derived from the fMRI data to locate neural 30 circuits that are organized by specific language statistical properties. Graph-space relations of visual 31 co-occurrence statistics derived from a large visual image database (VisualGenome; 32 https://visualgenome.org/) (Krishna et al., 2017) were also compared to examine the 33 universality/specificity of the observed word neural computations.

38
Three types of language computation mechanisms were implemented. Simple co-occurrence 39 counts were derived from a large-scale language corpus (Chinese Web n-gram Corpus, consisting of 40 approximately 883 billion Chinese words). These raw counts were PPMI-normalized to represent 41 first-order proximity between two words (Fig. 1a). Such normalized word co-occurrence of the 95 42 experimental stimuli (Fig. S1 & Table S1) was used to construct the 95 x 95 simple co-occurrence RDM.

43
Beyond the simple co-occurrence (i.e. edge RDM in the graph space), graph-common-neighbors 44 (graph-CN) and graph-shortest-path (graph-SP), two types of graph-related measures reflecting 45 different aspects of statistical properties were computed in a downsampled graph space (83,007 46 unique Chinese word samples as nodes, 34,586,840 PPMI-normalized simple co-occurrence as edges) 47 to yield a graph-CN RDM and a graph-SP RDM (Fig. 1b). A word2vec RDM was constructed based on 1 the cosine distance in a state-of-art pretrained word vector dataset (Li et al., 2018) (Fig. 1c).

8
RSA results: Relationship between language models and brain activity patterns 9 Neural RDMs of the 95 items were generated and fit with language model RDMs for each fMRI 10 experiment in an iterative sphere (10 mm) of each individual native space (an individually defined 11 gray matter mask), following the procedure of whole brain searchlight RSA (Kriegeskorte,Goebel,& 12 Bandettini, 2006) (Fig. 2a). In each experiment, analyses were carried out for stimuli peripheral 13 variables to perform sanity checks: pixel RDM and gist RDM in the oral picture naming experiment, 14 and pixel RDM and familiarity (button-press) RDM in the word familiarity judgment experiment. The

15
RSA results of these control models were highly consistent with the previous literature (Carota et al., 16 2021;Devereux, Clarke, Marouchos, & Tyler, 2013;Kriegeskorte, Mur, Ruff, et al., 2008) (Fig. S2). In 17 the main analyses of the language computation models, we first looked at the RSA results of each 18 model independently ("raw" effect), with the peripheral factors (pixel and gist RDM in oral picture 19 naming, pixel and word familiarity RDM in word familiarity judgment) regressed out. Given that these 20 RDMs of language computation models are correlated (Fig. 1d), we further carried out a "unique 21 effect" RSA for each language computation model, where the effects of the other language models 22 were further controlled for (all using partial Spearman rank correlations). The convention cluster 23 extent-based inference threshold (primary voxelwise p < 0.001, FWE-corrected cluster-level p < 0.05) 24 was adopted. The results for both fMRI experiments are shown, with positive results across both 25 experiments, i.e., the shared cognitive components (word meanings) across experimental 26 inputs/outputs, presented in detail (Table 1). As the conjunction methods are relatively conservative 27 (Nichols, Brett, Andersson, Wager, & Poline, 2005), we reported clusters that survived from the 28 threshold (uncorrected voxelwise p < 0.005, cluster size > 20 voxels) across two experiments.

29
Language model-brain RSA raw results. The maps of group-level whole brain searchlight RSA  Table S3).

33
First-order-edge (simple co-occurrence) distance. In the oral picture naming experiment, the 34 edge RDM correlated significantly with neural RDMs throughout the bilateral occipital-temporal 35 cortex, with peak effects in the lateral occipital cortex (LOC), extending into the early visual cortex, 36 pMTG, and the posterior division of the temporal fusiform gyrus (pFG) and bilateral ATL, including 37 the right temporal pole (TP), anterior division of the temporal fusiform gyrus (aFG) and 38 parahippocampal gyrus (aPHG). In the word familiarity judgment experiment, the neural effects of 39 edge RDM were confined to the bilateral ATL, including the TP, aPHG and left anterior division of the 40 middle temporal gyrus (aMTG), the dorsal part of the medial frontal cortex (medPFC), orbital frontal 41 cortex (OFC), right precuneus, precentral and postcentral gyrus, cingulate gyrus, insula and putamen.

42
The overlap analysis showed that the bilateral ATL, especially the ventral part, was sensitive to the 43 edge RDM in a task-invariant manner.

44
Graph-common-neighbors distance. In the oral picture naming experiment, neural effects of 45 graph-CN RDM were also found in occipital and temporal regions, including the bilateral LOC, pMTG, 46 pFG and ATL. Clusters in the bilateral ATL encompassed the TP, aPHG, aFG, aMTG and inferior 47 temporal gyrus (aITG). In the word familiarity judgment experiment, similar patterns were found in 1 regions of bilateral ATL. Clusters extended into the medial temporal fusiform gyrus (medFG), orbital 2 frontal cortex (OFC) and subcortical regions, including the hippocampus, amygdala, caudate and 3 thalamus. More dorsally, clusters were found in the medPFC, precuneous cortex, cingulate gyrus, 4 precentral gyrus and postcentral gyrus, right angular gyrus (AG), posterior part of the STG, bilateral 5 insular cortex and primary auditory cortex. The overlap analysis showed the task invariant neural 6 representation of the graph-CN RDM in the bilateral ATL, including the TP, aFG, aPHG, aMTG and 7 aITG, the OFC and the subcortical regions, including the hippocampus and amygdala. Graph-shortest-path distance. In the oral picture naming experiment, the neural activity 9 patterns in the frontal-temporal cortex were significantly associated with the graph-SP RDM, 10 including the bilateral LOC, pFG, pMTG, left insula and the pars triangularis part of the left IFG. More 11 robust results were found in the word familiarity judgment experiment, which spread across the 12 distributed brain regions, including bilateral temporal regions (with peak effects located in the left 13 aSTG, aMTG, left amygdala), frontal regions (with peak effects located in the bilateral OFC, right 14 caudate and right medPFC) and widespread clusters located in the parietal cortex. The overlap 15 analysis revealed that the task-invariant representation of the graph-SP RDM was located in the 16 bilateral LOC, bilateral pMTG, left pFG, ventral part of the left AG and pars triangularis part of the left 17 IFG, as well as the OFC and insula.

18
Word2vec cosine distance. In the oral picture naming experiment, the neural patterns of 19 bilateral LOC extending into pMTG were found to be significantly correlated with word2vec RDM. In 20 the word familiarity judgment experiment, significant mapping between the word2vec RDM and 21 neural RDMs was found in the bilateral ATL, extending into the medFG, OFC and subcortical regions,   Table S2; word familiarity judgment in Fig. S3b and Table S3). The overlap results across the two 31 experiments are shown in Fig. 2c and Table 1. First-order-edge (simple co-occurrence) distance. In the oral picture naming experiment, the 33 unique neural effects of edge RDM were located in the bilateral LOC, pFG and early visual cortex. In 34 the word-judgment experiment, no regions showed significant effects.  In summary, the raw effects (across fMRI experiments) of the different language computation 3 models were observed in both overlapping and different brain regions. The unique effects analyses 4 revealed interesting dissociations: language graph-CN exhibited unique, task-invariant effects in the 5 bilateral ATL, and language graph-SP exhibited unique effects in the left IFG and left pMTG/ITG (see 6 Fig. 2c for bar plots of the unique effects). Edge (simple co-occurrence) and word2vec did not show 7 overlapping regions of unique effects, i.e., those that cannot be explained by other models. Note that 8 the left SMA showed the unique effects of language graph-SP but did not exhibit significant raw 9 effects, which may result from complicated intercorrelations between these language computation 10 RDMs, and it was not included in the following ROI analyses.

12
Test of language specificity: Comparisons between language and visual computation models 13 To investigate whether the observed neural effects of the language computation models were 14 specific to computing cumulative language-derived information or reflecting certain domain-general 15 computations for cumulative information from any type of input, we constructed the same kind of

26
In the whole-brain searchlight RSA of the visual RDMs (Fig. S4), the only significant cluster was 27 the visual graph-CN in the right TOS (t(25) = 4.96, peak MNI, x = 18, y = -99, z = 21, cluster size = 113 28 voxels) in the oral picture naming experiment under the convention cluster-extent-based inference 29 threshold (voxelwise p < 0.001, FWE-corrected cluster-level p < 0.05). 30 We then focused on the ROIs from the language-model RSA unique effects ( Fig. 2c): bilateral ATL 31 (language graph-CN effect), left IFG and left pMTG/ITG (language graph-SP effect), to test whether 32 they are also sensitive to visual graph-CN and visual graph-SP RDM. The ROI results (Fig. 3c) showed 33 that none of these visual RDMs were significantly correlated with neural RDMs in the language 34 overlapping regions (Ps > 0.096, uncorrected). Furthermore, the RSA results of the language models 35 were preserved after regressing corresponding visual RDMs (Ps < 0.05, Bonferroni corrected).

37
Validation analysis 38 To test the robustness of the language graph representation, validation analyses were 39 performed to address the following concerns: (1) Are the results affected by the graph type Effects of graph types. The language graph was built with weighted edges in the main results, 44 and here, we reran the searchlight-based RSA mapping analysis using the binary versions of edges, 45 graph-CN and graph-SP. Using ROI-based analysis, we found significant unique effects of graph-CN in 46 the bilateral ATL and graph-SP in the left IFG, as well as the pMTG/ITG (Fig. 4a). We also conducted a 47 whole brain searchlight analysis and the overlapping maps across two experiments were similar, 1 except that the unique effects of SP were additionally found in the bilateral parietal cortices (Fig. S5).

2
Effects of window sizes. The main analyses considered immediate co-occurrences (window size 3 = 2). Here, we reconstructed language graphs based on trigram, fourgram and fivegram word co-4 occurrence patterns and reran the searchlight-based RSA mapping analyses in both fMRI experiments.

5
The main patterns of ROI-based results were largely maintained across all window sizes: significant 6 unique effects of graph-CN were found in the bilateral ATL when the window sizes were 2-4, with the 7 effects decreasing when the window sizes changed to 5; significant unique effects of graph-SP were 8 found in the left IFG and left pMTG/ITG (Fig. 4b). Overlap results of whole brain searchlight RSA using 9 different window sizes also yielded comparable results to those in the main analysis.  To understand whether and how the human brain represents word meanings derived from 34 cumulative language inputs, we mainly tested three different types of computations that extract 35 language statistical patterns from a large corpus in fitting word-processing brain activity patterns: 36 simple co-occurrence, two graph-space relations (graph-common-neighbors and graph-shortest-37 path), and neural-network-vector-space relations. In two fMRI experiments, oral picture naming and 38 written word familiarity judgment, which vary by input and output peripheral processes (visual 39 picture recognition, phonological lexical access and output; visual word recognition, button press) 40 but share common word meaning representation, we observed that word relations constructed from 41 all four models correlated words' brain activity patterns across broadly distributed brain regions.

42
However, the word relations derived from a topological raw graph space, and not the other two types, 43 have unique explanatory power for the neural activity patterns in brain regions that have been shown 44 to be particularly sensitive to language processes, including ATL, IFG, and pMTG/ITG. Intriguingly, the 45 organization of different graph relations was respected by these regions with ATL based on the 46 proportions of common neighbors in a graph and with IFG and pMTG/ITG based on the shortest path 47 distances. These neural results of language graph representation were robust across different 1 language co-occurrence measuring window sizes and graph sizes and were relatively specific to 2 language inputs, as they were not associated with relational structures derived from visual co-3 occurrence statistics when using the same computation methods. computations are driving these effects, given the medium-to-high correlations among different types 9 of language statistical models. Our study, by contrast, compared the effects of three different kinds 10 of computations of the language corpus and revealed that graph-related measures, constructed from 11 simple co-occurrences, specifically capture the neural activity patterns in brain regions that are 12 related to language -the effects of simple co-occurrence and w2v could be explained by graph 13 relations but not vice versa. Furthermore, different graph relations are captured by neural activities 14 in different brain regions: the neural activity similarity between two words in ATL is predicted by the 15 proportion of common neighbors these two words have in the word co-occurrence graph; in IFG and 16 pMTG/ITG, the similarity is predicted by the distance of the shortest path between two words in the 17 graph (see below).

18
What do the two graph relations reflect, how are they different from word2vec cosine distances, 19 and what are the implications for the representations in these three brain regions? The graph 20 representation retained the original dimensions given the size of the word co-occurrence matrix 21 (Jackson & Bolger, 2014), with the extraction of statistical information achieved through relationships 22 between highly informative neighbors and paths. In this way, more "historical information" can be 23 preserved. Furthermore, we can select subgraphs or pruning the edges based on word frequency 24 without dramatically changing the network structures, especially the interconnected neighborhoods. of the shortest path than mouse-cat (i.e., it takes more steps to go from mouse to cat than to 46 mousetrap). Thus, brain activity when accessing mice is more similar to that of cats in ATL and to that 47 of mice and mousetrap in IFG and pMTG/ITG. The shortest path captures long-range dependency in 1 graph space -it measures association strength between two non-adjacent nodes; the common 2 neighbors capture structural similarity based on second-order proximity. These two types of 3 statistical properties have been shown to be associated with different types of meaning relations, 4 with CN with taxonomy/semantic categorical similarity and edge with associative relations (Jackson 5 & Bolger, 2014). Here, we also performed an ad hoc analysis on the word sets with rated 6 taxonomical/thematic relations (Xu et al., 2018) and found that CN was correlated with taxonomic 7 relations (Spearman r = 0.48, P < 0.001), weakly correlated with thematic relations (Spearman r = 8 0.10, P = 0.003) after regressing out SP measure, and SP was positively correlated with thematic 9 relations (Spearman r = 0.41, P < 0.001), not taxonomical relations (Spearman r = -0.08, P = 0.98) after 10 regressing out CN measure. These observations align with the findings that word (semantic, we specifically considered relatively simple models that are fully data driven, without prior 47 knowledge such as grammatical information and attentional allocation mechanisms. In recent years, 1 there has been a surge of computational models with improved performances in various language 2 tasks, such as recurrent neural-network models (ELMo) (Peters et al., 2018) and attention neural-3 network models (BERT) (Devlin, Chang, Lee, & Toutanova, 2018), and whether their computational 4 architecture is relevant to brain computations in ATL/IFG/pMTG awaits further study.

5
To conclude, across two fMRI experiments investigating word meaning access, we tested the 6 potential computational mechanisms through which cumulative language experience is captured by 7 the human brain, evaluating the effects of three major types of statistical pattern extracted in the 8 large-scale language corpus. Graph-based topological models had unique explanatory power on 9 words' neural activity patterns beyond simple co-occurrence and vector-embedding models, showing 10 effects in the anterior temporal lobe (capturing graph-common-neighbors), inferior frontal gyrus and 11 posterior middle/inferior temporal gyrus (capturing graph-shortest-path). The latter two did not 12 show effects beyond graph models. These results highlight the role of cumulative language inputs in 13 shaping word meaning representations in this set of brain regions and provide a computational 14 account of how they capture different types of language-derived information.

18
Participants 19 Twenty-nine participants (19 female; median age, 20 years; range, 18-32 years) were recruited 20 in our study and were scanned in 2 fMRI experiments on separate days. All participants were right- Ninety-five objects were selected in our fMRI experiments, including 3 common categories (32 34 animals, 35 small manipulable artifacts and 28 large nonmanipulable artifacts). In two separate fMRI 35 experiments, objects were shown as words and colored pictures (see Fig. S1 and Table S1 for details).  validation analysis, we also adopted different keyword selection methods based on different word 12 frequency ranges and calculated the co-occurrence counts using the top 15%, 20%, 25% and 50% 13 most frequent words of a total of 864,629 Chinese word samples.
14 Importantly, pointwise mutual information (PMI)-normalized word co-occurrence counts 15 between two nodes, u and v, were adopted to construct the 83,007 x 83,007 simple co-occurrence 16 matrix, which reflects the direct proximity between two words in long-term language exposure   where nu is the neighbor node of u, and λ(u, nu) is the weighted PPMI value (edge) between u and nu. 38 The same is for nv and λ (u, nv). 39 For the calculation of weighted shortest path distance (Newman, 2001), we summed weights 40 along the shortest path between two words in a graph-defined space with inverted PPMI using 41 Dijkstra's algorithm in Neo4j (http://neo4j.com/). The measure was precisely calculated as: 1 where λ(u, ni) is the weighted PPMI values (edge) between u and a given node ni..

2
Notably, both first-order edges and common neighbors measure the similarity between two 3 words, while the weighted length of shortest paths measures the distance between two words. In 4 the following analysis, we converted the former two into dissimilarity representations using a 1 minus 5 calculation to obtain the corresponding RDMs.

6
Another proximity measure between words is Katzβ (communicability), which is calculated from 7 ensembles of all paths with a damping factor β to discount the path weights. We did not consider 8 this measure for the following reasons: 1) it is computationally expensive to calculate the global sums 9 over the collection of all paths given the high degree of interconnection in the language graph; 2) the 10 selection of dumping factor β is usually arbitrary and unexplored; and 3) the measure is similar to 11 common neighbors when β is very small (Liben-Nowell & Kleinberg, 2007).

12
Note that the graph-related measures we adopted here were based on the weighted version of 13 graph space (i.e., the edge in the graph was a PPMI-normalized word co-occurrence value) to 14 preserve as much statistical information as possible. In the validation analysis, we also calculated

3
We constructed the following low-level control RDMs (Fig. S1). First, to control for the low-level 4 visual similarity effects between pictures/words, we calculated the pixel and gist dissimilarity of during fMRI scanning were used to construct the button-press RDM based on the absolute difference 10 between the group-averaged familiarity scores of each object pair. The structural images were segmented into different tissue types; the resulting gray matter 2 probabilistic images were coregistered to the mean functional image in the native space, resliced to 3 the spatial resolution of functional images, and thresholded at one-third to obtain the gray mask of 4 each subject. The forward and inverse deformation fields of each subject's native space to the 5 Montreal Neurological Institute (MNI) space were also obtained at this step.

36
The brain results were projected onto the MNI brain surface using BrainNet Viewer

20
The authors declare no competing interests.

24
Supplementary material (Fig S1 -S6; Table S1 -S3).             Table S1 for the 4 corresponding names of these pictures. press) RDM was used to control the task-induced processing. was not the corpus size or content selection that contribute to the observed neural difference 12 between graph and vector embedding models.
13 Table S1. Words (Chinese words and English translations) used in the word familiarity judgment 1 experiment.