Integrative Analysis of Histological Textures and Lymphocyte Infiltration in 1 Renal Cell Carcinoma using Deep Learning

ABSTRACT

classification errors were related to tiles composed of a mixture of textures. For 117 instance, hemorrhage is a typical histopathological finding in ccRCC due to 118 neovascularization and structural instability of rapidly growing blood vessels 25 . 119 Therefore, the misclassification of blood tiles as cancer texture was likely due to their 120 co-occurrence. The algorithm misclassified only 2 out of 1,212 (0.2%) cancer texture 121 images as normal renal tissue and 8 out of 645 (1.2%) normal renal tissue images as 122 renal cancer indicating excellent distinction. 123 124 125 TCGA clinical sites differ by their texture characteristics 126 TCGA ccRCC samples have been collected from 16 participating clinical sites. The 127 tissue procurement protocol has been described 14 . However, the actual sampling 128 uniformity remains unknown, although this may significantly hamper analysis of TCGA 129 samples.

131
By comparing the texture composition of samples by their clinical site, we observed 132 that the proportion of cancer texture varied 4-fold between 22.0-80.0% ( Fig. 1f-g). We 133 noted substantial differences also in other texture types (Extended Data Fig. 2). A 134 median of >5% normal renal tissue was evident only in samples originating from the 135 MD_Anderson_Cancer_Center, BLN_Baylor and MSKCC indicating that histological 136 slides have not been submitted to TCGA with the same protocol ( Fig. 1f and Extended 137 Data Fig. 2). 138 139 We excluded 51 samples from 48 patients (9.6%) as these did not represent typical 140 ccRCC histology or included <5% cancer texture ( Fig. 1b and Supplementary Table  141 1). The reason for low cancer proportion was atypical or non-ccRCC histology (n=22), 142 poor histological quality (n=21), necrotic sample (n=4), and lack of cancer tissue (n=4). 143 These findings highlight the priority of computational review to ensure high sample 144 quality.

146 147
Tissue hemorrhage is associated with lower metastasis rate, less frequently mutated 148 mTOR and lower infiltration of regulatory T-cells 149 Next, we aimed to resolve the textural landscape of ccRCC patients and the 150 determinants regulating its composition. As expected, renal cancer tissue was the most 151 prevalent texture ( Besides tumor biology, the varying site-specific sampling protocols likely affected the 158 textural content. We reasoned that by dividing TCGA ccRCC samples into those 159 without (<1%) and with normal tissue (≥1%) we could examine tissue textures in two 160 distinct but histologically more coherent cohorts (Fig. 2b-c). Samples without normal 161 renal tissue (N-) were composed of 66.3% [IQR 46.9-82.8%] cancer texture 162 representing the tumor core compared with 41.6% [27.9-56.0%] in samples with 163 normal renal tissue (N+) reflecting a broader tumor microenvironment ( Fig. 2d and  164 Extended Data Fig. 3b-d).

166
As the proportion of normal renal tissue reflected tissue sampling practices and the 167 "other" texture class included various histological types, we focused on blood and 168 stroma-associated phenotypes. Higher hemorrhage in N+ samples was associated 169 with lower organ metastasis, lower tumor stage and superior Eastern Cooperative 170 Oncology Group (ECOG) performance status ( Fig. 2e-f). We also noted that the 171 peripheral blood (PB) platelet count gradually decreased with increasing proportion of 172 tumor hemorrhage in N+ samples (Fig. 2g). Elevated pretreatment platelet level is a 173 biomarker of poor survival and is incorporated in the prognostic Heng score 26 . Thus, 174 low PB platelet count could be due to high angiogenic activity and consequent tumor 175 hemorrhage.

177
When examining N-samples, we observed similar but less significant association of 178 tumor hemorrhage and lower rate of organ metastasis and lower PB platelet count (Fig.  179 2e). Tumor hemorrhage was not associated with overall survival in either N+ or N-180 cohorts (Extended Data Fig. 4).

182
We investigated next genomic alterations. In N+ samples, mTOR mut occurred in 24/396 183 patients (6.1%) and associated with lower hemorrhage (Fig. 2h). While mTOR has 184 been described to increase angiogenesis via HIF1a and VEGF-regulated pathways, 185 VHL mut was not associated with hemorrhage indicating another mechanism. No 186 association with other gene alterations, mutation burden or aneuploidy was observed 187 (Supplementary Table 2). No association between tumor hemorrhage and genomic 188 variables were observed in N-samples (Supplementary Table 3).

190
To identify hemorrhage-associated transcriptomic signatures, we first compared the 191 expression of individual genes ( Fig. 2i and Extended Data Fig. 5a). In N+ samples, the 192 HMOX1 gene was highly expressed in conjunction with hemorrhage ( Fig. 2i). HMOX1 193 encodes the heme oxygenase 1 enzyme catalyzing heme to biliverdin and increased 194 heme catabolism is consistent with increased tissue hemorrhage 27 . We observed 195 upregulated inflammatory, epithelial-to-mesenchymal (EMT) and hypoxia-related 196 pathways in samples without normal tissue but little difference in immune profiles (Fig. 197 2j and Extended Data Fig. 5b Tissue fibrosis is associated with high histological grade, low mutation burden and an 203 adaptive immune response 204 Next, we examined fibrosis-related manifestations. In N-samples, we observed 205 association between stroma and histological grade ( Fig. 2e and Fig. 2k). Similar 206 relation was less evident in N+ samples indicating that intratumoral but not peritumoral 207 stroma would be linked with poor renal cell differentiation (Fig. 2l). In line, N-stroma 208 associated with other established adverse prognostic biomarkers such as tumor size, 209 stage and anemia (Fig. 2e).

211
When studying genomic alterations, mutation burden was linked with lower histological 212 fibrosis both in N+ and N-samples ( Fig. 2m and Supplementary tables 4-5). In N+ 213 samples, PBRM1 wt and diploid haplotype were also associated with higher proportion 214 of stroma ( Fig. 2n   Extended Data Fig. 5g-i). As mutation burden and PBRM1 genotype have been 222 implicated with immunotherapy response, quantifying fibrosis could provide an 223 inexpensive biomarker to increase their precision or select patients for targeted 224 sequencing 28-30 .

226
Next, we inspected fibrosis-associated transcriptional programs. As expected, the 227 TGFb response pathway was enriched both in N+ and N-samples with high stromal 228 composition (Fig. 2j). High histological stroma associated only in N+ samples with 229 transcriptome-derived stromal score (Fig. 2j). We reasoned this to be due to more 230 abundant stroma in N+ compared to N-samples as visually confirmed by large 231 peritumoral stromal margins compared to intratumoral fibrotic islets (Extended Data 232 Fig. 6a).

234
Established stromal signaling pathways regulating EMT and the formation of integrin, 235 syndecan, and collagen were elevated in both N+ and N-samples (Fig. 2o). Moreover, 236 fibrosis was associated with coagulation and less active lipid metabolism (Fig. 2o). The 237 chromosomal locus 19p13 was overactivated in conjunction with enriched stroma in N-238 samples, while hypoxia and angiogenesis-related pathways were elevated in N+ 239 samples with high stroma (Fig. 2o). Similar findings were observed also at the gene-240 level ( Fig. 2p and Extended Data Fig. 6b). While many of genes were associated with 241 stroma formation in both sample categories, integrin alpha 11 (ITGA11), ephrin 5 242 (EFNA5) and cytokeratin 19 (KRT19) were enriched in N+ samples whereas aggrecan 243 (ACAN) in N-samples. These findings suggest that the extracellular matrix (ECM) in 244 the intratumoral and peritumoral tissue could differ by their adhesion, migration and 245 cell signaling abilities (Fig. 2q).

247 248
Lymphocyte infiltration is coordinated between malignant and surrounding tissue 249 Previous studies quantifying tumor lymphocyte infiltration in ccRCC have relied on 250 deconvolution of bulk RNA-sequencing data 17 , histochemical, 31 or antibody-based 251 detection such as flow cytometry 32,33 . While these approaches are precise to 252 approximate the lymphocyte population in a sample, they impose demands on uniform 253 sampling and sample processing.

255
Here, we built a deep learning model identifying images with high (90.3% classification 256 accuracy) and low (96.1%) lymphocyte density in the test set (Fig. 1e). We 257 hypothesized that texture-aware lymphocyte quantification could solve issues related 258 to sampling variation. The lymphocyte classification probability per tile reflected 259 lymphocyte density. Therefore, lymphocyte predictions [0-1] were proportioned by 260 texture surface area. The median lymphocyte proportion per sample was 20.3% and 261 varied between 2.3-82.5%. The highest lymphocyte density was unexpectedly in the 262 normal renal texture followed by cancer, stroma, blood, and lastly other texture types 263 (Fig. 3a). Texture-specific lymphocyte proportions shared high positive correlation ( Fig.  264 3b). Intratumoral infiltration explained most of the total sample infiltration variance 265 (0.85 2 = 73%; Fig. 3b).

267
Next, we rescaled the texture-specific lymphocyte densities so that their sum would 268 equal to 100% and observed heterogenous distribution at the patient-level (Fig. 3c). 269 Strikingly, the intratumoral lymphocyte density correlated negatively with the density in 270 the surrounding normal renal tissue (R -0.56, p<0.001; Fig. 3d) and stromal texture (R 271 -0.37, p<0.001) indicating controlled movement between the tumor and its surrounding 272 textures.

274 275
Stromal lymphocytes are associated with poor survival and high T-cell receptor 276 diversity 277 Texture-specific lymphocyte proportions differed by the proportion of normal renal 278 tissue (<1% vs. ≥1%; Fig. 3e and Extended Data Fig. 7a-c). To equalize sampling 279 differences, we normalized lymphocyte density with texture-specific weights (see 280 Methods) successfully reducing differences in the total lymphocyte infiltration by 281 clinical center (Extended Data Fig. 7d-e). However, samples originating from Fox 282 Chase contained higher lymphocyte density than expected (Extended Data Fig. 7e). 283 When examined visually, these samples were characterized with a high 284 hematoxylin:eosin ratio and lymphocyte scoring even visually was ambiguous 285 (Extended Data Fig. 7f-g). As a conclusion, these samples were excluded from 286 lymphocyte analyses.

288
To fully normalize staining differences, we categorized samples into "High" and "Low" 289 infiltration based on their lymphocyte density compared to the clinical center median 290 density. When examining clinical significance, we observed that lymphocyte infiltration 291 in cancer and stromal textures was associated with high histological grade (Fig. 3f). 292 Only stromal lymphocyte infiltration was related with poor overall survival, organ 293 metastasis and stage IV disease ( Fig. 3f-g and Extended Data Fig. 8). Patient gender, 294 age, smoking status, or laboratory values were not related to lymphocyte infiltration 295 (Fig. 3f).

297
We then studied genomic and immunological profiles. In line with our expectations, 298 infiltration in malignant renal tissue was associated with the transcriptomic CD8+ T-cell 299 signature (Fig. 3h). We also observed a CD8+ high /M2-macrophage low combination in 300 patients with high lymphocyte infiltration notably in the cancer texture ( Fig. 3h). B and 301 T-cell receptor diversity was associated with increased lymphocyte infiltration almost 302 irrespective of textural context (Fig. 3h). Of note, cell proliferation correlated negatively 303 with all except stromal lymphocyte infiltration (Fig. 3h).

305 306
Aneuploidy, chromosome 1p and 5q loci, and the EMT program identified as regulators 307 of tumor-infiltrating lymphocytes 308 Next, we examined genomic alterations associated with lymphocyte infiltration. 309 Aneuploidy was associated with higher intratumoral and total lymphocyte density ( Fig.  310 3i and Extended Data Fig. 9a). Mutation burden was related to higher infiltration to 311 blood texture (Extended Data Fig. 9b). When studying individual genes, SETD2 mut 312 trended with higher infiltration in stromal, other and blood textures (Extended Data Fig.  313 9c-e). The PBRM1 genotype has been associated with both nonimmunogenic 30 and 314 immune hot phenotype and anti-PD1 therapy response 29 . In our study, PBRM1 315 alterations were not associated with total or texture-specific lymphocyte infiltration 316 (Extended Data Fig. 9f-k). By analyzing the supplementary data provided by Braun et 317 al 29 , no association was evident between the tumor core CD8+ density and PBRM1 318 status validating our finding and indicating that its immunologic significance remains 319 unclear (Extended Data Fig. 9l). 320 321 Finally, we studied the transcriptomic programs associated with lymphocyte infiltration. 322 The top three pathways enriched with intratumoral infiltration were well-established T-323 cell activation signatures endowing confidence to our analysis (Fig. 3j). The 324 chromosomal locus 1p36 was the next most overactivated pathway while the 5q31 and 325 5q13 loci and metabolic pathways were most downregulated (Fig. 3j). 326 327 Given our previously described asynchronous lymphocyte enrichment in either 328 malignant or normal renal tissue, we interrogated which pathways are most commonly 329 altered in these two compartments. Adaptive immune response pathways were 330 activated in lymphocyte-rich samples (Fig. 3k). Exclusively in samples with 331 lymphocyte-rich cancer texture, the IL-12 pathway, and genes of the chromosome 332 1p36 locus were upregulated and genes of the chromosome 5q13 and 5q31 333 downregulated. On the contrary, the hallmark EMT pathway was enriched, while lipid 334 and energy production pathways depleted in lymphocyte-rich normal renal tissue (Fig.  335 3k). The cytolytic granzyme A (GZMA) enzyme illustrious of T and NK-cells and the 336 testis antigen SPAG5 were associated with infiltration to cancer tissue (Fig. 3l). On the 337 contrary, genes regulating the ECM and fibrosis such as MMP19 and CD44 were 338 upregulated in samples with infiltration to normal renal tissue (Fig. 3l). In summary, 339 immune, mesenchymal, and metabolic factors influence lymphocyte infiltration. 340 341 342 The textural composition of the tumor margin predicts prognosis and reflects the 343 tumoral genomic and transcriptomic alterations 344 Based on our previous findings that the tumor core and its surrounding peritumoral 345 tissue form two immunologically distinct regions 34 . Therefore, we were intrigued to 346 investigate the textural content of the peritumoral margin and its exterior non-margin 347 region (Fig. 4a).

349
The peritumoral texture was dominated by stroma and followed by textures in the same 350 order as observed in the entire sample (Fig. 4b). Blood and stroma textures were more 351 frequent in the tumor margin compared to both the non-margin region and the entire 352 sample ( Fig. 4c and Extended Data Fig. 10a). In contrary, normal renal tissue was less 353 common in the tumor margin than in the entire sample ( Fig. 4c and Extended Data Fig.  354 10a). These findings are concordant with previous findings on the fibrous capsule and 355 tumor-penetrating stromal islets (Extended Data Fig. 6).

357
The margin composition was heterogenous at the patient-level (Fig. 4d). We assigned 358 each patient with a textural enrichment score by comparing texture proportions 359 between the tumor margin to the non-margin tissue. This approach was unaffected of 360 technical differences in tissue staining as two regions of the same sample are 361 compared. Patients with a high margin:nonmargin normal renal tissue ratio had 362 significantly worse survival (Fig. 4e). The clinical profile of these patients included 363 higher tumor size, histological grade, organ metastasis and advanced stage (Fig. 4f). 364 These tumors shared also higher proliferation, heterogeneity and leukocyte fraction 365 including more frequent and clonally diverse memory B-cells (Fig. 4g).

367
In the opposite, patients with elevated blood texture in the tumor margin were 368 characterized with superior survival (Fig. 4h). These patients shared known favorable 369 clinical characteristics such as lower tumor size, more differentiated tumor cells and 370 less frequent thrombocytopenia (Fig. 4f).

372
The prognosis of patients with a high peritumoral margin:nonmargin stroma did not 373 differ from other patients, but were older at diagnosis (Fig. 4f and Extended Data Fig.  374 10b). Immunologically, these were characterized with a broader T-cell clonality 375 spectrum and more frequent M1-polarized macrophages (Fig. 4g). 376 377 Next, we examined the mutational landscape of the margin histological subtypes (Fig.  378 4i). We then asked how the expression of individual genes and signaling pathways would 388 impact the peritumoral textural composition. While the findings at single gene level 389 were challenging to interpret, we observed distinct pathway signatures (Extended Data 390 Fig. 10c-e). In samples with fibrotic margin, IFNg signaling, immune responses and the 391 EMT were activated resembling the previously described normal renal tissue 392 lymphocyte signature (Fig. 4j). High margin:nonmargin normal renal tissue was 393 associated with enriched cell cycle genes and downregulation of lipid and carbohydrate 394 metabolism (Extended Data Fig. 10d). Margin:nonmargin hemorrhage correlated with 395 downregulation of the cell cycle, chromosome 17q21, 17q25 and 6p21 loci, and MTOR 396 and MYC signaling ascertaining the link with prognosis (Extended Data Fig. 10e). 397 398 399 The lymphocyte-rich stromal margin associates with an adaptive immune response, 400 dampened EMT and non-smoking habit 401 To conclude, we quantified the margin:non-margin lymphocyte ratio. The highest 402 lymphocyte density was in the normal renal tissue (Fig. 5a). The texture order was 403 similar than in the non-margin and the entire sample, but the lymphocyte density was 404 higher in the tumor margin ( Fig. 5b and Extended Data Fig. 11a).

406
The lymphocyte density correlated over margin textures (Fig. 5c). The stromal 407 lymphocyte density explained most (82.8%) of the variance of total margin lymphocyte 408 density (Fig. 5c). Elevated stroma in the margin was associated with higher margin 409 lymphocyte density (Fig. 5c). On the contrary, the margin histological textures 410 correlated negatively with one another indicating that if one texture type increased, the 411 remaining textures decreased accordingly (Fig. 5c-d). This was especially evident 412 between stroma and normal renal tissues (Fig. 5d-e).

414
At first, we observed high concordance between the cancer texture and peritumoral 415 lymphocyte density (R=0.78 p<0.001, Fig. 5f). We replicated the finding in our 416 previously reported Helsinki dataset of 64 ccRCC patients 34 where instead of a cancer 417 texture and its margin we had annotated the intratumoral and peritumoral regions 418 (R=0.60 p<0.001, Fig. 5f). However, in closer inspection we could discern two 419 linearities and, therefore, divided patients into two groups: 25% highest and 75% 420 lowest tumoral lymphocyte density. Indeed, patients with high intratumoral infiltration 421 had lower peritumoral lymphocyte density suggesting distinct regulatory mechanisms 422 for lymphocyte penetration (Fig. 5f).

424
We studied next clinical correlates. The margin lymphocyte ratio was not clearly 425 associated with overall survival or prognostic variables ( Fig. 5g and Extended Data 426 Fig. 11b). Infiltration in the stromal margin was decreased in active smokers and partly 427 normalized in ex-smokers (Fig. 5g). It was also associated with a weaker TGFb 428 response and less abundant Th2-differentiated T-cells and macrophages (Fig. 5h).

429
Lymphocyte abundance in the normal renal margin tissue was instead associated with 430 monocytes and Th1 T-cells and suppression of regulatory T-cells and TGFb pathway 431 (Fig. 5h).

433
We observed that mutation burden was associated with infiltration to the stromal 434 margin compared to the non-margin tissue (ratio 1.8 vs. 1.6, p=0.068; Fig. 5i Lastly, we studied the transcriptomic signature of the peritumoral lymphocyte infiltration 442 (Fig. 5j, Extended Data Fig. 12). We observed activation of interferon signaling and 443 adaptive immune response and downregulation of EMT and other mesenchymal 444 pathways in samples with a lymphocyte-rich stromal margin (Fig. 5j). The immune-445 related genes included the transcriptional activator of interferon-inducible genes IRF1, 446 the class II human leukocyte antigen CIITA required for antigen presentation and 447 immunoregulatory genes LILRB1 and LILRB2 (Fig. 5k) signaling and alterations in commonly mutated genes (Fig. 6).

471
Half of the lymphocytes in sections were located among cancer cells and shared 472 moderate concordance across textures. Aneuploidy, the IL-12 pathway, chromosome 473 1p36 activity and chromosome 5q13 and 5q31 inactivity were linked to intratumoral 474 infiltration. Copy number losses of 1p36 and gain of 5q35 are highly common in 475 ccRCC, and the latter has been suggested to induce activation of the activation of 476 PI(3)K pathway 15 . These loci might have now another immunoregulatory role.

478
The relative proportion of lymphocytes in cancer tissue correlated negatively with their 479 relative proportion in the surrounding normal and stromal tissue textures possibly 480 indicating two gravitation centers. We have previously shown that intratumoral T-cells 481 are immunophenotypically more experienced based on higher expression of cytolytic, 482 immune checkpoint and senescence markers and closer intercellular proximity 483 compared to the peritumoral and normal renal regions 34 . In both the previous and 484 current study, we demonstrated that peritumoral and intratumoral lymphocyte densities 485 correlated if the intratumoral density was low. Possibly due to restricted lymphocyte 486 penetration, elevated stromal lymphocyte infiltration associated with higher metastatic 487 rate, poor survival, T and B clonotype diversity, and increased cell proliferation (Fig.  488 6). In summary, peritumoral lymphocyte aggregates could act as a lymphocyte 489 reservoir, but could be unleashed by combining fibrolytic drugs to immune-based 490 therapies.

492
The comprehensive annotation data and algorithms are available to expand the 493 textural and lymphocyte analyses to other datasets, cancer types or even to a pan-494 cancer study. In summary, this study highlights how computational analysis of routine 495 H&E staining can help to detect sampling fallacies, quantify tumor-infiltrating and 496 tumor-surrounding lymphocyte infiltration, and discover novel histopathological 497 associations both in the entire tumor sample and in the tumor margin. 498 499 500

501
TCGA ccRCC patient cohort 502 We collected histology images from TCGA portal (https://portal.gdc.cancer.gov/) and 503 clinical and processed transcriptome data from https://gdc.cancer.gov/about-504 data/publications/panimmune 20 and https://gdc.cancer.gov/node/905/ (PanCanAtlas; 505 Fig. 1a). Samples were digitized mainly with an imaging resolution of ~0.25 mm/px 506 (n=497), except for 18 samples scanned at ~0.50 mm/px, which were excluded (Fig.  507 1b, Supplementary Table 1). A feature matrix of all available data was built, consisting 508 of only numeric or binary features as rows, with missing data reported as NA. 509 Categorical variables were transformed into binary factors. 510 511 The somatic mutation calls (SNVs and indels) were collected from the supplementary 512 First, we determined the main texture classes to annotate. We prioritized classification 528 reliability and therefore minimized the number of tissue classes as often tiles included 529 multiple texture types and some histological patterns commonly co-occurred, for 530 example smooth muscle, fibrous stroma, and blood vessels. In addition, some patterns 531 were challenging to differentiate reliably from individual tiles without larger context 532 information for example torn, adipose, and necrotic tissue. We ended up with the 533 following texture classes: renal cancer ("cancer"; n=11,755 image tiles, 29.7% of all 534 tiles); normal renal ("normal"; n=6,313, 16.0%); stromal ("stroma"; n=3,027, 7.7%) 535 including smooth muscle, fibrous stroma and blood vessels; red blood cells ("blood"; 536 n=544, 0.9%); empty background ("empty"; n=11,609, 29.4%); and other textures 537 including necrotic, torn, and adipose tissue ("other"; n=6,210, 15.7%; Fig. 1c). We 538 annotated in total 39,458 randomly-selected tiles sized 300 × 300 px located in the 539 center of a larger 900 × 900 px image to improve the annotation accuracy. 540 541 To quantify lymphocytes, we annotated 256 × 256 px tiles (n=25,095) containing none 542 or few lymphocytes as "Low" (n=20,092, 80.1%) and the rest as "High" (n=5,003, 543 19.9%; Fig. 1d). As areas of high lymphocyte density were substantially less common, 544 we speeded up annotation by extracting regions of high lymphocyte aggregates and 545 from regions of low lymphocyte infiltrate using the open-source software QuPath 22 546 0.2.0. We selected ~20 digital TCGA samples originating from various clinical sites. All 547 texture and lymphocyte images were evaluated twice to minimize annotation errors. 548 549 550 Texture classification 551 For texture classification, we trained a multi-class CNN. We employed the deep 552 residual network ResNet as it has been commonly-used in computer vision tasks 23 . 553 Transfer learning is the process of repurposing parameters of a previous algorithm to 554 optimize training on a new dataset 24 . Here, we adapted transfer learning by combining 555 the ImageNet-pretrained ResNet-18 infrastructure with a fully-connected layer, a 556 rectified linear activation function (ReLU) activation and a softmax layer for prediction. 557 Training occurred at all CNN layers, with the Adam optimizer tuned with a fixed learning 558 rate of 10 -4 , batch size 4, and the cross-entropy loss function until the validation loss 559 did not decrease for 5 consecutive epochs. We randomly cropped 256 × 256 px tiles 560 from the annotation images and augmented these with horizontal-vertical rotation and 561 without balancing texture classes. Models were composed with Python 3.9.1. with 562 libraries Pytorch 1.9, Torch 1.11.0 and Torchvision 0.12.0. 563 564 The classification resulted in tessellated texture areas (Extended Data Fig. 13). For 565 instance, cancer regions were disrupted by sporadic tiles of other textures. To smooth 566 texture masks, we slid the 3 × 3 tiles' window size and stride of 2 over the texture map 567 and unified the texture class in each window by the most common texture. In some 568 occasions, two tissue textures occurred equally often for instance 4 cancer, 4 stroma 569 and 1 blood tiles. If the most common class was a tie between cancer and another 570 texture, the cancer class was prioritized. If the most common class was a tie between 571 stroma and another texture, the stroma class was prioritized except if the other was 572 cancer. Stroma and cancer textures were prioritized as these occurred most in tiles of 573 multiple textures. In other cases of tie, the pooled texture type was randomly selected 574 from the equally most occurring textures. 575 576 577 Lymphocyte classification 578 For the lymphocyte classification, we trained a binary-class CNN using the same model 579 infrastructure and hyperparameters as was used in the texture classification. However, 580 to quantify the lymphocyte infiltration in a continuous range [0-1], we used the argmax 581 function on the sigmoid layer. Therefore, no post-pooling of lymphocyte masks was 582 performed. 583 584 585 Tumor margin 586 To analyze the histological and lymphocyte content immediately exterior to the tumor, 587 we defined the tumor margin as the first two non-cancer tiles around each cancer tile 588 with the maximum_filter function of the Python Scipy 1.8.1. library. The margin was 589 512 px or ~128 µm wide. For reference, the average lymphocyte diameter is ~10 µm. 590 The remaining tissue not included as tumor or tumor margin was classified as the non-591 margin tissue. To avoid sampling bias, we included only samples with ≥1% normal 592 texture. 593 594 595 Model metrics 596 We divided annotation datasets into training (70%), validation (20%), and test (10%) 597 sets. The final model fitness was evaluated in the test set by comparing classification 598 accuracy and a confusion matrix. 599 600 601 Statistical analysis 602 Median and interquartile ranges (IQR) were used to report average values and ranges.

603
We Total lymphocyte infiltration normalization 614 The texture-specific lymphocyte infiltration varied by clinical centers partly due to 615 differing texture proportions reflecting sampling conventions (Extended Data Fig. 7d).

616
We harmonized total lymphocyte infiltration in two steps (Extended Data Fig. 7e). First, 617 we multiplied the proportion of each texture area by its relative lymphocyte proportion. 618 The relative lymphocyte proportion reflects the lymphocyte enrichment to each texture 619 in comparison to other textures. Thus, the sum of relative lymphocyte proportion in 620 blood + cancer, + normal + blood + other = 100%. In the second step, we divided the 621 sum of the normalized lymphocyte proportions by the total sample area e.g. 622 623 where represents the relative lymphocyte proportion (%) and the area (px) of the 626 texture , which is part of the texture list ∈ [ , , , , ℎ ].

628
To fully normalize staining differences, we categorized samples into "High" and "Low" 629 infiltration based on their lymphocyte density compared to the clinical center median 630 density. Small batch size increases the high risk for nonparametric data distribution. 631 Therefore, we included only centers with more than 20 samples to the lymphocyte 632 analyses. 633 CODE AND DATA AVAILABILITY.