Obesity is a well-established risk factor for colorectal cancer (CRC), but the association with the tumor microenvironment has been sparsely described. Herein, we examined the relationship between pre-diagnostic anthropometry and CRC risk according to tumor immune cell composition, with particular reference to potential sex differences. The density of different immune cell subsets was assessed by immunohistochemistry in tissue microarrays with tumors from 584 incident CRC cases in a prospective, population-based cohort (n = 28098). Multivariable Cox regression models, adjusted for age, smoking, alcohol intake, and educational level, were applied to calculate risk of immune marker-defined CRC in relation to quartiles of pre-diagnostic height, weight, body mass index (BMI), waist and hip circumferences, waist-hip ratio (WHR), and body fat percentage (BFP). Obesity was all over significantly associated with risk of CRC with low density of FoxP3+ T cells and low programmed cell-death protein 1 (PD-L1) expression on tumor cells, but with high density of CD8+ T cells and CD20+ B cells. In women, obesity was significantly associated with risk of PD-L1 high tumors (p= 0.009 for weight, p= 0.039 for BMI). Contrastingly, in men, obesity defined by all anthropometric factors was significantly associated with PD-L1 low tumors (p= 0.005 for weight, p = 0.002 for BMI, p<0.001 for waist, p= 0.011 for hip, p<0.001 for WHR, and p= 0.004 for BFP). In summary, obesity appears to influence the immune landscape of CRC, possibly in a sex-dependent manner. Thus, anthropometry and sex may be important factors to take into account when assessing the prognostic or predictive value of relevant complementary immune biomarkers.