The analysis of age-specific genetic effects on human survival over extreme ages is confronted with a deceleration pattern in mortality that deviates from traditional survival models and sparse genetic data available. As human late life is a distinct phase of life history, exploring the genetic effects on extreme age survival can be of special interest to evolutionary biology and health science. We introduce a non-parametric survival analysis approach that combines population survival information with individual genotype data in assessing the genetic effects in cohort-based longitudinal studies. Our approach is characterized by non-parametric analysis of late age survival to capture the observed pattern of mortality deceleration and frailty modeling to account for individual heterogeneity in unobserved frailty. The method is applied to ApoE genotype data in the Danish 1905 birth cohort to estimate effect of the e4 allele. Our results revealed an age-specific relative risk of the allele that increases nonlinearly with age and non-proportional patterns in hazard of death for carriers and non-carriers of the allele, suggesting that the e4 mutation preserves its deleterious effect that progressively affect human survival even at extreme ages.