This work proposes and analyzes an immune-structured population model of tilapia subject to Tilapia Lake Virus (TiLV) disease. The model incorporates within-host dynamics, used to describe the interaction between the pathogen, the immune system and the waning of immunity. Individuals infected with a low dose acquire a low immunity level and those infected with a high dose acquire a high level of immunity. Since individuals' immune status plays an important role in the spread of infectious diseases at the population level, the within-host dynamics are connected to the between-host dynamics in the population. We define an explicit formula for the reproductive number R0 and show that the disease-free equilibrium is locally asymptotically stable when R0 < 1, while it is unstable when R0 > 1. Furthermore, we prove that an endemic equilibrium exists. We also study the influence of the initial distribution of host resistance on the spread of the disease, and find that hosts' initial resistance plays a crucial role in the disease dynamics. This suggests that the genetic selection aiming to improve hosts' initial resistance to TiLV could help fight the disease. The results also point out the crucial role played by the inoculum size. We find that the higher the initial inoculum size, the faster the dynamics of infection. Moreover, if the initial inoculum size is below a certain threshold, it may not result in an outbreak at the between-host level. Finally, the model shows that there is a strong negative correlation between heterogeneity and the probability of pathogen invasion.