Putting a change of medium between source and detector provides another particular line in the space the photon explores. Try waypoints along the surface where the medium changes. Again explore the contributions from triplets—do the contributions line up or curl up?
Again, there is a small region of the surface from which arrows line up, suggesting that all other regions could be more or less ignored, or covered up. So it's as if a beam of light is bent as it enters the medium. This effect is refraction, and now you have a reason for it.
Where the differences in trip times between adjacent paths are small, the arrows will line up, because they've spun by nearly the same amount. Explore how the trip times vary for refraction.
The differences in trip time are smallest for paths near the paths where the trip time is least. It's just geometry. Each photon explores the space between source and detector. The properties of the space set contributions from the paths. These contributions sum to set the chance that the photon arrives at the detector.
The least-time path is not a straight line because the arrows spin more times per metre in one medium than the other. To make up for this, the path in one medium is shorter than the other.
The arrows spin at the same number of times per second in both media because you don't change the colour at the detector.