Definitely shifts to B minor, but "modulates" (to my mind anyway) suggests an actual key change. B is the relative minor to the key of D (2 sharps), so there is no actual key change there -- equivalent to shifting to A minor in the key of C -- all the same notes appear in both scales.
If my flapper ears doesn't cheat me, I hear the following chords with the banjo.
In the first and second part lots of D and very short A, and reminds me immediately to Sally Goodin or Indian on a Stomp
In the third part I hear Bminor for sure and A.
If this is called a modulation , I'm not sure.
That's because the A-chord doesn't change
I thought that a modulation changed the key of a whole song ?