Oh man, is it baseless speculation time? Okay here's mine.

Titans eat shifters to regain their humanity. Pretty standard bullshittery, right? Moving on.

All the titan shifters ate their parents. The longer a titan lives, be it in human or titan form, the harder it is to control the titan/ that driving need that you use to first transform is less and less legible until you become essentially a wild titan.

They get around this by passing it on to their kids, using whatever Grisha's serum is to induce an initial transformation and then once they're devoured, their memories and ability to transform back and forth are absorbed by the child/titan. Ymir broke this cycle of transference when she ate one of the shifters and "stole the power" that was to be passed down within the village.

Grisha had fled the village, or perhaps was a sleeper agent like BRA who had a change of heart. When the walls came down he realized the jig was up and passed his power off to Eren, since he couldn't use it safely anymore, and gave him the amnesiatic to suppress the memories of the previous holders because A) Eren's a kid and he'd probably f*** it up B) because it would bring him to the attention of interested parties (Grisha would recognize that BRA are probably already skulking about) before he was ready to wield it properly.

cont.

Annie was too young to properly learn martial arts from her dad, but after eating him retained his muscle-memory and gained his skills. His line about always being on her side is sort of a "a part of me will always be with you haha" thing.
The reason Berholdt was so passive when talking about how he felt with Eren's mom is because he had to eat his OWN mom for the mission. Get over it Eren, your shit's weak.
Reiner belonged to a line of warriors and guardians, but the memories of being a warrior for the village clash with HIS personality which was better suited to growing up as a soldier within the Walls.

The reason why they all talk about going back to the village, and not back to their families, is because their families all got ate, and they're all they have now. They're all determined to see the mission through because they paid a terrible cost already and don't want it to be for nothing. Bert doesn't understand familial attachment too well, but when his friends start guilt tripping him he loses it, because they were the closest thing he had to a family.