You have to fight against 6 or 7 troops with 15 attackpower (35 pop / 5 = 7 troops). Your troops only have an attackpower of 10. In one fight-round you only have a winning chance about 25%. (You have 3d10 and the minor has 3d15 [d = dice]). Because the troops from the minorrace are created from population, they have full power and health after each fight-round. Often one troop of the minor kills about 3 till 5 troops of yours. That's the random in calculation. Most of the time your invasion is successful. But you are losing many troops. Sometimes it isn't successful, because you lost all of your troops.
But an invasion was planned by me to be a little bit randomly. You should never be too sure to be successful.