I used to feel this way as well. The few times it happened to me I have passed on animals that would have ended up being my largest, and have taken some to then feel a bit bummed by the fact that the hunt was over so quick.
Like you, it has not been a big issue for me, because I generally have terrible luck. So, I am way more likely to go into overtime, or eat tag soup despite hunting more hours than anyone at camp.
However, (and maybe because of my luck) I have shifted my perspective on this. I now think, if you consider how many things in life are incredibly difficult, and more difficult than they have to be, why pass up an opportunity and a gift from god (or whoever you want to attribute it to), on the rare occasion you are given an easy win. Take the easy win, be grateful and show gratitude for it, and enjoy the moment. At least that is how I look at it now.