Herein lies the conundrum, my friend!

A lightweight .375 bolt-action is generally easier to carry, aim, shoot and cycle than a heavy, 470, or a .500 double, let alone a .577.
Now, if we agree on that much, the next question is whether one can realistically
carry, shoot and reload those big bores as quickly and effortlessly as a .375 bolt-action.
If you reckon that’s entirely possible, or that the big bore can actually be quicker, then I’ll happily rest my case and let you have the last word!
No offence intended, my friend. Just a bit of friendly campfire debate!
My .450 Dakota 76 (without scope) and 4rds weighs 10.8lbs.
Buffalo round is 500gr/2350fps.
It produces 76.8 ft-lbs Free Recoil Energy at 21.4 fps Free Recoil Velocity.
It feel about like a 10ga Mag shotgun to me. Stout, but not punishing.
I'm just under 6' and ~185lbs
sans clothes; in the field about 220lbs with daypack.
Here's a .gif file of my second shot on buffalo. The recoil and muzzle rise were not excessive. If I of average height/weight can handle a 500gr/2350fps like that, probably most hunters could if they learned how and practiced.
Of course one needs a bit of extra time to recover from it vs. a .375H&H, Newton's Law and all that.
However, IMO, the extra caliber and power are worth it to me, and the marginal quickness of a .375H&H would not be determinative in most DG scenarios.
In IPSC and other defensive handgun competitions, the shooters vastly prefer the 9x19 due to its lower recoil and speed. I've made the point in other forums that shot pairs with a .40S&W would take only 0.05 seconds longer (half an eyeblink), and a 10mm only 0.10 seconds longer (an eyeblink). Those 9mm gamers are so focused on speed at all else that they skimp on power. I'd rather have a .40 or 10mm in a gunfight; the half- or full-eyeblink of speed penalty won't make any difference.
When I waggishly admonish the 9mm Boyz to forsake it for the smaller .30SS because it's even faster to wield, they clam up because I've exposed their fallacious doctrine.
DVC (Power, Accuracy, Speed) is a formula of
balance. I think most could effectively wield the next size up in calibers, and should. Now, admittedly, at some point too much Power (.577?) will compromise Speed and Accuracy, but for me, my .450 Dakota load is not even close to that. I don't load it to its 500gr/2550fps capability for that reason.
Thanks for the question; I hope I've clearly explained myself. Good day.