I use both options rifle depending. All of my rifles get broken in with factory ammo. I usually try 3 or 4 different factory loads and can find something it either loves or a bullet that is workable, some guns prove they are a hand loading endeavor, either way this also gives brass to begin the process. I believe your theory is sound that higher end custom rifles will use hand loaded ammunition more commonly due to several reasons and it isn't necessarily that they are high end or custom. I believe the majority of custom rifles are built in wildcat, customized factory cartridges, or no longer commercially produced cartridges, meaning they are a hand load only proposition from the beginning. The alternative to this is guys who are meticulously and build exactly what they want. This scenario will likely shoot exceptionally with factory ammo but is hand loaded for regardless because the user enjoys it or can not leave well enough alone.
Both of my wifes rifles, a Weatherby Vanguard Camilla 6.5cm and Sako 85 Finnlite 270win, are factory loaded ammo. The 6.5cm is frustratingly accurate with 143 ELD-X ammo, consistent 3/4 moa groups from field positions, and I can not efficiently build a better load for it. Her 270 also shoots the ELD-X exceptionally but the 136 Terminal Accent is the go to for this, similar accuracy but far tougher and flatter shooting. I prefer hers to be factory options as well to reduce her stress while preparing for her girls hunting camp.
My 6.5cm, a Bergara B-14 Carbon, and 7mm prc, a Rugera American gen II, are both factory loads. The 6.5cm prefers the 130 Scirocco II loaded by federal, likely to become a hand loading rifle due to that ammos discontinuation through federal. The 7prc shoots the 175 ELD-X better than it has any right to do so.
I have a second custom 7prc built on a Sako action underway currently, I am hoping it won't be as picky as the 7mm rem mag it used to be! My 9.3x62 is a hand loaded cartridge due to the fact that it came with everything required to feed it for a lifetime when I purchased the rifle. My winter project is to find a load it truly likes. 30-30 is another hand load cartridge, not by necessity just for nostalgia. 300prc is hand load to use the 212 Bore Rider from Barnes. 6mm ARC is also hand loaded for bolt gun pressures rather then factory gas gun loads.