Erik...........it is not a big deal. As goes without saying you have to have the appropriate licences and tags properly cancelled and affixed to parts of the carcass as per the local state regulations where you were hunting. If your licence from the state in question is not also an export permit, then you need to get an export permit prior to leaving the state.
As I am sure you know, local provincial regulations can differ a bit from province to province and so when you cross the border back into Canada your game has to meet CFIA standards as well as the province you are crossing into and subsequent provinces you travel through on the way home.
Bottom line is.............bone the meat and avoid bringing any bone back with you, especially the spinal column. Hides or capes should be free of tissue and salted or frozen. Antlers and horns/skull plates should be completely free of any meat, blood, brain tissue, etc. I recommend boiling out the skull or skull plate and dipping in a 5% bleach or chlorine solution. This should meet any regs coming back from the US and in fact is not dissimilar to current requirements between most provinces and what the USDA wants from hunters heading home to the US from Canada. Primarily this is all due to the concerns about bovine TB and CWD.
If you need more specific info please drop me an e-mail.
Generally you just get waived through because a lot of the CBSA have no clue and if you have your tag and licence that is usually all they ask about.