That is an interesting problem. So when running on shore power the 50 amp 12vdc mini breaker feeding the coach batteries does not trip, but when running off the generator it trips. First question is are your coach batteries fully charged (12.6vdc wet cell; unloaded) ?
Second question is when you start the generator does the mini breaker trip at exactly the time the transfer switch operates (usually 30 seconds after gen starts the transfer switch disconnects shore ac and connects gen ac)?
Both ac sources should be supplying the same 120vac to the input of the converter, depending on the transfer switch (one source or the other, but not both). The transfer switch should also break the shore power connections first before making the gen connections last. If the coach batteries are in good condition and fully charged the charging current from the converter output should be only a few amps at 13.1 vdc approx. So how could the coach batteries pull more than 50 amps to trip the mini breaker when on generator ac? I would be tempted to inspect the transfer switch (no shore connected and no gen running) to make sure all connections are correct, clean and tight. Then with the gen running, “carefully” measure the voltage at an ac outlet to see what it is.....not sure if an under voltage on gen ac output would cause the converter output DC voltage to dip and raise the output charging current enough to trip the mini breaker.