I am going to go out on a limb and guess it's the AGM batteries that are the issue.
Typically when using AGM batteries, you must have an AGM capable charger as they cannot tolerate the higher overcharge current that standard lead acid batteries can.
Often this is not an issue with the engine battery as there is always a load (engine) when the alternator is charging the battery. But in stand-by situations, where the charger is charging the battery in trickle charge into an overcharge condition, a non-AGM specific charger can damage your expensive batteries.
You are likely to encounter this situation when using AGM for the house battery.
Also, AGM produces slightly different voltages which can also trick the charger.
I would recommend replacing your converter/charger to one that can handle AGM.
If your alternator is setup to charge the house batteries, you would also need to replace the charger on your alternator to an AGM capable charger. You can buy marine engine alternators that do this, but I am not sure if conventional vehicle chargers have this option or not.
And then you would have to also change the vehicle's battery to AGM too, as all batteries on a string need to be the same type for proper charging.
I have never switched to AGM simply because of the extra expense of making sure the chargers all work with AGM, and having to swap out both engine and house batteries.
The only thing that works on a RV is the owner...