My motorhome is very similar... 25' long E-350, but with the legacy V-10. I tow a 3,800# Jeep Wrangler TJ using a Blue Ox tow bar. I have an InvisiBrake setup.
I am right at the weight limit when fully loaded for travel... so anything heavier would bust my GVWR. I have towed the Jeep from Ohio to Colorado and back, and all over West Virginia mountains. After I got the InvisiBrake dialed in I can barely tell the Jeep is back there. There's no side to side wagging and no "pushing" that I can detect.
Are you CERTAIN the truck drivetrain is completely disengaged per the towing setup procedure in the manual? The only thing I can think of is some sort of drivetrain resistance, or the truck is wildly out of alignment. Have you done a "walk test"... someone walk alongside the towed while you pull it slowly for about 50 yards? Swerving serpentine will also verify the front wheels are turning/tracking (verify steering wheel isn't locked). It will also help the tow bar arms to lock into place. That is part of our towed hitch routine.