Magnetometer measurements from CAN conflict with other measurements from CAN in OpenIMU 300RI

I am trying to estimate orientation using the CAN readings from OpenIMU 300RI.

On CAN there are three frame IDs being streamed, one for each sensor.
Each CAN frame contains the measurements for each axis, though the interpretation seems to be inconsistent and different from the default coordinate frame shown in the user manual.

I am using imu_filter_madgwick https://github.com/CCNYRoboticsLab/imu_tools/tree/galactic/imu_filter_madgwick/ with the decoded measurements from CAN.

When including magnetometer readings in the orientation estimate, something very weird happens. If I apply a yaw rotation the estimate is reasonable, but with pitch/roll rotations, as soon as the IMU becomes still and the magnetometer reading becomes stable, the estimated orientation starts drifting, eventually stabilizing to a wrong orientation. This suggests a conflict between the magnetometer reading and the motion estimated by the gyro and accelerometer.

Can someone point me to how the exact way the readings streamed through CAN should be interpreted for a default 300RI setup?