In this paper, we present I’m-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting, using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU).