[Question] The correct way of integrating external frame tracking to an ActionTerm

[ozhanozen]

Hi Isaac Lab team,

I would like to ask how to properly integrate the pose tracking of a frame—external to the articulation asset—within an ActionTerm implementation.

Here is the context:
I am currently working on PR #913, which includes the ActionTerm implementation for the operational space controller. Sometimes it is beneficial to apply the control on a coordinate frame that is attached to the target object we are interacting with (e.g., we want to use force control along the rotation axis of a screw while controlling the position in the other axes). This requires tracking the pose of this coordinate frame (or an object), which can potentially move. While this is relatively straightforward for the controller implementation itself (and direct workflows), it is unclear how to do it for the ActionTerm (and manager-based workflows). I have checked the existing ActionTerm implementations, and they mostly use only the internal information of the articulation asset. I didn't see any example of injecting information from other assets (e.g., the pose of a rigid object).

For the moment, I did the following as a workaround:

One provides the path of a RigidObject, whose pose should correspond to this external coordinate frame, within the ActionTermCfg. Internally, ActionTerm creates a FrameTransformer, whose task frame argument is this RigidObject. This FrameTransformer is used to track the world pose of this frame and its pose w.r.t. the robot root frame is calculated and used by the controller. While this works, there are a few caveats:

• There has to be a RigidObject to track. One cannot simply provide an arbitrary tensor free from an asset, which I guess is a limitation of the manager-based workflows.
• FrameTransformer requires another RigidObject as a source frame to be provided, which is actually not needed for the controller. I arbitrarily assign the first RigidObject child of the articulation asset but not use this information later.
• ActionTerm needs to handle the FrameTransformer operations manually.

Is this a proper way of integrating external frame information to the ActionTerm or would you suggest something else? I am asking the question outside of the PR, as it might also be relevant for other future ActionTerm implementations.

I appreciate your feedback.

[mpgussert]

This is a fantastic question! Thank you for asking it!

I believe this is the intended way to use the frame transformer sensor (so far as I understand it). Concerning your points

1. Yes, I believe this to be the case. The frame transformer was made to help with tracking "pick and placeable" objects and their targets, so the sensor hijacks the rigid body physics view pipeline in order to get the relative transformations between objects. I think this can be mitigated with an "Xform" asset equivalent, giving us access to the xform view, and thus vectorized transforms, but we need to figure out how best to do that

2. If the changes i suggest above go through, this would require any object being tracked by the sensor to have an Xform prim. this means it would have a path in the stage. Is this acceptable? We would be able to support arbitrary xform tensors at construction, but the path would exist in the context tree. I don't think that should be a problem.

3. I'm not sure what you mean here. I need to check your PR

Thanks again!

[ozhanozen]

@mpgussert Thanks a lot for your answer and clarifications!

1. Yes, being able to track Xform assets would provide huge flexibility.

2. My main concern with this point is that the FrameTransformer requires two assets —source and target— while I am only interested in tracking target_w so that I can explicitly calculate target_b (I already have the root_w). Choosing the source as anything works; it does not matter, therefore, choosing the source frame looks redundant. Nevertheless, to be able to track Xform assets would also be acceptable here, as I can then choose the source asset of the FrameTransformerto be the articulation root Xform. Then, target_b would correspond to target_source and be tracked by the FrameTransformer without any explicit calculations from my side. This would be nicer.

3. The typical usage of the sensors I saw consists of adding configuration files (e.g., FrameTransformerCfg) to the derived classes of InteractiveSceneCfg. This way, the scene registers the FrameTransformer entity and performs its common operations automatically, such as initializing FrameTransformer, and resetting/writing/updating its buffers during the simulation flow. In my case, the FrameTransformerCfg is not directly within the InteractiveSceneCfg (because it is a member of the OperationalSpaceControllerAction, this controller is the one under InteractiveSceneCfg). Hence, these common operations are not handled by the scene for the FrameTransformer. I need to do them manually instead, within the member functions of the OperationalSpaceControllerAction. Actually, I also have a ContactSensor within the OperationalSpaceControllerAction, and the same issue applies to this sensor. I wonder if there is a way to recursively register entities to the scene, such as registering OperationalSpaceControllerAction also registers its member sensors, and these sensors are managed by the scene as well. Do you think this is possible?