-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to perform depth integration with the predicted deformation #2
Comments
Hi @BaldrLector, thanks for your interest in the project! Regarding 1., unfortunately we didn't have the time to properly clean up the part of the code for depth integration. But it's basically an implementation of DynamicFusion. As to how to get the other needed data, you can check out the Data section in the readme. You should first download DeepDeform by following the instructions in that repo. Then use the link we provide in the readme to get the additional data. You should then merge both datasets, such that within a given sequence folder (see Data Organization of DeepDeform) you have all the data from DeepDeform plus the graph nodes, edges, etc. Hopefully this clarifies it a bit more. Thanks for pointing it out. |
|
Oh, got you. We are planning to clean up the code for generating this part of the dataset. It wasn't a priority, since we also released the corresponding graph data for DeepDeform, but we'll keep it mind now that you mentioned it. Essentially you need to sample nodes in the source depth map. Then compute your graph edges using geodesic connectivity and find your anchor nodes for each pixel, as well as their weights. We'll try to release the scripts for this relatively soon. Thanks for letting us know. |
Got it, thanks for your responsive reply! |
@pablorpalafox
Now I finished the 1) and 2) steps, I face a problem at step 3) How to rotate and translate each voxel center with predicted deformation? Although the resp provide the "warp_deform_3d" function, which utilizes the pixel_anchors pixel_weights, node_positions, node_rotations and node_translations. Now we have node_positions, node_rotations, and node_translations. How I compute the pixel_anchors pixel_weights for each voxel? |
@pablorpalafox, I'm also interested in the DynamicFusion portion of the code, whenever you can get to it. I'll add a +1 here so others can simply upvote in the future. |
Hi @BaldrLector and @Algomorph! We're now working on a coming deadline, but we'll definitely let you guys know if we eventually upload the reimplementation of DynamicFusion. @BaldrLector, your pipeline makes sense, the problem is that the |
@pablorpalafox thanks for your reply, I will keep work on the reconstruction part, and I am still looking for your new info eventually. By the way, have a nice year :) |
Hi, @pablorpalafox @AljazBozic, I am confused about the integration of the various keyframes(every 50 frames in the sequence). Previous methods take an initial frame (generally it is the frame at time step-1). Then we keep on updating the canonical pose using only the RGBD image at the next timestep.
@BaldrLector if you can provide some insight that would be very helpful. Thank you everyone. |
@shubhMaheshwari and @BaldrLector , it seems like all three of us are working on the same thing: trying to integrate the neural tracking code from the current repository into a complete DynamicFusion pipeline. Would either of you like to pool our efforts together? What I have, at time of this post:
What is still missing:
If you'd like to collaborate, please, reply so here. I'll email you at your respective email in your GitHub profile, and we can continue the discussion. |
@Algomorph definitely that's a great idea. |
Hi, @shubhMaheshwari @Algomorph until now I finished:
I almost finished the reconstruction part, but there are a bug to be fixed:
I'd like the idea of collaboration, please contact me. : ) |
Hey @pablopalafox and @AljazBozic,
|
@pablopalafox and @AljazBozic, do you guys regenerate the graph at every frame or is the graph persistent throughout the sequence processing (with, potentially, topological updates such as in DynamicFusion that @shubhMaheshwari mentions above)? Also, what voxel size and TSDF truncation distance do you use? |
Hi @Algomorph, The deformation graph is persistent throughout the sequence, the only changes are addition of new nodes, or marking existing nodes inactive (basically removing them). But the canonical positions of nodes are not updated, because when we tried re-generating graph every frame, that often lead to tracking drift. The graph edges are updated at every frame though, using the connectivity of the mesh. For depth integration we used voxel size of 0.5 cm and truncation of 2.5 cm. Let me know if you have more questions :) |
Thanks, @AljazBozic! I believe @shubhMaheshwari also has some questions above that are also relevant to the rest of us. a. [frame 0] and [frame t], or It is my understanding that DynamicFusion-style algorithms (including Fusion4D) follow schema (b) above, whereas there are some others (SobolevFusion/KillingFusion) that follow schema (a). Which schema do you use? |
Also, can you elaborate on marking existing nodes inactive (basically removing them) part? For example, I am able to register simple movements such as shirt (val/seq009) final_60e434286a2ebb0059382794_716143.mp4But with a complex example such as human(train/seq071) where the topology changes I am unable the graph update is generally wrong. Look at the hands in the below example. final.mp4This is happening with or without a keyframe update. My steps for adding graph nodes is similar to dynamic fusion and create_graph_using_depth.py.
@AljazBozic @pablopalafox thank you for your time. This is the only problem I am facing. Once I have fixed this I think I can have completed your non-rigid registration pipeline. Kindly help me in this step. I could also create a pull request or zip and email the code if that helps. |
Bump. |
I'm going to reply my own question here, almost a year later. The answer can be obtained from the NNRT article section 4.5 and appendix D.3. The correct answer is neither (a) nor (b). The flow network+differentiable optimization mechanism is trained on matching (very) sparse sequence frame pairs that are ~50 frames apart, so, naturally, the algorithm really only works well on these intervals. Hence, so-called "keyframes" are sampled from the sequence every 50 frames, and the tracking is performed between the last-encountered keyframe and the current frame. This requires a separate motion graph to be used for each keyframe, which is initialized with the cumulative motion graph with transformations applied to the nodes (translations added to node positions, rotation matrices reset to identity). In order to integrate the current frame into the canonical volume (or "reference" volume at frame 0), the motion from the latest "keyframe" motion graph is combined with the cumulative motion graph (translations added, rotation matrices multiplied). The system relies heavily on the accuracy of the correspondences, so there are two additional filtering systems that are not present in the code. I've implemented the bi-directional consistency to the best of my knowledge in my fork (https://github.com/Algomorph/NeuralTracking). My collaborators and I have a pretty decent idea about how to implement the multi-keyframe consistency as well. |
I'm going to answer all the questions by @shubhMaheshwari in order with my best guess ATM, for posterity's sake. I'm hoping @AljazBozic will at least correct me if I get something wrong here.
|
Hi, @pablorpalafox @AljazBozic ,
thanks for your kindly sharing, I learn so much from this source code, and I love it!
I have some questions:
However, it seems the depth integration is not implemented at this repository.
Could you shed some light on how to perform depth integration, as the gif presented?
Is there any code for reference?
I plan to perform non-rigid reconstruction form RGBD images with this method. Now i can get rgb image and depth image, how should I prepare other needed data (the graph nodes, edges, graph_edges_weights, graph_clusters, pixel_anchors and pixel_weights ) ?
Thank you in advance, your reply will vary appreciated!
The text was updated successfully, but these errors were encountered: