Events, how do they work?

[Pasarus]

Hi!

New to working with Trame, and it seems fantastic so far with a few caveats. A big issue we have run into with Trame is that it swallows events. We have 2 viewers that work as a normal python VTK application. One is for 2D slices and one is for 3D slices.

In both cases, we have an issue where many events never end up triggering either an observer added to the vtkInteractorStyle object or when adding KeyPress as an event to the vtkRemoteView with interactor_events as a kwarg.

Example for observers (this just does nothing on trame but in a normal application it works):

    style.AddObserver("MouseWheelForwardEvent", partial(onMouseWheel, reader, num_slice, voi, 1), priority)
    style.AddObserver("MouseWheelBackwardEvent", partial(onMouseWheel, reader, num_slice, voi, -1), priority)

Example for interactor_events using the tutorial as guidance:

    html_view = vtk.VtkRemoteView(renWin, interactor_events=("events", ['KeyPress']), KeyPress=onKeyPress)

This gives the error in the command line:

Traceback (most recent call last):
  File "/home/username/scd/CILViewer/Wrappers/Python/trame/app.py", line 254, in <module>
    start_viewer_edo_pipeline()
  File "/home/username/scd/CILViewer/Wrappers/Python/trame/app.py", line 243, in start_viewer_edo_pipeline
    layout.start()
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/layouts/core.py", line 125, in start
    self._init_app(_app)
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/layouts/core.py", line 86, in _init_app
    _app.layout = self.html
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/layouts/core.py", line 55, in html
    return self.root.html
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/html/__init__.py", line 372, in html
    out_buffer.append(child.html)
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/html/__init__.py", line 372, in html
    out_buffer.append(child.html)
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/html/__init__.py", line 362, in html
    self.events(*self._event_names)
  File "/home/username/mambaforge/envs/trame/lib/python3.9/site-packages/trame/html/__init__.py", line 271, in events
    if name in self._py_attr:
TypeError: unhashable type: 'list'

Then when taking it out of the list:

    html_view = vtk.VtkRemoteView(renWin, interactor_events=("events", ['KeyPress']), KeyPress=onKeyPress)

I get the following errors in the terminal, and nothing occurs when I do the events:

 > JS error | TypeError: this.interactorEvents.forEach is not a function
 > JS error | TypeError: this.interactorBoxSelection is undefined

I have a couple of questions with regards to this:

1. What am I doing wrong? or is this just not supported?
2. What is the best way to handle input events, be it KeyPress or MouseWheelForwardEvent etc?
3. What events can be handled? It doesn't seem that there actually support for the list provided in the tutorial (https://kitware.github.io/trame/docs/module-vtk.html#How-to-use-it) here (https://github.com/Kitware/vtk-js/blob/b92ad5463150b88514fcb5020c1fa6c7fcfe2a4f/Sources/Rendering/Core/RenderWindowInteractor/index.js#L23-L60)

[jourdain]

Hi @Pasarus,

First, thanks for trying trame with VTK. Let me try to answer the various questions you have.

1. VTK observer on interactor style

So the issue here is the way we are mapping "web events" to "vtk events". Basically we are using the original implementation of VTK/ParaView Web which only forward Mouse[Press/Move/Up] and fake the wheel into mouse move. Updating that mapping would be great assuming we can better represent web-event to vtk-event with as little conversion as possible. And by doing so, the interactor style will then became aware of them and you will be able to observe them. I've create an issue to track that as it will indeed be great to support such use case.

Doing so should represent a couple days of creating a new server side protocol along with the matching code on the JS. If you feel like contributing I can try to guide you on how to do it. Thinking more about it, it may require some C++ fix in VTK as well but nothing really bad or difficult.

2. Binding method directly from JS

I think the second html_view = vtk.VtkRemoteView... is missing a change as it look the same as the first one.

And looking at the code, it should work but I guess I will have to try it myself to see what might be going wrong.

Also I would think you would need to do the following to get the info you need within your method.

html_view = vtk.VtkRemoteView(
 	renWin, 
 	interactor_events=("events", ['KeyPress']), 
 	KeyPress=(onKeyPress, "[$event.keyCode]"),
)

Anyhow, I will need to test it myself and report back to either fix it or guide you on how to do it.

[Pasarus]

I think the second html_view = vtk.VtkRemoteView... is missing a change as it look the same as the first one.

And looking at the code, it should work but I guess I will have to try it myself to see what might be going wrong.

I am using the latest version 1.19.3 the issue is in the trame/html/__init__.py file on line 271, specifically I believe the input values for interactor_events are not properly handling lists, and only handling single input values as it takes the ['KeyPress'] as a list and try to search for it in the _py_attr of HTML which is a dictionary, and you can't have a list as a key in a dictionary. What probably needs to happen is an update to handling interactor_events with lists, by inserting the list if it's a list instead of just appending to the list.

Doing so should represent a couple days of creating a new server side protocol along with the matching code on the JS. If you feel like
contributing I can try to guide you on how to do it. Thinking more about it, it may require some C++ fix in VTK as well but nothing really
bad or difficult.

I am not promising anything but if I knew roughly where to look I could have a discussion with my manager and see if we have the time for this.

Anyhow, I will need to test it myself and report back to either fix it or guide you on how to do it.

If you could that would be fantastic.

[jourdain]

So it won't be exactly what you were hoping for as I did not fixed the VTK/C++ side of things. But the trame part of all the VtkXxxView should work as expected.

I've updated that example to illustrate what you can get as events and related payload.

Also you will need to run pip install -U pywebvue trame

[jourdain]

Does the example help in handling your use case?

[Pasarus]

It does! Thought I would note with the v2 documentation/examples changes coming, the link is now dead! but I can find it in the repo regardless. I will either reply here or start a new discussion if I have further issues.

[jourdain]

Just fixed the link, thanks for pointing that out...

[dieterrosch]

For anyone reading this, the updated example link is

trame/examples/06_vtk/01_SimpleCone/RemoteRendering.py

Line 77 in fbaffe3

def on_event(*args, **kwargs):