Companion beyond 3.0

[dnmeid]

Compañeros,

a few ideas and thoughts about future Companion development. I think everyone of you has some ideas, visions, dreams how to develop Companion. Some of them are in the Github issues, some in the Github projects, some of them are buried behind the 90 days Slack paywall, some are in private conversations or still in your head. So forgive me if I'm not up to date what may have already been discussed. These are just my ideas and some of them are going back to the very early days of Companion.

Before I start with the technical stuff, let me say that I also see organisational challenges. We do not have a roadmap, we do not have an organisation, just some dudes who do whatever they like. On one hand I like the freedom here but on the other hand I feel a little more structure couldn't hurt. E.g. I don't think that we should have a super tight release schedule but we should have a clear workflow towards a release, with feature freeze and time for QA. We should have agreed standards for modules and check them and we should have a way to find out how to agree on something and how to make team decisions.
We have for example tons of feature requests, often a user requests a feature only he needs and even he needs it only for one very special purpose. These feature requests are often very specific and the proposed solution tackles only exact one problem. We don't discuss new features and in the lack of an overall strategy often features are implemented just because they are easy to implement.
This has led to a bunch of new checkboxes here and there, modals, new tabs and so on.
I'd vote for having a general vision how Companion should look like and work. Feature requests should always be checked if they fit in and if not they either are rejected or we find a way to come up with a fitting feature which is also good to solve the requester's problem.

So let me elaborate on how I could imagine Companion to look and work.
I think Companion has grown to a state where it looks blown, not very intuitive any more. We have to admit that after the years we have so many great new features that we just can't reach the simplicity of the first Companion any more. But at the moment every new feature makes Companion look even more convoluted. I think with the existing GUI we don't have much headroom.
My proposal is a three columns design.

• Left column: a structured tree of all the objects where a user can adjust something. E.g. general setup, surfaces, connections, variables, triggers…
• Middle column: a context sensitive adjustment pane
• Right column: a layout pane, i.e. what the Button tab is today. This should have the possibility to switch between different items to be placed somewhere.

It could look like this:

Control layout

Surface layout

Window layout

At the moment we have three menu like elements, we have the sidebar which annoyingly is always extended at page load and only adds very few essential links, we have the main tab selector and in some tabs we have sub-tabs changing only a part of the tab. The tree would unify all of them making it more intuitive to navigate to the wanted part. Additionally it would be quite easy to dynamically add branches and leaves to the tree. So it would be maybe one click more to get to some places but at least one click less to get to most of the places. Most of the users with large configurations will have to do some scrolling but I think it should still be an overall improvement. Additionally scrolling could be reduced by something like a customisable favourites section.
The adjustment pane always has the possible adjustments for the branch or leave, e.g. if the connections branch is selected you have the today's connections tab and can add connections. If you select a connection directly, you have the configuration of the connection.

On to surfaces. We've been talking about this for long time and this is quite a challenging topic but I think we agree that 1. we want to open Companion for even more different surfaces, 2. the current implementation of everything which is not a Streamdeck XL is not optimal and everything which is not a Streamdeck Classic or Mini or Pedal is even worse, 3. we want and have to improve in this area.
Let me break it down in different aspects.
The surfaces themselves should be defined by plugins, so we have an abstraction layer between the core and the real device. This would make it much easier to adopt for new surfaces. For this we need a description language of all possible types of surfaces, where we can describe input elements like buttons, faders, joysticks, other proportional inputs, gyros, touch elements and so on, output elements like LEDs, LED-strips, displays, speakers, and so on. I want to emphasise the distinction between input and output elements. At the end the description should have all the information for being able to draw an emulator. It doesn't have to be photorealistic, only functional identical and also should offset some controls to better fit in the regular raster.
Communication with the surface is handled by the plugin, so it would be easier to integrate devices with their own drivers, midi and serial stuff should be possible. Different webbuttons would be just surface modules. Surface modules? At the moment a companion module for a connection is able to talk to an external device and to transport data from Companion to the device and transport data from the device to Companion. There are so much similarities that it should be possible to make the normal modules able to work for surfaces. At the same time the normal connection could also benefit from it, maybe it is possible to have an emulator for the device defined by the module. And finally what are the big differences between a surface and an API? It is just the method of how to assign an incoming command to an executable element. So why not have the APIs in modules too. More modularity is more security, more performance and easier maintenance.

How to deal with that surfaces in Companion. I think while the concept of the pages shared by all surfaces, like we have it, is quite easy, once you got the point, we nonetheless have to replace it. This is going to be far more advanced and flexible for the power user but should be similar as easy for the novice user. First of all I suggest that we stick with a grid, let's call it the canvas. Every cell in the canvas has x and y coordinates and the canvas' size is only limited by the js number range.
Now we can place surfaces on the canvas completely to our liking, but a surface always snaps to a grid cell. The point is that we should have matches between I/O elements of our surface and the grid cells. That doesn't mean that a control has to be exactly a 1x1 cell, it just has to be at least 1x1.
Surfaces can also overlap partly or completely but don't have to. So now we can emulate what we are doing today: Button 1 is button 1 on all surfaces.
Surface rotation is tbd, maybe one individual module per orientation, maybe an option.

How can the actions and feedbacks be placed on a control? Easy would be to just assign everything to a cell, all output-controls from all surfaces positioned at that cell get the output data and all input-controls feed the action. Only difference compared to today would be that we don't number the buttons page/number, but page/x/y.
But I think that is too easy because now our page is huge and we are always switching the whole canvas. Actually many users have built complex menu structures where they shuffle around buttons, all with absolute addressing. A nightmare to program and to maintain but the pages won't allow for more. More and more complex surfaces will rise the already existing demand for a more flexible system.
The solution could be windows just like computer desktop windows. A window would have a x/y position which always snaps to a canvas cell, the window size also snaps to the grid. That means the smallest window is 1x1.
A window has a content which is a grid like the canvas, that means the window's content can be larger than the shown part of the window. The visible part of the window can be scrolled to show any part of the window's canvas.
A window cell is used like we use a button today, you assign actions and feedbacks to a window cell. A window cell can be referenced by a three part number window/x/y.
There should be internal actions to scroll the visible part of the window to fixed positions or e.g. by the window height or window width. That exactly can resemble todays pages, page up/down just scrolls a window.
A user can create as many windows as he likes. Windows live in a window bin and they don't have to be open all the time. A user can open, close, reopen a window or drag it to a different location at the canvas. Windows also can overlap or cover each other, windows have a z-order and there are actions to move a window to top or bottom.
Unlike the surfaces which are "transparent", i.e. control data will be passed to all surfaces under the cell, windows really cover each other, so only the logic of the topmost window cell will interact with the surfaces below. But covered or not shown window cells can be used by APIs all the time.
You see how this can improve flexibility? E.g. if I create a 2x1 window and place it on top with two buttons, I created sticky/fixed buttons which always stay there even if a window below scrolls. If I create a 1x5 window and use the buttons to scroll a different window to defined positions, I have created a tab controller.
Two more aspects of windows: 1. Today we have a shared set of pages and each surface can be at a different page. I think this should be an option with the windows. That means by default all surfaces under the window show the same part of the window's canvas. If you change the position on one surface, it changes for all surfaces. Optional you can have individual positions per surface. Why? The most common use of Companion is a single operator use. The question is only relevant if you have more than one surface, so let's think of single operator with multiple surfaces. They more often than not want to synchronise their surfaces and this not only would allow it, but is also the expected behaviour if you never had worked with Companion before. The legacy workflow would be to have all surfaces placed on top of each other and opt for individual positions with a single window. The new workflow would allow to place surfaces side by side and span a large window across multiple surfaces or to have individual windows for each surface or whatever you can imagine.
2. When working with multiple surfaces at different positions and with different sizes you may find yourself programming the same buttons in different windows because you want to have them on different surfaces but the surface's layouts are too different. That's why all windows should share the same canvas, just scrolled to an individual position. So actually the user is programming all of his buttons or controls on one huge canvas and then has the possibility to take parts of the canvas and arrange them like crazy on the surfaces. Since we are loosing page names and user programming may spread over a large area, named bookmarks for positions should be available. They also would make it possible to scroll to a bookmark position and thus making it easier to move stuff around if you find out later that you need more space somewhere.

I suggest to separate the buttons from their actions and unify this with the triggers. What do I mean with this? A button has some steps with actions, a trigger has some actions. A button's actions are triggered by a press or release, trigger actions are triggered by a self made condition. What if we tell the trigger to run on a specific button press? Or the other way round if the only action of a button is to run a trigger. Let's take the steps and stuff from the buttons and move it to a new element, let's say "stepped script". The actual trigger and condition part of the trigger is part of the element stepped script, because it defines when to run or advance the stepped script. A button can also trigger or advance the stepped script, so we actually are generating a new state (at which step are we now) and many buttons can share that script. In the long term there can also be other types of scripts, e.g. user defined javascripts. They would have the trigger part to decide when the script is run and then they could access variables and run actions and so on but all from the javascript.

So far I've been silently ignoring surface heterogeneity. I said a windows cell interacts with all controls below it. How can this be done? I think the solution we have with the rotary controls for SD+ was good for making rotaries available in a very short time, but this can't be the way to have more and more checkboxes and special actions for different controls. First we need to decide when an action is triggered which is a "bang" but not all controls generate a bang. Think of the X-Keys T-Bar. Today's workflow is to have a trigger watch for variable change and then trigger an action which uses the variable in an option.
If every surface exposes it's control's values like internal does for X-Keys, all these variables could be used to trigger stuff like scripts with ultimate flexibility and very fine grained control. But at the cost of ease of use. I think for 98% of todays Companion programming it is good, if we just use all the bangs of controls below to trigger zero to x scripts. So if you don't want to use something more complicated to trigger a script, you can decide to add more events to the script trigger itself or omit triggering it from a button and do all the logic in the trigger of the script.
Next we need to deal with values which are not a bang. How should they be transported? Exposing all this to variables is good, but would require much programming at the user side and is likely to break if a button is moved and also it would only help a little if there are different controls triggering a button. I think it would be good to transport normalised and original non-bang values with the event. We actually need this today for the rotary which not only gives a bang (turning speed is not 0), but also a directional turning speed. There needs to be some standardised and normalised format and the possibility to also transport raw data. The rest is up to the action. Not every action has to be 100% compatible with all events.
At the moment rotaries trigger different special actions, with this proposal it would be only one action with only one event and the direction is in the event data. For the future and also for many other analog and even fancier controls I think this will simplify programming for the user dramatically because the action can decide what to do with the extra values. E.g. a set fader action can use a proportional value from the event to set the the value at the connection. A joystick can send its x/y position and the action decides what to do with it. API calls to Buttons, e.g. OSC, can pass additional data. According to that logic actually an ordinary button would have to send a value too, wether it is pressed or released. That means press and release would trigger the same action and the state of the button would be in the event data.
Today all actions are just triggered and have only very little knowledge of why they are triggered. The decision between different usages like press, release, turn left, turn right is made upstream in the button. I think it is not practical to have even more possibilities at button level in the future for other input elements and map them all to just a trigger. With the proposal the decision of how to react on a value change of an input would be possible in the action.
On the other hand we have to provide a way to not break the existing release action workflow or the existing rotaries workflow and generally it would be good if users had the ability to be more creative with extra event data.
This can be done with internal helper actions, which can manipulate the event data or change the actions execution in a conditional way.
Also it would be possible to use this event data in the trigger expression of the script. Like I said earlier the actions wouldn't be part of the button any more, they would live in a script which is triggered from the button (actually the controls below the button) .
It would be easy to extend the script's condition to only run the script if the event.button.state === 1 or if it is 0.
Or within one script we could have an action which aborts the script if there is no event.tbar.value, or whatever you can think of.
That way we can make all the existing actions compatible with additional event data by optionally externalising the branching decision and todays release actions or rotary actions would be scripts with a special condition based on event data.

This being the input side, there's also an output side, like LEDs and displays.
Just like inputs in my dream outputs should be reusable and a thing independent from the button. One or many buttons can use a set of output elements, maybe also use a combination of more than one output set. That means what today the combination of the button styling and feedback is, would be an output set. Each output set can contain feedbacks and the like.
It is no secret that our current graphics generation is outdated and actually never has been very sophisticated. I plan to start a large overhaul soon, which not necessarily has to end in what I describe here but would be a foundation for it.
I would introduce a user defined amount of layers to the graphics. Today we actually have layers with the background color in the back, then a user image or images from feedbacks, finally the text on top. Users should be able to use more layers and define the content of a layer by choosing among different elements like text, image, rectangle, triangle, value formatter, gauge. Layers have some sort of id and feedbacks can reference the id, that means a feedback could e.g. change the color of the rectangle "background" or of the rectangle "left indicator" or of the text "top textbox" and so on.
Additionally it would be good to have a way of changing the look of the output completely without having to change many things in many layers. I was thinking of different states earlier but now I think it would be even better to have layer groups which can be activated and deactivated. Group activation for sure is handled best with an expression. (another benefit is, that unlike with states all the manipulation targets are there all the time)
I think now it should be left up to the surface module to render or show the information. A small display will render at a small skia-canvas, a larger display at a larger skia-canvas, a RGB-LED will show the color of the lowest rectangle, a on/off LED may light if brightness is greater than 50%, a LCD-text-display maybe show the first text layer.
Even if not every possible layer combination is not 100% compatible to every possible surface output element, I think this is quite advanced and with the surfaces being able to be at different positions I don't expect too many fancy combinations of output capabilities.

To wrap it up, in this proposal buttons would actually loose most of their own functionality. The functionality would be provided by other, from the button independent elements like scripts or output sets. The button just uses those elements and they are not exclusive to a button. For convenience it could be possible to have inline creation and editing possibilities, then programming buttons would feel much the same like today. You just start with an empty button, add an existing or new output set, the options of the output set will be shown in the button editor. Same with scripts. And of course button wouldn't be the best name for this control.

I've been thinking about this concept for quite a while now and I think it is an overall solution which is consistent, easy to understand and works. I've tried to omit details as much as possible and only give the big picture.
So, although this may seem all together like a huge effort and many big changes, I think it is quite doable in smaller chunks and with coordinated collaboration. The overall goal should be to make Companion a tool which can grow further and can offer more functionality by replacing the design by feature approach with a flexible and expandable workflow and gui design.
Now it's your turn. What do you think about it? Should we take this route or are there other proposals? Anything you really dislike or really like? Any comments are welcome but please don't get lost in details, this is about the big picture.

[Julusian]

I think there are a lot of ideas here, that want discussing separately, so I'm going to start new comment threads for each area I have comments on. (these will trickle in as I find time time to think about them today)

At the moment we have three menu like elements, we have the sidebar which annoyingly is always extended at page load and only adds very few essential links,

Yeah, the left sidebar is something I rarely interact with, so I think there is a lot of scope to move more things over to there, but I wonder if it will decrease discoverability with some things ending up too hidden? I don't know, something to be careful of.

I do wonder if a tree view like that could end up being a bit overwhelming to new users, who aren't really sure where to start.

[dnmeid]

My idea is to get rid of the sidebar at all. The tree is just one always visible part of the UI.
For new users with a complete clean installation, the layout pane would be in "windows". Found surfaces would automatically be added to pos 0/0 and one window would be allready there. So they just had to select a cell and are immediately able to work with it in the settings pane. Well, you have to find where to add a connection, but that is the same today. Maybe an interactive GUI tour at first run would be an idea.

[Julusian]

On to surfaces. We've been talking about this for long time and this is quite a challenging topic but I think we agree that 1. we want to open Companion for even more different surfaces, 2. the current implementation of everything which is not a Streamdeck XL is not optimal and everything which is not a Streamdeck Classic or Mini or Pedal is even worse, 3. we want and have to improve in this area.
Let me break it down in different aspects.
The surfaces themselves should be defined by plugins, so we have an abstraction layer between the core and the real device. This would make it much easier to adopt for new surfaces. For this we need a description language of all possible types of surfaces, where we can describe input elements like buttons, faders, joysticks, other proportional inputs, gyros, touch elements and so on, output elements like LEDs, LED-strips, displays, speakers, and so on. I want to emphasise the distinction between input and output elements. At the end the description should have all the information for being able to draw an emulator. It doesn't have to be photorealistic, only functional identical and also should offset some controls to better fit in the regular raster.

I 100% agree that surfaces should be done via some plugin system. Whether that is the same plugin system used for modules, or a separate one is open for discussion. I haven't put much thought into this yet, as without being able to represent surfaces properly in Companion I am not keen on opening the gates to support many more types.
I have spoken to xkeys about some open way of defining device layouts and functionality (intended in part to be used to power https://gitlab.com/chris.willing/xkeys-server), but I am not sure on the status of this. But yes, some way of defining these layouts in an open format to be interpreted by our ui (probably with a json schema to validate it), is the key part to make this happen. That could start of fairly minimal for now on types of control, if we can design it in a backwards compatible way for adding new types.

Communication with the surface is handled by the plugin, so it would be easier to integrate devices with their own drivers, midi and serial stuff should be possible.

The bit that concerns me about plugins is making sure that the various native libraries they each need will behave. But seeing as this is working today for native libraries in modules today, this may be a solved problem.

[Julusian]

This being the input side, there's also an output side, like LEDs and displays.
Just like inputs in my dream outputs should be reusable and a thing independent from the button. One or many buttons can use a set of output elements, maybe also use a combination of more than one output set. That means what today the combination of the button styling and feedback is, would be an output set. Each output set can contain feedbacks and the like.
It is no secret that our current graphics generation is outdated and actually never has been very sophisticated. I plan to start a large overhaul soon, which not necessarily has to end in what I describe here but would be a foundation for it.
I would introduce a user defined amount of layers to the graphics. Today we actually have layers with the background color in the back, then a user image or images from feedbacks, finally the text on top. Users should be able to use more layers and define the content of a layer by choosing among different elements like text, image, rectangle, triangle, value formatter, gauge. Layers have some sort of id and feedbacks can reference the id, that means a feedback could e.g. change the color of the rectangle "background" or of the rectangle "left indicator" or of the text "top textbox" and so on.
Additionally it would be good to have a way of changing the look of the output completely without having to change many things in many layers. I was thinking of different states earlier but now I think it would be even better to have layer groups which can be activated and deactivated. Group activation for sure is handled best with an expression. (another benefit is, that unlike with states all the manipulation targets are there all the time)
I think now it should be left up to the surface module to render or show the information. A small display will render at a small skia-canvas, a larger display at a larger skia-canvas, a RGB-LED will show the color of the lowest rectangle, a on/off LED may light if brightness is greater than 50%, a LCD-text-display maybe show the first text layer.
Even if not every possible layer combination is not 100% compatible to every possible surface output element, I think this is quite advanced and with the surfaces being able to be at different positions I don't expect too many fancy combinations of output capabilities.

This is an area I havent thought about much myself, so I dont have much insight.
I do think we need an overhaul of the drawing, and I like the concept of layers.
I am already wanting to work on 'feedback logic' in 3.1 (special 'internal' boolean feedbacks which have other boolean feedbacks as children), I dont think that conflicts with anything here and is simply intended as an alternative to expressions that can use boolean feedbacks instead of just variables.

I think now it should be left up to the surface module to render or show the information.

I think we need to be careful here. While this will give the surface modules large amounts of flexibility in how they draw, it will also make it immensely hard to add/change anything about the drawing. Instead of changing a couple of patches of code that we control, adding a new 'font-flip' property will mean updating 10s of surface integrations.
For this reason I am leaning towards we should keep the drawing code inside of companion, with surface plugins simply able to say 'draw this at 72x96px. Or to as you suggest get a subset of properties, such as a single colour or line of text to display. (like we can already do today in the satellite api)

[dnmeid]

Instead of changing a couple of patches of code that we control, adding a new 'font-flip' property will mean updating 10s of surface integrations

Yes. If there is a new property or something and we manage the presentation on the sufaces at a central point, we have to patch the presentation there for all the surfaces. So the work is the same, it is just distributed to the modules.

[Julusian]

So the work is the same, it is just distributed to the modules.

Before we go that way, I would like to see us try to have a builtin rendering system. I am fine if some 'unique' devices need something different and do their own thing (eg loupedeck-ct wheel), but I feel like 90% of them will want the exact same just at different resolutions, and doing something distributed sounds like a maintenance burden both for us and anyone else who wants to make their own unofficial surface plugin

[dnmeid]

Agreed, I'm always talking here of a possible mid/far future. I'm already working at the builtin rendering. This may or may not be transerred to modules later. Or maybe we provide this as a function to a module. I think there always has to be a road from now to then and maybe we will end up with something we do not foresee today.

[Julusian]

So far I've been silently ignoring surface heterogeneity. I said a windows cell interacts with all controls below it. How can this be done? I think the solution we have with the rotary controls for SD+ was good for making rotaries available in a very short time, but this can't be the way to have more and more checkboxes and special actions for different controls. First we need to decide when an action is triggered which is a "bang" but not all controls generate a bang. Think of the X-Keys T-Bar. Today's workflow is to have a trigger watch for variable change and then trigger an action which uses the variable in an option.
If every surface exposes it's control's values like internal does for X-Keys, all these variables could be used to trigger stuff like scripts with ultimate flexibility and very fine grained control. But at the cost of ease of use. I think for 98% of todays Companion programming it is good, if we just use all the bangs of controls below to trigger zero to x scripts. So if you don't want to use something more complicated to trigger a script, you can decide to add more events to the script trigger itself or omit triggering it from a button and do all the logic in the trigger of the script.
Next we need to deal with values which are not a bang. How should they be transported? Exposing all this to variables is good, but would require much programming at the user side and is likely to break if a button is moved and also it would only help a little if there are different controls triggering a button. I think it would be good to transport normalised and original non-bang values with the event. We actually need this today for the rotary which not only gives a bang (turning speed is not 0), but also a directional turning speed. There needs to be some standardised and normalised format and the possibility to also transport raw data. The rest is up to the action. Not every action has to be 100% compatible with all events.
At the moment rotaries trigger different special actions, with this proposal it would be only one action with only one event and the direction is in the event data. For the future and also for many other analog and even fancier controls I think this will simplify programming for the user dramatically because the action can decide what to do with the extra values. E.g. a set fader action can use a proportional value from the event to set the the value at the connection. A joystick can send its x/y position and the action decides what to do with it. API calls to Buttons, e.g. OSC, can pass additional data. According to that logic actually an ordinary button would have to send a value too, wether it is pressed or released. That means press and release would trigger the same action and the state of the button would be in the event data.
Today all actions are just triggered and have only very little knowledge of why they are triggered. The decision between different usages like press, release, turn left, turn right is made upstream in the button. I think it is not practical to have even more possibilities at button level in the future for other input elements and map them all to just a trigger. With the proposal the decision of how to react on a value change of an input would be possible in the action.
On the other hand we have to provide a way to not break the existing release action workflow or the existing rotaries workflow and generally it would be good if users had the ability to be more creative with extra event data.
This can be done with internal helper actions, which can manipulate the event data or change the actions execution in a conditional way.
Also it would be possible to use this event data in the trigger expression of the script. Like I said earlier the actions wouldn't be part of the button any more, they would live in a script which is triggered from the button (actually the controls below the button) .
It would be easy to extend the script's condition to only run the script if the event.button.state === 1 or if it is 0.
Or within one script we could have an action which aborts the script if there is no event.tbar.value, or whatever you can think of.
That way we can make all the existing actions compatible with additional event data by optionally externalising the branching decision and todays release actions or rotary actions would be scripts with a special condition based on event data.

Ignoring the window and cells portion, as I have that as another point.

I have a very different proposal, #1187.
With that, it would be possible to 'bind' a property to an encoder, optionally defining some min/max/step values.

While we could make it so that the modules are responsible for understanding what an encoder is and how to map that into things on the device they control, that will continue down the route of every module is slightly different. One module will let you choose how much single click of the encoder changes, others wont expose it and it will be fixed. In some, pressing that encoder will reset it to 0, others will mute/unmute.
When working with an analog value (fader/tbar) some might allow you to set different start & end points, or apply a formula to give a custom curve. Others will be a straight one to one mapping.
I am afraid that going down this route will lead to every module being different, and making complex setups impossible due to the differences. This is essentially the same problem we have today with modules not supporting variables in their values.

So in my proposal, an encoder would have the same press & release actions as today, but the rotate actions would be replaced with 'binding' a property. I am unsure if it should be possible to keep with rotate actions like today for compatibility.
A fader/tbar would translate to a way to bind to properties, ideally with the min/max/offset/formula ability to make it configurable, no need for any groups of actions.

[Julusian]

I think with my proposal, there may be more different implementations in the modules but I don't see this as a drawback.

I think this is a major drawback. Users will be subject to the decisions of the module author, and unless we define rules and the people including module updates check those rules are being followed correctly, it will be users who feel the pain with very little ability to work around it.

One of my main aims with this is to give the users a consistent experience, and to give them the power over things instead of the module author.
Another one of those aims is to minimise the amount of code module authors have to write, especially the amount of repetitive code either within or across modules. (Inspired by experience of doing audio control in multiple modules, defining fades is very repetive boilerplate https://github.com/bitfocus/companion-module-bmd-atem/blob/master/src/transitions.ts https://github.com/bitfocus/companion-module-behringer-x32/blob/master/src/transitions.ts)

Using that audio example again. Rotate changes level, and this module author made it so that pressing the encoder resets it to 0. Problem is that I have the level at around -25db. So 0 is way too loud, and will be incredibly startling to hear.
The options I see for the user are:

• hope none of the volunteers ever press the encoder
• pressure the module author to change that behaviour, and wait a couple of months for that to be released (if ever)

But if behaviour was composed as separate rotate and press actions/properties/whatever, then while the module author will provide presets which do the 'wrong' thing, the user is free to then edit them to behave how they want.

Or maybe I'm trying to do a really compact rgb color adjustment on a single encoder. with the press cycling through which property is controlled. I can't do that if the module author is the one who defines what pressing the encoder does. Unless they implement that as an action.

---

This is a similar kind of motivation I had for boolean feedbacks. They were also motivated by wanting to use them as event sources for triggers, but also that users are entirely at the mercy of whatever properties the module authors let them change in feedbacks. Boolean feedbacks gives module authors less to need to implement, and the users more power

[dnmeid]

One of my main aims with this is to give the users a consistent experience, and to give them the power over things instead of the module author.

Totally agree with that.

Users will be subject to the decisions of the module author

Maybe I've not been clear enough. Yes, I think the module author is the one who knows best how to communicate to a device, what actions and what options to provide and so on. So this is what I would want to be a the module level. A linear value can be interpreted in so many ways and although it may often be useful to bind a companion value to a device property, it often is not useful as well.
So I with this proposal I do not want to take any flexibility away from users and I do not want to combine stuff into one action where the user doesn't have a chance to change behaviour. In my idea there would e.g. be an action which sets audio to a level defined by an option, exactly like we have today. If you push a rotary, a press event is sent to the action and it sets the audio level. If you turn a rotary, a rotary event is sent to the action and now it can have a look if it can handle rotary values. A "set audio level" action should be able to do this and can now increase or decrease the value. Maybe it is useful for a "set audio level" to increase or decrease the value faster the faster you turn. Maybe it is useful to have a little snap point at 0dB. Maybe a "set x-position" of a media server may want to adjust speed adoption to the current value range (a range of 32.000 pixels should ramp up faster, than a range of 640 pixels). Maybe a rotary wants to scroll thru a list of options and set this, this is not a linear value any more and there shouldn't be a speed ramp. Maybe this list should be cycled infinite, maybe not. I think that there is so much behaviour specific to a device and the action that this can only be solved by the module author. The question is, should we take this flexibility away from the module author just because there also will be some bad decisions by module authors? I think this flexibility should be left in the module.

Or maybe I'm trying to do a really compact rgb color adjustment on a single encoder. with the press cycling through which property is controlled. I can't do that if the module author is the one who defines what pressing the encoder does. Unless they implement that as an action.

Well, my idea is not to have super-actions which combine super complex scenarios all in one. The push triggers one action, which can cycle, the rotation triggers other action which can do other things, all left to the programming of the user. The module designer can make a preset to combine those different actions. One action can be able to react approriate to different type of control data, but it also can only react to only one type of data, e.g. only push/release. If the user does not agree that the action should process this or that data, he can influence the event data and process it, e.g. strip it out, swap it...
Let's take exactly the rgb color adjust example. In my idea it could look like this: The button has one or more actions which makes a variable cycle 'r', 'g', 'b'. This could be running a different button with steps or three internal:set variable with expression". These actions react naturally to the press of the rotary and the user doesn't have to take care of it. Now I add an action like "set color intensity" where I can generate the destination of the parameter with an expression/variables parsing. This action now would usually react to the press and rotary. That means when I turn the rotary, the first actions don't react but my set-action changes the value. But unfortunately whenever I toggle through my colors, the set-action reacts too and a fixed value would be set. To change this, we have to make sure that at this bank the set-action doesn't get the press events. And this can either be done by splitting the actions out in different scripts which only run for different type of control data or by a helper action to strip out this data for the remaining actions or by adding properties to all actions. I wouldn't suggest the properties because it would be a visual element being there all the time for all users but only some users would need it.

Another one of those aims is to minimise the amount of code module authors have to write

Totally agree with that too. So with your proposal the core would generate actions, feedbacks... for such parameters and I like that idea. I would just handle different how the action is called. I still think that value binding is not flexible enough and catching every possibly wanted solution is not achievable at core level. So again, I see no interference of the ideas here. A module could define a linkable parameter. The core exposes an action, a variable and a feedback for it. In the example the generated "set audio level" action would react on all types of linear controls and can have additional parameters for e.g. the level when used from a push button. If the user doesn't like how the automatic generated action works, he can tweak it.
If the module author doesn't like how the automatic generated action works, he can build a different action by hand.

[Julusian]

If the user does not agree that the action should process this or that data, he can influence the event data and process it, e.g. strip it out, swap it...
To change this, we have to make sure that at this bank the set-action doesn't get the press events. And this can either be done by splitting the actions out in different scripts which only run for different type of control data or by a helper action to strip out this data for the remaining actions or by adding properties to all actions. I wouldn't suggest the properties because it would be a visual element being there all the time for all users but only some users would need it.

I think that is the bit I was missing, the ability for the user to decide and 'block' the press/release event from going to these actions.
I don't like this suggested workaround to strip out the events I want to block, for one thing it could prove rather hard to figure out what events an action might react to.

Maybe I've not been clear enough. Yes, I think the module author is the one who knows best how to communicate to a device, what actions and what options to provide and so on. So this is what I would want to be a the module level. A linear value can be interpreted in so many ways and although it may often be useful to bind a companion value to a device property, it often is not useful as well.
So I with this proposal I do not want to take any flexibility away from users and I do not want to combine stuff into one action where the user doesn't have a chance to change behaviour. In my idea there would e.g. be an action which sets audio to a level defined by an option, exactly like we have today. If you push a rotary, a press event is sent to the action and it sets the audio level. If you turn a rotary, a rotary event is sent to the action and now it can have a look if it can handle rotary values. A "set audio level" action should be able to do this and can now increase or decrease the value. Maybe it is useful for a "set audio level" to increase or decrease the value faster the faster you turn. Maybe it is useful to have a little snap point at 0dB. Maybe a "set x-position" of a media server may want to adjust speed adoption to the current value range (a range of 32.000 pixels should ramp up faster, than a range of 640 pixels). Maybe a rotary wants to scroll thru a list of options and set this, this is not a linear value any more and there shouldn't be a speed ramp. Maybe this list should be cycled infinite, maybe not. I think that there is so much behaviour specific to a device and the action that this can only be solved by the module author. The question is, should we take this flexibility away from the module author just because there also will be some bad decisions by module authors? I think this flexibility should be left in the module.

I think it is a balancing act of what should be left to the module and what to the user.
Almost of this to me sounds like it could be left to companion to do, with the benefit of it being implemented once.
Of course the module developers need some say in it, part of which will be normalising the values, and defining what is valid

And don't forget that every encoder is different, some have 10 steps in a full rotation, some 100, some 20. Some report each individual step (loupedeck), others combine and report larger deltas (streamdeck-plus). So companion will need to do some form of normalisation, and if modules arent careful in their ramp algorithms they could find they only work with some encoders.

In my idea there would e.g. be an action which sets audio to a level defined by an option, exactly like we have today. If you push a rotary, a press event is sent to the action and it sets the audio level. If you turn a rotary, a rotary event is sent to the action and now it can have a look if it can handle rotary values. A "set audio level" action should be able to do this and can now increase or decrease the value.

By how much each click of the encoder? When being used with rotation events, the action will need to know a delta/step. When used on a button (or the press of an encoder) an absolute value is needed (perhaps it should be able to do a builtin fade too, atem and x32 support this). When used on a fader, a min and max would be nice. I expect different things to happen depending on the type of control it is being used with, and each of those requires the action to know different details on how the user would like it to behave. Sure the module author has an opinion on what they recommend, but they wont always be what the user wants.
So as a module author, I will either be creating one action with many different input fields to cover all cases, or I would create different actions for each event to allow for prompting for only the relevant things. This is one of the main motivations for properties, expose a value once and let companion create all these actions.

Maybe it is useful to have a little snap point at 0dB.

This is the one that I don't have a suggestion for how to make generic.

Maybe a "set x-position" of a media server may want to adjust speed adoption to the current value range (a range of 32.000 pixels should ramp up faster, than a range of 640 pixels).

Sure, but as the property will be telling us the valid range, and a step value we should be able to do a ramp just as well as the module could?
And don't forget that the user will likely want a say in the speed of the ramp, or if there should even be a ramp.

Maybe a rotary wants to scroll thru a list of options and set this, this is not a linear value any more and there shouldn't be a speed ramp.

Unless a property can change from being a list of options into something with a ramp, I dont see this as a problem.

I still think that value binding is not flexible enough and catching every possibly wanted solution is not achievable at core level. So again, I see no interference of the ideas here.

The bit of your proposal I have a problem with is an action listening to different types of events. I have no problem with combining the rotate left and right into a single action, I have concerns about differing opinions on different types of events (rotate vs press vs swipe).
But at the same time, I question the benefit of allowing modules to define these smarter value change actions and feel like this could be abstracted and handled just as well by companion core. After all, they are just numbers, or a value in a list of options. Is there really something so special about them that cant be defined as properties in a blob of json? Maybe I just cant think of any myself.

Was binding the right word to use here? I am wondering if I am mixing my own terms up now..

[Julusian]

After discussing this further, our agreement was:

Actions should be able to declare what event types they can handle, and so when the action is able to listen to a rotary it will get the appropriate event as well as some metadata about the event. In this case the velocity/amount of the rotation.

We agree that the checkbox 'enable rotatary actions' is not a good long term solution, but we didnt conclude on how to replace it. Perhaps it should be one long set of actions, but then thought needs to be done on how to define press/release/delayed. Will this result in the list of actions being unreadably long?
Some experimentation/design work is needed here.

I have some unresolved questions, which need more investigation:
Should press/release be separate events that an action can listen to? How can the user choose what events it is listening to?
How can we allow for remapping events? (eg firing a press action on a rotary event)

[dnmeid]

I think we have a long enough agenda now for 3.1 and suggest we start here with a general implementation of passing additional metadata to actions and replacing the dedicated rotary actions with this. Replacing release actions can be the next step for 3.2, but it is good to keep in mind for now that there eventually will be more than rotary metadata.

[Julusian]

How to deal with that surfaces in Companion. I think while the concept of the pages shared by all surfaces, like we have it, is quite easy, once you got the point, we nonetheless have to replace it. This is going to be far more advanced and flexible for the power user but should be similar as easy for the novice user. First of all I suggest that we stick with a grid, let's call it the canvas. Every cell in the canvas has x and y coordinates and the canvas' size is only limited by the js number range.
Now we can place surfaces on the canvas completely to our liking, but a surface always snaps to a grid cell. The point is that we should have matches between I/O elements of our surface and the grid cells. That doesn't mean that a control has to be exactly a 1x1 cell, it just has to be at least 1x1.
Surfaces can also overlap partly or completely but don't have to. So now we can emulate what we are doing today: Button 1 is button 1 on all surfaces.
Surface rotation is tbd, maybe one individual module per orientation, maybe an option.

When I first heard the pitch for an infinite canvas and windows a while ago I was on board. But over time I have been going off it.
In a world where everything is simply a grid of buttons this works fine, but when you encounter things that arent just buttons, or arent a uniform grid I don't think it will work well.

Take the loupedeck-live as an example. (the ct has the same problem, but also more).
The vertical strips to the side would be nice to use as sliders/faders, but then which of the 3 cells in the grid would that translate to?
If a cell can be used on each of a button, and encoder, a fader, and a joystick, how will that be programmed in a sensible way? Or will it only be possible to set it up for one of those? (that could be acceptable given the ability to windows)

I would also like to question whether a large grid is in fact easier to work with instead of pages. While on a technical level it is superior, but from a usability I dont think it is. I think it will be much easier to understand having a streamdeck on page 50, than having a streamdeck at position 0x250. I am tempted to say it should be possible to do both, but I am not convinced that is worth the additional complexity.

---

I have been leaning towards a concept I am calling 'spaces'. It essentially boils down to having multiple sets of pages, and the ability to create a 'space' for a certain device type. So that when I want to use a loupedeck-ct, I will have to create a new 'space' for it, and it will get its own set of pages.
Streamdecks and other grid shaped devices should be able to continue to co-exist on a grid, ideally of a customisable size similarly to now. Having multiple of any type of space should be possible, to have fully isolated surfaces.

I think that this will solve a common confusion for new users, in that it will then be possible to have different pages for different surfaces and not to have to assign different ones to different surfaces. And it will give a better experience when programming for non button-grid surfaces, as the ui will tailor the programming to the device you are working with.

We can make it possible to 'symlinks' and 'hardlinks' the control definitions into multiple places (the new data structures in 3.0 are designed to allow for hardlinks), as long as the control type matches. So it will be possible to have the same buttons in different places and not have the maintenance cost of editing multiple places.

I don't have an alternate proposal for windows here, and I do wonder if lacking that will be a dealbreaker. I do see some benefits to them that I don't have an alternative to replicate (tabs, sticky buttons)

[Julusian]

That is exactly the very key question where a lot of my proposal is related to. I really can't say I'm 100% on the grid side. If I think of the grid vs. freely positionable I'm tossed back and forth. The result is: I tend to the grid.
Both aproaches have their points and both have their disadvantages. My main reasons for the grid are: it is the established trademark of Companion and it works best for the surfaces used the most; it is easy to understand and to address; it gives the user programming a visual structure and helps to keep things organised (think of some person's completely messed computer desktops).

I am not strongly against a non-grid solution, but I think the grid has more advantages.

To clarify, I am not suggesting a freely postionable system (not until we want to allow for building customisable webuis as a better alternative to the emulator/tablet views), but different fixed layouts defined by the surface plugins.

So it won't give the chaos of free positioning, but doesnt limit you to thinking solely in a grid.

The big question is how to fit non-grid like stuff in the grid.
So the easy part you can see in my gui mockup. I just pushed the rotaries of the loupedeck a bit to fit in the grid.
The slightly more advanced part is what to do with the touch sliders. In my idea, as I have written in the OP, a control/button does not have to be 1x1 grid cells. 1x1 is the minimum. Tbd if you can resize controls or if there are different control types which come with fixed sizes. But I think if you put a 1x3 control at the top of a loupedeck slider, we have all possibilities. Rendering graphis would be left to the surface module, so the loupedeck can render a 1x3 as one strip and a streamdeck at the same position can render it in three parts. Input from the loupedeck would be be whatever the touch generates in the extra data. Input from buttons at the same position would be just a press to the control and so on.

Which cells would the events be fired on? The top, middle of bottom of the three? (or all of them?) And similarly, when Im defining the style, which one should I do it on?
Sidetrack: When first implementing the loupedeck, I almost did support those vertical strips, but it looked as bad as the touch strip on the streamdeck plus. I did try and generate something, but I knew that someone would complain about it looking weird/different or didnt do pngs or something so I decided it wasnt worth the effort until it could be done properly.

Something I am wondering about, is that it would be nice (but maybe not essential) to allow for choosing how those strips behave. sometimes they could be used as 3 buttons, and other times as the full touch strips. In a spaces system, I can see that mapping as an option on the page, for which layout that page should use.

Actually I don't think that users are going to place mixed surfaces on top of each other but they could and the system would allow for it. For example putting a stream deck mini on top of a loupedeck live would be a perfect match.

There is stuff around, which really doesn't fit well in a grid. My only solution for this so far is upscaling. E.g. I have a midi-controller with eight buttons and five faders which take the same width as the eight buttons. There would be some grid cells with no surface element under it.

So is it beneficial to having a single large grid in this case? Maybe having multiple of these grids would solve it.

There would be some grid cells with no surface element under it.

I find this problematic. When configuring, and your surface is in another position (pretending to be on another page), will we be able to give enough of a hint of how the surface will align to this portion of the grid to help you remember which cells you want to be editing?

I would say it is. If you want to move something you are always at the same space. And you can scroll through your grid which is by far better then page changes. Pages only add to clarity if you can see everything on a page at once.

There is nothing stopping us from showing the pages as an infinitely scrollable list like in the tablet view. And I find that once the pages have been given names they make for nice bookmarks of positions (this is solvable in the canvas).
There is also a request to be able to insert and reorder pages. I am not sure that will be solvable on the canvas.

I think users could back themselves into layout issues if they dont plan well enough too. Perhaps wanting to change their 2-XL into 3-XL, but there is something else to the right of their 2-XL layour which will need to be moved. This could happen in either solution, but is more likely with the canvas, as there will be a temptation to not create a secondary grid when there is space to the right of everything else.

If you have to scroll to see everything on your page, then you can omit the pages at all.

I'm not sure I agree with this.

This is exactly one problem that I want to tackle too. I think my proposal is not the only way it could be done, but it has the advantage that existing configurations can continue working without any reconfiguration work and existing behaviour of multiple surfaces can be kept and is possible. And at the same time new users will be able to intuitively configure their system in a way where they assign different things to different surfaces.

I don't think your solution is any worse in this regard than we have now, but Im not sure it is definitely better. If its one grid for everything, then it will be the same but telling them to give an x-offset instead of assigning a different page. If there are multiple grids, and we can tell them to use a different grid then it would be better than now.

I really hope your plan here is to have multiple grids, as that is something I would really like to have. In my home/office setup I have 5 different surfaces using their own set of pages (an XL with 9 pages, something with 3 the others with 1 or 2), which will give a lot of scrolling to navigate to find everything. As these are various different controllers, serving different purposes it would be nice to isolate them in separate grids (or my plan would give them a separate space each). Also the pages for the XL are in a bit of a random order at this point, as I've had to add new pages after the others instead of in the middle next to the ones they belong with. Every one of the pages has navigation buttons to 4 of the others, so reordering them is a pain. But windows and conditional actions would most likely avoid this pain

[dnmeid]

different fixed layouts defined by the surface plugins

I was thinking about that too, but got away from it because then we either can't match different surfaces on top of each other or need an additional method how to map them.

Which cells would the events be fired on? The top, middle of bottom of the three?

There is still some room for discussion. One approach would be that if a surface element is visually larger than 1x1, it has a range of covered cells but only one lead cell, usually top left, but the surface module designer may choose a meaningful mapping (think of jog/shuttle). What would happen with events routing: the strip is used and there is a 1x3 control on top -> the 1x3 control gets an event with touch data from the lead cell; one of three buttons is used which happens to be positioned at the same place as the strip -> the 1x3 control gets an event with press data from the button and with position data within the 1x3 range from companion (combining multiple surface elements to one control); the strip is used and there is a 1x1 control at the lead cell postion -> the 1x1 control gets an event with touch data; the strip is used and there are some smaller controls within it's area, but not at the lead cell position -> tricky case (splitting one surface element to multiple controls), either only control at lead cell gets the event or all controls are getting the same event with some additional info from companion or the surface module has to define different methods for sub-areas.
Another approach would be to have different surface modules or being able to switch behaviour at module configuration, so the module can expose the strip either as one large 1x3 control or as three 1x1 controls. That would be easier to design and to understand but than the user has to make a usage decision which can't change in one configuration.

In a spaces system, I can see that mapping as an option on the page

That would definately be an advantage of pages. At the same time I'm afraid that will be another step in the direction what I call design by feature. Think of a loupedeck ct, you need two options for the strips one for the jog, one for the shuttle and then it will take two weeks until the first user says why not make the rotaries optionally splitting in two buttons too.
My usual approach is to develop a design which covers maximum features and not to start with features and then find a place where I can put this in the ui.

There is nothing stopping us from showing the pages as an infinitely scrollable list like in the tablet view.

Page size is one problem and a non-grid layout would be another problem.

There is also a request to be able to insert and reorder pages. I am not sure that will be solvable on the canvas.

Correct, that would be a different workflow like selecting multiple controls and dragging them to a new position or cut/paste.

I think users could back themselves into layout issues if they dont plan well enough too.

Yes, I don't think we will find a solution which is able to cope with users not planning well, but the need for unforeseen changes can hit you at any time and we should provide a system which can adopt to changes with minimum effort. With any solution they have to learn how to make the best use of it. Today's pages are not very flexible and the question of inserting and moving pages is exactly one way to face it.
In my idea the combination of the grid and the winwows would make it quite easy to adopt for newly added devices. With a controls grid and windows two functions are separated: the grid makes a logical layout of controls where the position of controls in relation to each other is defined. The windows present the controls at surfaces. In fact the idea is when I need to add a surface because I need more different actions, I can do this without problems and when I need an additional surface for extended acces to the actions I allready have, I can do this super fast. At the same time the additional surface can be a different size and it is easy for the user to use all his banks on different surfaces.

I really hope your plan here is to have multiple grids, as that is something I would really like to have.

I've been with multiple grids for a long time. It is a trade between multiple grids or pages or tabs or windows. I started writing down this idea last year and in between I changed the part what is there multiple times multiple times. In the end I think having one grid and multiple windows to generate different views of the grid is a flexible and intuitive method.
The only real drawback I see is large scrolling distances in a large project. I think navigating to a bookmark should be a good solution for this and not take more time than switching to a different grid. Missing isolation I do not see as a drawback but more as an advantage. I guess the most common usage would be to have multiple surfaces side by side. If you want to do syncronized "page" changes on some of them, you span one window over these surfaces and scroll that window up/down. Other, independent surfaces can have their own window which can scroll up/down a different part of the grid.
Much more advanced solutions are possible and actually not that complicated to achieve, e.g. tab-controller, sticky buttons, context sensitive pop-up windows, nested windows...

The teamplay of surface, grid and windows is very hard to imagine at first and sometimes I struggle with it myself. But once you see this in action it will be very easy.

[Julusian]

I was thinking about that too, but got away from it because then we either can't match different surfaces on top of each other or need an additional method how to map them.

I don't think we need to be able to match surfaces on top of each other, but I agree it would need (eventually? for the MVP?) to be able to do a remap while importing the config.
The benefit of the single grid would be that this remapping would be done post-import in the same ui as normal editing. Unless we add some bulk editing tools, it would require updating every 'page' individually, which could become tedious.

Which cells would the events be fired on? The top, middle of bottom of the three?

There is still some room for discussion

I am happy enough with this for now, I don't think we need define the exact solution just yet. The main thing to be aware of is that I think this will be a source of frustration/confusion for users of these devices, when configuring these controls which span multiple cells.

That would definately be an advantage of pages. At the same time I'm afraid that will be another step in the direction what I call design by feature.

Isn't all design driven by features? If a feature is needed that can't be satisfied by a design, then isnt that typically the point of redesigning?

I admit this is driving the design based on expected UX, but I dont see that as a bad thing.

then it will take two weeks until the first user says why not make the rotaries optionally splitting in two buttons too.

I see these options as things that surface modules can expose. Either options, or perhaps simplified as choosing a 'layout' for each page.
Not impossible to achieve with the canvas, but I dont think it will be as nice.

There is also a request to be able to insert and reorder pages. I am not sure that will be solvable on the canvas.

Correct, that would be a different workflow like selecting multiple controls and dragging them to a new position or cut/paste.

And then also updating any 'page change' actions to use the new positions. 'bookmarks' with a way to jump to those would solve it for most cases, but not 100% of all cases.
Spaces has a similar problem, I intend to make it so that pages are internally referenced by a guid to allow for this, it would still be possible to address them by index.

In my idea the combination of the grid and the winwows would make it quite easy to adopt for newly added devices. With a controls grid and windows two functions are separated: the grid makes a logical layout of controls where the position of controls in relation to each other is defined. The windows present the controls at surfaces. In fact the idea is when I need to add a surface because I need more different actions, I can do this without problems and when I need an additional surface for extended acces to the actions I allready have, I can do this super fast. At the same time the additional surface can be a different size and it is easy for the user to use all his banks on different surfaces.

I don't see this as being any harder in my proposal, or much different to today?
In any case, you need some cells in an appropriate layout for the device you are adding. This probably means you have a page/window already in use, which wants to be 'reassigned'. That reassignment flow will be different for each, but is not too complex in any of them.

The bit I think is harder is when you need new cells for the new device.

I've been with multiple grids for a long time. It is a trade between multiple grids or pages or tabs or windows. I started writing down this idea last year and in between I changed the part what is there multiple times multiple times. In the end I think having one grid and multiple windows to generate different views of the grid is a flexible and intuitive method.

I've thought about it some more, and I think multiple grids is a much appreciated feature. It should be possible to assign as many surfaces to each grid as you wish, but I think having isolated grids is important.

Pretty much just as a way of splitting things up into logical chunks and to avoid getting yourself into layout issues.

1. If you have an installation which is running for years, and new surfaces get added and old ones removed. You will gradually end up with gaps, where removed surfaces used to live or will need to move things around to keep it compact.
2. If multiple users are configuring the same system, then they will need to coordinate what goes where. Doable, but an extra burden. I expect the other option would be for one of them to use a large x offset, which they will find annoying
3. Part of your config is persistent, and part of it changes weekly. Part of a flykit, a section of it is built as a system setup tool, and a section used for operating the show. This is more of a 'peace of mind' thing, knowing you wont be messing with the wrong section and can clear everything out of the dynamic section much easier.

The teamplay of surface, grid and windows is very hard to imagine at first and sometimes I struggle with it myself. But once you see this in action it will be very easy.

I definitely see the value and power of it (I wish I could find a way to have windows in my pitch), my concern is that as part of pivoting to non-grid devices I worry that forcing everything into a grid will be less friendly to users.

A big part of that worry, is that in your control and window layout screenshots, I know these are a crude proof of concept, but its a compact grid that we would likely be aiming for. The problem is that I can't tell where those loupedeck encoders will reside. Or for those touchstrips, whether the module author mapped them to the top cell, or the middle cell. And I don't see how that can be solved without making the grid incredibly noisy.
The surface view kind of helps here, but only for where it is currently positioned. Unless we show the other possible positions of a device (which could be anywhere) I don't see how this could be conveyed to the user in a helpful way.

[dnmeid]

Correct, that would be a different workflow like selecting multiple controls and dragging them to a new position or cut/paste.

And then also updating any 'page change' actions to use the new positions. 'bookmarks' with a way to jump to those would solve it for most cases, but not 100% of all cases.

No, when I want to move controls to a different page, I just move the controls to a different position. The page change or bookmarks are not affected. When I want to reorder pages, I move the controls together with the bookmarks (for me a bookmark has a visual representation in the grid). Updates to user programming would only be needed depending what type of references had been used (scroll by offset, scroll to absolut position, scroll to bookmark).

The bit I think is harder is when you need new cells for the new device.

That is something I actually think is avoided with my idea. Let's look at the following example: a user has made a programming for one sdXL and now adds a sd+ to the existing set. He now has the option to put both devices on top of each other and opt for individual positions/pages per surface. This would give the exact same behaviour like we have today. No need for new cells or any new programming. He also has the option to position the sd+ right to the xl and make a new window for it with the same size of the xl window. The window uses the same grid with the same controls, so again no additional programming needed. It gets more interesting if he makes two windows for the sd+. One window for the buttons can use the same controls as the xl. Scrolling in a page manner can be influenced by the size of the window, make the window as alrge as the xl window and you have e.g. a top/left part of your xl window pages, make the sd+ button window 4x2 and you can reach all controls (assuming "page" change is done with scroll by window height). The second window would cover only the rotaries and the user usually would want to program some new controls only for the rotaries.
So reusability and adaptability actually are big goals of my proposal.

3. Part of your config is persistent, and part of it changes weekly. Part of a flykit,...

That's the point where you get me! I change my mind again and now favour multiple canvases again.

I worry that forcing everything into a grid will be less friendly to users

That is definitely a big discussion point. Grid vs. non-grid. Usually I'm not a big fan of restrictions and a grid is a restriction. In our case I think that a grid has more advantages than disadvantages. If I think of the average user I guess a grid is exactly what makes it easier for him to keep things organised and aligned and makes mapping of different surfaces possible for the average guy at all. There will be situations where surface elements need to be offset to fit into the grid and depending how fancy the surface is, this mapping to a grid cell may not be obvious without a visual hint.

The problem is that I can't tell where those loupedeck encoders will reside. Or for those touchstrips, whether the module author mapped them to the top cell, or the middle cell. And I don't see how that can be solved without making the grid incredibly noisy.

I think that is not a real issue. In my screenshots I just used pictures of the devices. Actually the idea is being able to draw a device simulator from the description language, so the exact design of how the real surface will be shown in one of the grids is easy to manage. Maybe we do some color coding for element types and rotaries are always a yellow circle and buttons are always a red square (this is not a design proposal). Also we are not forced to stay with the ratio of control area to gaps between them. And last but not least I expect that users will usually not so much focus on aligning controls to surface elements, but rather align windows to surface elements and then align controls to windows.

Unless we show the other possible positions of a device (which could be anywhere) I don't see how this could be conveyed to the user in a helpful way.

I don't get what you mean here. Why would I have to show any other position of a surface than its current one?
What we should do, is give the possibility to add surfaces to the configuration, which are currently not connected, so you can do some preprogramming.

[Julusian]

After discussing this further, our agreement was:

We should start with the grid layout rather than device specific for now. We can add device specific layouts later on, and whether we do that now or later will still need some backwards compatibility of being able to put non-grid surfaces on the grid. The benefit of the grid is the ability to quickly change what surface is being used at the last minute without needing some lengthy rebuilding/remapping.

We want to keep pages, with each becoming an 'infinite' grid. This will give us the best of both worlds, being able to use different pages for different surface types, or sharing. The ability to add/remove/reorder pages is important to this

[Julusian]

I suggest to separate the buttons from their actions and unify this with the triggers. What do I mean with this? A button has some steps with actions, a trigger has some actions. A button's actions are triggered by a press or release, trigger actions are triggered by a self made condition. What if we tell the trigger to run on a specific button press? Or the other way round if the only action of a button is to run a trigger. Let's take the steps and stuff from the buttons and move it to a new element, let's say "stepped script". The actual trigger and condition part of the trigger is part of the element stepped script, because it defines when to run or advance the stepped script. A button can also trigger or advance the stepped script, so we actually are generating a new state (at which step are we now) and many buttons can share that script. In the long term there can also be other types of scripts, e.g. user defined javascripts. They would have the trigger part to decide when the script is run and then they could access variables and run actions and so on but all from the javascript.

I am not a fan of moving the actions off of buttons in the ui. I think that doing so will make it much harder to follow what will happen when you press a button. It is possible that this can be solved with some UX work, but does clicking a cell in the grid, then clicking 'view actions' which opens another panel/view help with understanding what happens or make it harder?

I wrote about this the other week somewhere else, but I view this as triggers are an 'advanced' feature that are designed to be powerful and as a consequence are less intuitive and harder to use. But buttons/controls should be pretty self explanatory and understandable to beginners. Ill admit that it is entirely possible that the steps functionality goes against this, but that should be easily possible to make less prominent.

That said, I have no problem having the ability to create stepped scripts or other complex things, as long as they dont increase that beginner burden. Perhaps this will even be 'free' with the ability to 'hardlink' controls? I am expecting them to live in a pool, so it would be possible to have a control not assigned to a button. That control would be possible to invoke directly, so would in effect be this stepped script.

Fun fact: triggers are also a different form of 'control'. They can't be placed, but live in the same data structure as buttons do

[dnmeid]

I am expecting them to live in a pool, so it would be possible to have a control not assigned to a button.

I guess we are at the same page, that is exactly one part of the idea.

I am not a fan of moving the actions off of buttons in the ui.

Let me qoute myself from the op:

For convenience it could be possible to have inline creation and editing possibilities, then programming buttons would feel much the same like today.

[Julusian]

Let me qoute myself from the op:

For convenience it could be possible to have inline creation and editing possibilities, then programming buttons would feel much the same like today.

Ah my mistake. it was a long block of text :)

[Moulaythami]

@dnmeid your ideas for a new interface looks wonderful!
I hope someday the leftsidebar will be indeed more usefull than it is today.

[isaacdantzler]

Seeing a lot of the proposed ideas here integrated in Buttons and I really like them there from what I've messed with.

I think this sort of position-based management is much more intuitive than pages. I think having the surface overlay grid be separate from the buttons grid is the way to go.

Hoping that Buttons will help enable Companion to continue to grow!