Call for discussion: Announcing a OH3 beta program and involving the community

[ghys]

Hi follow AC members!

As openHAB 3 snapshots become more and more usable, I'd like to spark a discussion about how we'd want to get to the point that we would consider it stable enough to be beta tested by the community.

Indeed a lot has happened in the last few months, and there are several major changes and enhancements in the core functionality which might be widely untested and under the radar right now, until we communicate on them properly:

• all OH1 bindings are now deprecated and cannot be used anymore, the good news is that there are modern equivalents on the way for most of the popular ones;
• the dashboard has been replaced with a "main UI" which, while still allowing other alternative UIs to be discovered and launched, will hopefully offer all the necessary features for general usage (and in the process deprecates Paper UI and HABmin which are not offered in the distribution anymore);
• the potentially destructive admin-only API operations are now behind an OAuth2-based authorization flow;
• the sitemap UI concept has been complemented with a more generic way of storing UI components in the JSON DB; this led to various additional types of visual interaction being introduced in the main UI (under the "pages" moniker): layout pages, map views, floor plans and charts (charts are on the way but still a WIP as of this post). Pages may reuse custom widgets designed with the same component-slot paradigm, reference each other in modals/tabs and (spoiler alert) there will also be options to restrict their display to authenticated users with a certain role, so that admins and power users can have ways of interaction not meant for end users or guests.
  Additionally, components defined in a special namespace (system:sitemap) are translated to sitemaps via a provider, so the main UI now has the ability to design "managed" sitemaps as an alternative to .sitemap files (similarly to items and things) which are compatible with Basic UI, the native mobile apps and other sitemap-based UIs;
• the persistence addons can now define default persistence strategies, and rrd4j is installed by default as part of the standard package and enabled for all numeric items and switches;
• admins are strongly encouraged to build a comprehensive semantic model of items (locations > equipments > points) in the main UI, with dedicated screens (Model) and operations (Add Thing As Equipment etc.) - hopefully they will understand it's a good idea to do that, because easily-accessible automatic views of semantically-tagged hierarchies of items will become available, as well as a prominently featured HABot texbox in the home page when appropriate ;);
• the DSL rules are in the process of being exposed as regular NGRE rules so that they may be viewed in the UI - and potentially the DSL will become an alternative to ECMAScript and Jython to write rules directly in the UI (not sure about that?)

So I guess my point is: at some stage (maybe now or maybe not, that's the purpose of this post) we might benefit a lot by giving the go-ahead to the community to start contributing, not merely with comments & criticism in what we're doing, but to help with:

1. a presumably major overhaul of the docs;
2. i18n - I had a very good and humbling experience with Crowdin for HABPanel so I'm sure community translations will come if we provide the infrastructure;
3. general issues - not personal ones but obvious things we might have overlooked.

For 1. it will involve some work with the docs repository and Jenkins which is still not complete, but I think we're close to the point where we can start accepting non-technical contributions.

What do you think?

[rkoshak]

I don't have any argument against this proposal but it's not clear to me how this would look from a practical perspective and I was too much of a neophyte when 2.0 was made available for use to remember how that was done and get any lessons learned from that. Perhaps an announcement thread saying that the 3.0 SNAPSHOTS are usable now but only for those willing to dedicate the time and effort to figure stuff out on their own?

I definitely agree, there will likely need to be a near complete rewrite of the docs. And we need people to start working with the new baseline to gain the knowledge and experience necessary to build up those docs. And we are kind of stuck in a Catch-22 where most users won't be able to use the new stuff without docs and we won't be able to build the docs until we have some users who know how to use it.

Some questions I have:

• Do all the 2.x bindings appear to work as well on 3.0 as they do on 2.5? If not do we have a list of things we should watch out for?
• Beyond 1.x bindings no longer being supported, are 2.5 configs migrateable to 3.0 with minimal or no changes? If changes are required, are they all known? Is there a difference between text based and JSONDB configs?
• Are text configs still supported? If not is there an import mechanism for them (I know you wrote an importer for .items files which I use quite a bit actually.
• What's going to be the "default" rules language? I'm not sure we have the manpower or expertise to write docs for 4 separate languages with two different approaches to building Rules for each, at least not to start out.
• Are the devs willing to spend a bit of time helping on the forums for a bit until the regulars get smart enough to help?
• Are there any radically changed features or new features (e.g. scheduler, replacement of Expire binding, new rule triggers, etc.) that might make a simple migration more challenging? I worry that the semantic model might be challenging for some users to understand and apply when they migrate. Are there other features like that to watch out for?

If we don't know the answers to these questions I think it might be too early to bring in the community. I'm not looking for perfect of complete answers, though. But in the announcement we need to have enough information for those who are not spending all their time on GitHub watching the issues and PRs to get something up and running so we can play and start to learn the rest. If we do know the answers to these I say bring it on, just make sure to provide the answers to the above in the announcement. I personally am looking forward to having better tools to move everything to JSONDB. Based on my experiences with HestiaPi which runs OH on an RPi 0, the difference in performance and reliability is huge.

Those are my thoughts which may not be completely reasonable given that I've not been following the 3.0 development that closely so don't really know what to expect.

[ghys]

Some questions I have:

Do all the 2.x bindings appear to work as well on 3.0 as they do on 2.5? If not do we have a list of things we should watch out for?

For now, yes! If you don't use 1.x bindings you can install any 2.x binding in your 3.x instance (in fact they will be offered) and they should work!

Beyond 1.x bindings no longer being supported, are 2.5 configs migrateable to 3.0 with minimal or no changes? If changes are required, are they all known? Is there a difference between text based and JSONDB configs?
Are text configs still supported? If not is there an import mechanism for them (I know you wrote an importer for .items files which I use quite a bit actually.

for now there's no major changes in the way things and items are provisioned - the UI would be enough (with a little lock to depict his status?), but in the case it doesn't fly, those coming from the configuration files will have a little lock besides their name meaning they can't be altered.

What's going to be the "default" rules language? I'm not sure we have the manpower or expertise to write docs for 4 separate languages with two different approaches to building Rules for each, at least not to start out.
Are the devs willing to spend a bit of time helping on the forums for a bit until the regulars get smart enough to help?
Are there any radically changed features or new features (e.g. scheduler, replacement of Expire binding, new rule triggers, etc.) that might make a simple migration more challenging? I worry that the semantic model might be challenging for some users to understand and apply when they migrate. Are there other features like that to watch out for?

[digitaldan]

Thanks @ghys for getting this thread going, I have been thinking about it as well.

@rkoshak you bring up excellent points (as usual). There is a mountain of work to get down between now and GA, but also before beta, some of it is probably in issues on github and being tracked, many are not being tracked.

For the larger group, do we want to brain storm what this list is? From this we can then create a new ticket for each item not already present and tag/milestone it with "oh3-beta", "oh3-rc", etc.. to signify when it should be completed by? There already exists a OH3 label and similar milestone, but not many open issues are there.

I'm not trying to create more busy work, but i don't know how else to keep track of everything. I am open to other ideas for sure. Ultimately i would like to be able to say "we need to complete these X things before we have beta 1"

For example (non exhaustive, off the top of my head)

• Documentation
  • Upgrade from 2.x
  • Binding updates from 2.x
  • New UI
    • Welcome
    • Admin
    • Sitemaps
    • Plans and Pages
• WebUI
  • Legacy Sitemap support
  • Authentication / Authorization
  • Rules configuration
  • Services Configuration (specifically persistence)
  • Textual import compatibility
    • Things
    • Items
    • Rules
    • Sitemaps
• Core
  - [ ] Authentication / Authorization
• Bindings
• ETC

There is also the Github Project board we can use to help manage this, i don't have a preference either way, but it would be nice to see all the work in one place.

@rkoshak as for your other questions:

Are text configs still supported?

@ghys posted as i was typing this, but one thing i will add to his comment is that he created an AMAZING config importer in the new UI, I was able to dump hundreds (thousands ?) of lines of items and things into the UI, and it recreated those using the the API.

I have not yet tried running more then a bare OH server yet. i'm sure once i do i will want to start documenting issues and gotchas.

[ghys]

@ghys posted as i was typing this, but one thing i will add to his comment is that he created an AMAZING config importer in the new UI, I was able to dump hundreds (thousands ?) of lines of items and things into the UI, and it recreated those using the the API.

Glad to hear that 😄
But how I wonder if we could focus on that even more, like have a command in the VS Code extension, which would "apply" the content of a file to an openHAB instance (that could be things, items, or whatever) similar to thekubectl apply -f ... Kubernetes command (sorry those not familiar with it) - ultimately everthing would end up in the JSON DB,.. that would really be a game changer I think

[digitaldan]

But how I wonder if we could focus on that even more,

I like the idea of the vscode extension for sure. What also might be nice is a UI (vue) based "upgrade" wizard to walk one through importing text files ( and instructing one to then remove them). While still a manual process, my experience is the process would take only a few minutes to move a large system, just copying and pasting from files. I was also contemplating what an upgrade binding might look like, not sure if that's even feasible, or maybe just too much work given everything else that needs to be done.

[rkoshak]

I've been very happy with the very early prototype Items importer @ghys let me play with a few months back. I keep coming across places where I need to use it and it is a really handy tool, and I've not yet even migrated my Items to JSONDB yet for my main OH instance. :-D It will truly be one of the more used features I'm sure.

I too like the idea of a way to import the text configs being worked straight from VSCode, but I worry that if you can go one direction (text to config) the natural question will be why not the other direction (JSONDB to text)? For a lot of cases, I can see the workflow of pull the configs, edit them, push them back to OH being very powerful and help a lot of the "text configs first and only" crowd adjust. I've personally found this to be a rather natural approach even in OH 2, though I've been using the REST API and the raw JSON to do it. It works great for Items and Things, not quite so nicely with JSONDB stored Rules as the JSON replaces all the new lines with URL encoding so anything more than a couple of lines of code long is really hard to read and edit by hand, though I'll still do it in a pinch.

NOTE: On the HestiaPi which runs on an RPi 0, I rewrote their OH config so everything possible (Items, Things, and Rules using ECMA so there is no extra steps or external requirements) is in the JSONDB and it took the boot time from over 30 minutes to less than 10 (still too long but it's an RPi 0, not exactly supported hardware. Because I can't help myself, the new configs actually do more in much less time. Anything we can do to get people to move to JSONDB configs as their default and if we can get old timers to migrate too I think it will be a huge net improvement for the community, even if people fight against it with all they have. I'm really looking forward to the new UIs and anything we can do to ease the migration will be be huge.

As for the brain storming list of things needed before the beta I can't speak to the feature list. You all know more about what's being worked and the progress being made. But I definitely agree with the list of docs. Something should be there for the beta so users can start to play without spending too much time spinning their wheels. But it doesn't have to be polish nor wholly complete I think. Just enough to get over the big gotchas. During the beta period we can flesh out the docs with participation from the community.

But before the general release I think we need the documentation to be pretty solid and complete. And we need a real new user's tutorial that is complete. The OH 2 tutorial never got finished and I'd like to avoid that situation again. I think this is especially true with the rising importance of semantic tagging.

[ghys]

I too like the idea of a way to import the text configs being worked straight from VSCode, but I worry that if you can go one direction (text to config) the natural question will be why not the other direction (JSONDB to text)? For a lot of cases, I can see the workflow of pull the configs, edit them, push them back to OH being very powerful and help a lot of the "text configs first and only" crowd adjust. I've personally found this to be a rather natural approach even in OH 2, though I've been using the REST API and the raw JSON to do it. It works great for Items and Things, not quite so nicely with JSONDB stored Rules as the JSON replaces all the new lines with URL encoding so anything more than a couple of lines of code long is really hard to read and edit by hand, though I'll still do it in a pinch.

FYI I've made an habit of adding a Code tab to a number of screens lately (but for individual objects, not collections) because it's often easier indeed to copy & paste repetitive stuff (like rule actions etc.) than clicking around.
See for instance:
https://ohui.azurewebsites.net/#!/settings/rules/updatelocation https://ohui.azurewebsites.net/#!/settings/pages/layout/page1
https://ohui.azurewebsites.net/#!/developer/widgets/weatherforecast

For sitemaps I've kept the sitemap syntax (https://ohui.azurewebsites.net/#!/settings/pages/sitemap/add) to ease migrations and prevent a lot of documentation and examples, including on the forum, from becoming obsolete. For object types which don't have a description syntax, however, like (NGRE) rules, they are represented in YAML; for all its flaws, it is good enough in most cases.
I'm thinking about doing the same for items and things, those screens were designed earlier so they don't have code views - the Thing page has a button to get a read-only textual representation but it's a little buggy.

So in theory exporting the data as text could be done. The advantage compared to the current way of listening for changes in the configuration directory is that they can be edited from anywhere, you can keep your "source files" on your local computer (or clone a Git repository with your config), edit them in VS Code or another editor and apply changes to these source files to your remote instance - no need to set up Samba shares or things like that. The drawback is that the "source" and the actual instance config are not necessarily in sync.

But before the general release I think we need the documentation to be pretty solid and complete. And we need a real new user's tutorial that is complete. The OH 2 tutorial never got finished and I'd like to avoid that situation again. I think this is especially true with the rising importance of semantic tagging.

Totally, that's the main reason why I wanted to discuss this with you.

[kaikreuzer]

Hey all, I've discussed this topic p2p with some of you, so let me briefly provide my pov here:
I am with @rkoshak that as soon as we publicly ask people to try out 3.0-SNAPSHOT, a lot of questions will arise (like what's the roadmap, what's the suggested way of configuration, rules, etc.? How do I do X and Y? I found a bug in binding Z...) and I am pretty certain that this will bind a lot of time from the developers, which would currently be better be invested in, well, developing.

I understand that @ghys works at least 10x faster than me at the moment and that the new UI is in a fascinating state that could already have users playing around with it to get some feedback. I personally am still struggling on replacing the old rule engine, while keeping DSL rules still working on NGRE (which will imho be hugely important for the acceptance of 3.0). My personal goal with this exercise is to get asap to a point, where 3.0 has all required features to use it as my production system. Once I use this code base myself, there is a much bigger motivation to quickly fix bugs, find things that need to be improved and how configuration management feels in a real system that isn't just 10 test items.

My preferred first step would therefore be to have the AC and core maintainers actively use 3.0-SNAPSHOT. When we all have real-life experience with it, discussions on what still needs to be worked on are probably much easier, because there's a common understanding/experience. -> Eat your own dogfood!

Everyone of you will have different requirements of what is still missing, before it is usable for you, so we definitely have to come to some kind of feature/todo list or roadmap. We had established https://github.com/orgs/openhab/projects/3 for this purpose and I'd suggest to use it - it is clearly currently missing many important things, probably also stuff for which we never created a formal issue (like the removal of the old rule engine, which I am working on). We should imho try to get this in shape, so that it can serve as a point of reference and also helps us tracking the progress.

We had planned to move add-on development from 2.5 to 3.0 (master) by late summer. Imho this would be the best moment to bring 3.0 to a wider public, because all binding developers will naturally have to try it out. And assuming that other community users will quickly start reporting binding bugs on 3.0, I think it is helpful, if the binding developers are also on board already.