bndtools should assist with importing projects from known bnd workspaces

[joe-chacko]

Problem description

In the bnd workspace model, when bnd is "in charge" of eclipse project dependencies, there are various usability issues:

1. Importing the projects can take a very long time.
2. If the projects are imported out of dependency order, errors can show up due to missing dependencies. Resolving these errors can take a lot of trial and error with cleaning, rebuilding, and even closing and re-opening projects.
3. Even once the projects are imported, the workspace can be slow to load, slow to rebuild, and very susceptible to incoming changes from version control that alter the dependencies.

Proposed solution

The eclipse bndtools plugin should offer a project import wizard that looks at a specific bnd workspace on the filesystem and allows the user to select which projects to work on. It would then import all the dependent projects too (as known to bnd), and optionally the using projects too (the ones that depend on the selection).

Proof of Concept

While working on OpenLiberty, we developed some tooling that used bnd to discover and navigate the dependency tree and allow the importing of related projects too. This brought down the time to start working on a new workspace with some projects loaded from several hours to 1 minute.

The eclipse plugin we created did the following as a project import wizard:

• asked the user to identify the bnd workspace
• discovered the projects in it
• presented these to the user to select from
• automatically imported depended-upon projects
• optionally imported projects that depended on the selected projects (including their dependencies)

Although our implementation is specific to OpenLiberty, the approach is easily generalised to the bnd workspace model.

Here is the (liberty-specific) workspace location dialog:

Here is the project import page. I have selected three projects to import, and requested the users of those projects be imported too:

One minute later, we had a working workspace, compiling without errors, with 144 projects imported:

[pkriens]

This is really good! Where is the frigging PR? :-) I like the idea!

Interested how you do this. You got one bnd/git workspace on location A and then open up different new Eclipse workspace on location B with this partial view?

There is a synchronize button on the Bndtools Explorer. This currently synchronizes with all the projects in the bnd workspace. This will need some adaptation so it won't drag in the rest. Your UI could be connected to this button though.

Looking forward to a PR.

[joe-chacko]

Cool — it will take me some time to port it, but good to know the idea has legs.

The screenshots shown are for a new Eclipse workspace. It is in a separate location from the bnd workspace.
Once you tell the import wizard where your bnd workspace is, it imports the cnf project automatically (as you can see in the background of the first screenshot).
After that, you select which projects to import from the bnd workspace into the Eclipse workspace.

Your comment has me thinking though — what do we do when the workspaces are the same filesystem location?
It should be doable, but it will need some consideration.

[pkriens]

I am working on a new bnd workspace wizard and the automatic syncing with Eclipse. This is awfully close. I wonder if we could integrate this?

My current focus is to use fragment templates that could be easily extended with a simple PR. They are fragments that just do one thing: gradle, maven layout, Java 17, OSGi R8, distribution. etc.

What I could so is start with a page that selects between the classic workspace templates (which should be deprecated over time), my fragment templates, and sub-setting an existing workspace (as you propose).

In the subset case, we only need a File to the bnd workspace and let the user select the projects it is interested in. You can only create/delete projects in the current instance of Eclipse so it will require acknowledgment to delete the Eclipse projects. With some changes to the WorkspaceSynchronizer, the project creation/deletion should be straightforward. It now compares against all projects in the bnd workspace but it should be simple to give it a list.

You have a fancy drag/drop UI. Although you got >1000 projects, I think this could be easily handled with a CheckboxTableViewer? When you click on the projects you want, we can immediately calculate which projects are included. I guess we need a search bar on top of it.

The workflow would then be. Open a new Eclipse workspace, create new bnd workspace wizard, select the sub-set, chose the bnd workspace, pick the projects, and sync against the list.

If you like this idea, could you make your sources available on github? Maybe we can work together on this?

[pkriens]

I've done some more pondering and came to the conclusion that adding it to the new bnd workspace wizard is not efficient.

Trying to see the use case I'd say a user creates an Eclipse workspace on a bnd workspace. We currently have the Bndtools Explorer sync action to synchronize this completely. What we need I guess is a button to show up your UI that allows you to efficiently select another subset of projects. It would then rearrange the current Eclipse workspace. as fast as possible.

Anyway, commit a PR and lets take it from there.

[joe-chacko]

Your new workspace wizard — is that to create a new BND workspace? Presumably the user already has an Eclipse workspace they are using at that point.

I will look into the checkbox table viewer. I'm not particularly wedded to the drag & drop UI. That was just what we came up with starting from a non-UI background (our first few iterations were command-line tools that kicked off Eclipse project imports).

[pkriens]

Yes, the new workspace wizard is always from existing workspace. However, I allow the selection of another bnd workspace and then use that as the Eclipse workspace.

The CheckboxTableViewer is a very easy way to select from a large input set.

[habiblawal1]

@pkriens Hey Peter, I've tried opening the bnd repo in Eclipse to try copy over the elph plugin, however I'm struggling to fix the issue of a bunch of exclamation marks and red crosses on the folders which are leaving me unable to run the .bndrun files from the bndtools.core project as shown in the following guide https://bndtools.org/development.html. Is there any steps you recommend to help try fix this? (I do have bndtools installed in Eclipse and have also tried building from the command line too)

[chrisrueger]

@habiblawal1 have you installed bndtools itself too into your Eclipse? I could not see that from your screenshot.

UPDATE:

I do have bndtools installed in Eclipse and have also tried building from the command line too

sorry, my bad I did not read carefully.
Hmm strange. Which version of Eclipse are you running on?

One more idea:
Can you switch to the bndtools perspective where you see the Bndtools Explorer and the Repositories?
And then click on this refresh button:

Also maybe check if you can group your Problems view by type to see if you see Bndtools errors. In my case it looks like this, for a Workspace which builds fine (I'm on Eclipse 2023-09 (4.29.0)):

[habiblawal1]

A lot of the bndtools path problem issues tend to be about different project directories not being subdirectory of the bnd workspace /Users/habiblawal/Documents/GitHub/bnd/bndtools.core.test/resources/workspaces/editors/quickfix .

I've tried on both Eclipse IDE for Eclipse Committers Version: 2024-03 (4.31.0), and Version: 2023-12 (4.30.0) Eclipse IDE for Java Developers

[chrisrueger]

different project directories not being subdirectory of the bnd workspace

Ok you found the reason.
The projects must be all inside the bnd workspace.
The question is: how did you import them? Did you follow the instructions on https://bndtools.org/development.html#importing-into-eclipse

I tried to do repeat this with a new workspace and it looks like this:

1. in a new Eclipse workspace do "File / Import / Existing Projects into workspace"

2. Select "Browse" and go to the folder which contains your checked out bndtools git repo

3. After importing it starts building which takes some minutes. In my case it did not completely build the first time. I just had to select and refresh all projects (Ctry - A all Projects + F5 ). Then the build completed and it should look like this:

[habiblawal1]

Thanks for the quick feedback, its much appreciated. I'll give that a try now :)

[habiblawal1]

Yup its fixed, I think the issue may have been that I had search for nested projects ticked. Thank you!

[chrisrueger]

@habiblawal1 great to hear 👍 Glad I could help. If you think something should be added to https://bndtools.org/development.html#importing-into-eclipse then let me know.

[habiblawal1]

I've created a pull request to inlcude elph I've anyone is able to help review it please :)