Epic: Variables pane categories and Package/module/submodule view (handling language-pack differences)

[jgutman]

There are a number of open questions around the Variables pane and more generally, how we want to surface to the user the packages/modules they have installed/imported/attached in their environment/namespace.

1. Language-pack specific categories/kinds. Right now, all possible kinds for the variables pane are hard-coded in the Positron front-end, and each language-pack then specifies for any given variable what "kind" it is. This forces a common vocabulary between the language packs, which may be awkward if different languages use different terminology to refer to the same/similar concept (e.g. "Packages" or "Libraries" versus "Modules" although in this case we could potentially use "Packages" for both R and Python) as well as if a certain concept applies only to one language but not another (e.g. "Classes" -- in Python if I import or create a class definition it will appear in the Variables pane, but this doesn't appear to apply to R). Could we consider not hard-coding any category names and letting each language pack provide all relevant categories (and then we'd perhaps just sort the categories alphabetically in the Variables pane)?

• if we decide against that approach, can we alternatively hard-code all possible categories in the front-end, with the idea that perhaps some categories may only be used by certain language packs? This wouldn't require changing any code and is already possible today, but is a shift from our current approach where we've just assumed common categories even though that is not enforced in any way. (for example, PR #2088, Python PR # 320)

2. How do we want to represent packages? We have some current issues around displaying packages in the Variables pane

• Python: Support module objects in the Variables pane #1788
• Enhanced Environments Management pane #1391
• Provide a way to see loaded packages (R) and modules (Python) #1369
• ARK: Show loaded R packages in the Variables view as a new kind #1827

But exactly how we do so, across language packs, requires a bit of nuance. For example, are users interested in seeing all installed packages with versions in their current interpreter/runtime (and would this be displayed in a new pane yet to be created)? Or do users just want to see what has been imported/attached? (I think there's probably more value in the former, but there could be a case made that users might be interested in both under different circumstances). If there is value in surfacing what is imported/attached, this works a little differently in R vs Python -- in R a package as a whole is either attached or not, and can be referenced (with explicit namespacing) whether or not it is attached. In Python, a module, or parts of a module, must be imported before being referenced, and a user may import the whole module, specific submodules from the module, or just specific functions/classes/etc from a module. These imports are also commonly aliased in Python (e.g. import pandas as pd; import matplotlib.pyplot as plt). Should these imports and aliases be displayed in the Variables pane, even if all installed modules and version info are displayed elsewhere (e.g. in a new Packages pane?)

3. Similar to the above, how do we handle differences in behavior across language packs, that may require language-specific terminology in the frontend? For example, "Delete all objects" behaves differently in R vs Python. In R, all objects/variables get cleared from the global environment, but any packages that have been attached remain attached -- the only way to reset and remove all packages is to restart the kernel. In Python, however, this button also removes all imports -- any modules, submodules, classes, or functions that were previously imported need to be re-imported in order to be referenced. Do we force all languages to behave in exactly the same way here (which may require digging to find commands that are perfectly comparable in order to implement this behavior in all languages), or is it possible for language packs to also contribute language that provides the tooltip language for the Delete button as well as the language for the confirmation modal that pops up when a user hits the delete button? If the behavior of the languages differ, then the user-facing language should make clear to the user what is about to happen.

There may be some other issues that come up we can link here, but in general, we need to decide on some approaches for handling differences in terminology, philosophy, and behavior between language packs that affect the Variables pane

[juliasilge]

Thank you for writing out some of these big picture questions around the Variables pane. 👍

I have a response on question 2 about representing packages. At least from my perspective/background, it would feel pretty strange to represent packages/modules (plus whether they are attached/imported and versions) in the Variables pane. IMO we should not try to squeeze those into the Variables pane but should plan for a separate Packages pane where we can find this info.

[jmcphers]

Could we consider not hard-coding any category names and letting each language pack provide all relevant categories (and then we'd perhaps just sort the categories alphabetically in the Variables pane)?
...
can we alternatively hard-code all possible categories in the front-end

I think having a standard set of categories is valuable. For example, "Data" usually wants to be front and center, and may have other special treatment. If the front end doesn't know anything about the categories, we have to treat them all the same.

What do folks think of a hybrid approach, wherein Positron ships with some standard categories for common types (data, functions, etc.), but allows language packs to contribute additional categories for objects that don't fit in the standard ones?

[juliasilge]

Would something like "Classes" be better in the standard set along with "Data" and "Functions", or an example of something worth building out a provider/contribution point for? It's the only one that has really come up so far AFAIK.

[jgutman]

I have a response on question 2 about representing packages. At least from my perspective/background, it would feel pretty strange to represent packages/modules (plus whether they are attached/imported and versions) in the Variables pane. IMO we should not try to squeeze those into the Variables pane but should plan for a separate Packages pane where we can find this info.

I definitely agree with this for R, absolutely. For Python, I'm of 2 minds.

1. I think no matter what we need a separate Packages pane for Python (just as we do for R) which will show all installed packages in the environment and their version numbers. This would basically be the same for R and Python
2. I'm less sure about this, but potentially I could see that for Python, in addition to the packages pane, we might potentially include in the variables pane imported modules and submodules in their own category. This could be a category configurable by user setting to be visible or not visible as was suggested. This could potentially address certain oddities that I think right now only apply to Python

• for Python, you can import a module, submodule, a class, function, constant, and probably even other things. If you import a class, function, or constant, it shows up in the Variables pane. If you import a module, or submodule, it does not. So for example:

import sklearn # doesn't appear in Variables
from sklearn import preprocessing # doesn't appear in Variables
from sklearn.ensemble import RandomForestClassifier # does appear in Variables
from sklearn.feature_selection import f_classif # does appear in Variables

import numpy as np # doesn't appear in Variables
import numpy.random as rand # doesn't appear in Variables
from numpy import NZERO # does appear in Variables
from numpy import ndarray # does appear in Variables

This feels somewhat inconsistent to me, and is a consideration that wouldn't really apply in R

• for Python, hitting the Delete All Objects button clears away all of the above imports. If we decide to retain that behavior, it's a bit odd to clear away something that is never displayed in the pane. I think it would be more consistent to either 1) not clear away the imports or 2) display the imports in the pane, and clear them away when the user requests. I probably prefer the latter, but it's just more intuitive that the action will clear away those imports if they appear in that pane to begin with
• again, for Python we can also alias our imports. So for example, it's typical to do

import numpy as np

x = np.ndarray((2,2))
print(np)

which functionally is quite similar to

import numpy
np = numpy

x = np.ndarray((2,2))
print(np)

In either case, np will not show up in the Variables pane, despite it being a real thing in the user_ns which I can reference or print to the console or call methods on.

Are these arguments enough reason to show modules and submodules for Python in both places? I'm not 100% sure, but I do think that they require some consideration.

[isabelizimm]

I am in support of having core categories, and then allowing the language packs to add more specialized ones. At the very least, I do like the idea of having Classes as a separate category.

in addition to the packages pane, we might potentially include in the variables pane imported modules and submodules in their own category. This could be a category configurable by user setting to be visible or not visible as was suggested

A packages pane would show what could be imported (listed by name used to pip install, not necessarily import), package versions, etc. I perceive the Variables pane to be used more as "what is in the namespace that can be operated on," in which case it feels like it should not omit imported modules. I also think this would ease the confusion as to what is being cleared when you clear the variables pane. However, showing modules means they would have to conform with other views of the Variables pane, where things like "size" don't make as much sense. I don't know what it would look like to just never integrate modules into that view.

editing to add: IF we were to decide to show modules, I would be against separating modules/submodules as I think that is too many categories and the distinction isn't any more useful than a single category that is displayed in an isort-y way

Side note: I don't know if we need to be able to configure the visibility, as long as you can collapse each category?