recursive_folder_or_rp_moid_search awaits objects_moid to be in order (but it isn't necessarily)

[dbk-rabel]

Hi.

We stumbled across this, because sometimes folder lookups worked fine and sometimes they didn't.

While troubleshooting we noticed that recursive_folder_or_rp_moid_search awaits objects_moid (which it gets from the API) to be in order of the path.
If they're not, https://github.com/ansible-collections/vmware.vmware_rest/blob/main/plugins/plugin_utils/lookup.py#L472 will never match.

But the API does not always send them in order.

We have a lookup that looks like this:

lookup('vmware.vmware_rest.folder_moid', '/my_datacenter/my_topfolder/my_subfolder', **connection_args)

In get_all_objects_path_moid there is an API call to a ressource like this:

https:///api/vcenter/folder?datacenters=my_datacenter_moid&names=my_subfolder&names=my_topfolder

And it receives this from the API:

[{'folder': 'my_subfolder_moid', 'name': 'my_subfolder', 'type': 'VIRTUAL_MACHINE'}, {'folder': 'my_topfolder_moid', 'name': 'my_top_folder', 'type': 'VIRTUAL_MACHINE'}]

This only happens for folders that we've created in the last days. Not sure if this is the reason, why the API behaves like that.

But since the API is not consistent with the ordering, I think we can say that recursive_folder_or_rp_moid_search is broken right now.

Yours
David

[dbk-rabel]

Could this maybe already have been fixed by #445 ?

Edit: Just copy-pasting the code from the PR does not seem to work. It runs into an ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure

[reusch]

Maybe this is also the same as #324

[mikemorency]

hi @dbk-rabel im having some trouble reproducing the issue. Im also experiencing the EmbeddedModuleUnexpectedFailure when I try to use these lookups.

Could you provide the version of the collection your using, the version of ansible packages (ansible and/or ansible-core), and your vcenter version? Also maybe a playbook to reproduce the issue?

I dont see any reason the playbook in 445 wouldnt work for you but other playbooks would, so thats interesting. If you have any insight into why that might be happening that would also be helpful

[dbk-rabel]

Hello @mikemorency .

We are using vmware.vmware_rest 3.0.1, ansible-core 2.15.11
vSphere Client version 8.0.2.00200

Here is an example playbook, but of course you would need to create the folders in vcenter first:

---
- name: Test
  hosts: localhost
  gather_facts: false
  connection: local
  vars:
    directories:
      - folder/subfolder1
      - folder/subfolder2
  tasks:
    - name: Test
      ansible.builtin.debug:
        msg: "{{ item }} has MOID {{ lookup('vmware.vmware_rest.folder_moid', '/                                                                                                                                                             my_datacenter/' + item, **connection_args) }}"
      loop: "{{ directories }}"

As I said: It works if and only if the API returns the moids of the folders in the correct order. This is not guaranteed. We did not find out when or why it does so, but since it fails especially with newer folders, it might have something todo with the date when the folders where created.

[mikemorency]

Hi @dbk-rabel , thank you for your patience. I just wanted to let you know that I have a PR with a fix for the lookup plugins. It would be great if you could test it before we release in a few weeks, just to be extra sure your issue is solved.

To install my branch:

ansible-galaxy collection install 'git@github.com:mikemorency/vmware.vmware_rest.git,mm-bugfix/inconsistent-lookup-results'

Please note the modules on my branch are from release 4.0.0 and are based on vSphere 8. The lookup plugins should work on either 7 or 8

[dbk-rabel]

Hi @mikemorency .
With the code from the PR I get the error

fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'vmware.vmware_rest.folder_moid'. Error was a <class 'ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure'>, original message: module 'asyncio' has no attribute 'create_task'"}

[mikemorency]

Thanks for testing! Can you confirm the version of Python your using on the Ansible control node?

create_task was added in python 3.7, so I could see you getting that error using Python 3.6. But I believe the version of Ansible core you mentioned using (2.15) only supports python 3.9 and above on the control node

[dbk-rabel]

I tried to use Python3.9 in my virtualenv, but somehow I seem to keep failing:

fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'vmware.vmware_rest.folder_moid'. Error was a <class 'ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure'>, original message: Unable to connect to vCenter or ESXi API at <censored>: Failed to import the required Python library (aiohttp) on <censored>'s Python /usr/bin/python3.9. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter. Unable to connect to vCenter or ESXi API at <censored>: Failed to import the required Python library (aiohttp) on <censored>'s Python /usr/bin/python3.9. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

~/misc $ ansible --version
ansible [core 2.15.12]
  config file = ~/misc/ansible.cfg
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = ~/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.19 (main, Aug 26 2024, 11:01:47) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python3.9)
  jinja version = 3.1.4
  libyaml = True

~/misc $ python3.9 -m pip list
Package          Version
---------------- -------
aiohappyeyeballs 2.4.0
aiohttp          3.10.5
aiosignal        1.3.1
async-timeout    4.0.3
attrs            24.2.0
frozenlist       1.4.1
idna             3.8
multidict        6.0.5
pip              24.2
setuptools       50.3.2
yarl             1.11.0

I tried to add -e 'ansible_python_interpreter=~/misc/.venv39/bin/python3.9' , but ansible seems to keep ignoring that.

Yours
David

[dbk-rabel]

@mikemorency Since you already closed this, I guess you were able to test that it fixes this issue?

[mikemorency]

The PR I had opened was merged, and some automation closed the issue. Ive re-opened it.

Thats weird, but Im able to see the same results as you (python interpreter not taking effect). Let me see if I can find out why

[mikemorency]

OK I think I understand the issue. The ansible_python_interpreter variable only sets the python executable on remote hosts. Since the lookup plugins are executed on the controller, the version of python used depends on whatever python the ansible executable is installed under on the controller.

So you would need to activate your virtual env before running the playbook, or install aiohttp using the system python. Let me know if you have any questions or issues

Heres a playbook showing the difference:

(ansible-vmware-3.10) ~ $ ansible-playbook test.yml -e ansible_python_interpreter="~/miniconda3/envs/test/bin/python"

PLAY [localhost] *********************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************
ok: [mikemorency]

TASK [debug] *************************************************************************************************************************************************
ok: [mikemorency] => {
    "ansible_python_interpreter": "~/miniconda3/envs/test/bin/python"
}

TASK [debug] *************************************************************************************************************************************************
ok: [mikemorency] => {
    "ansible_facts.python.executable": "/home/mikemorency/miniconda3/envs/test/bin/python"
}

TASK [output which python using 'lines' lookup] **************************************************************************************************************
ok: [mikemorency] => {
    "msg": "/home/mikemorency/miniconda3/envs/ansible-vmware-3.10/bin/python"
}

PLAY RECAP ***************************************************************************************************************************************************
mikemorency : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

For reference:
https://docs.ansible.com/ansible/latest/reference_appendices/config.html#interpreter-python

[dbk-rabel]

Ok. I kinda understand the issue. I installed aiohttp using the system python and then it worked.

And yes, it does seem to fix the issue! :-)

At least for my testcase: Cannot find folder moid with upstream version. Can find folder moid with your fix.

Thanks a lot!
David

[mikemorency]

Excellent! The fix will be included in the next minor version release for version 4 and 3 of this collection. It should be available around September 19th.

Im going to close the issue, but feel free to open a new one in the future or let me know if you think it should be re-opened.