Can not manage equinix_metal_vlans in v0.7.0

[displague]

What happened?

After following the guides on following https://docs.crossplane.io/latest/software/install/ and https://deploy.equinix.com/labs/crossplane/, which install and configuring Crossplane and this Equinix provider, I installed a VLAN:

apiVersion: metal.equinix.jet.crossplane.io/v1alpha1
kind: Vlan
metadata:
  name: upjet-example
spec:
  forProvider:
    projectId: "<redacted>"
    vxlan: 1000
    metro: sv

This was successfully installed and all aspects of management were working.

$ kubectl get vlans.metal.equinix.jet.crossplane.io  -o wide
NAME            READY   SYNCED   EXTERNAL-NAME                          AGE
upjet-example   True    True     3fdebf48-4838-4c7f-a1b3-893ac8884866   5m15s

Upon updating to 0.7.0, the previously created VLAN resource lost sync status:

    - lastTransitionTime: "2024-06-28T12:28:24Z"
      message: 'observe failed: cannot run plan: plan failed: Instance cannot be destroyed:
        Resource equinix_metal_vlan.upjet-example has lifecycle.prevent_destroy set,
        but the plan calls for this resource to be destroyed. To avoid this error
        and continue with the plan, either disable lifecycle.prevent_destroy or reduce
        the scope of the plan using the -target flag.'
      reason: ReconcileError
      status: "False"
      type: Synced

I was able to delete the resource, and it was successfully removed from Equinix Metal.

I could not recreate the resource, receiving the same error you received:

    - lastTransitionTime: "2024-06-28T12:31:46Z"
      message: 'observe failed: cannot run refresh: refresh failed: Error fetching
        Vlan using vlanId:  is not a valid UUID'
      reason: ReconcileError
      status: "False"
      type: Synced

How can we reproduce it?

Replay the steps above. Install v0.7.0 and attempt to create a VLAN.

What environment did it happen in?

Crossplane version:
Provider version:

$ uname -a 
Linux dev-qemu 6.2.0-39-generic #40-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 23:07:44 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
$ kind version
kind v0.20.0 go1.20.3 linux/arm64
$ kubectl version 
Client Version: v1.28.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3
$ helm version
version.BuildInfo{Version:"v3.14.2", GitCommit:"c309b6f0ff63856811846ce18f3bdc93d2b4d54b", GitTreeState:"clean", GoVersion:"go1.21.7"}
$ helm list -A
NAME      	NAMESPACE        	REVISION	UPDATED                               	STATUS  	CHART            	APP VERSION
crossplane	crossplane-system	1       	2024-06-28 11:49:30.66116225 +0000 UTC	deployed	crossplane-1.16.0	1.16.0     

[displague]

The same is not a valid UUID error is being returned for new Organization resources. This error is built into packngo and is returned when a request to Get() a resource by ID included a string that was not a valid UUID. I have observed in v0.9.0 that this error is being returned for VLAN and Organization resources before a Create API request is attempted. If the UUID was part of a secondary request during Create, a different error would be returned by terraform-provider-equinix saying that the resource could not be fetched after creation.

crossplane-p… │ 2024-07-28T21:34:43-04:00 DEBUG provider-jet-equinix Cannot observe external resource {"controller": "managed/metal.equinix.jet.crossplane.io/v1alpha1, kind=organization", "request": {"name":"tf-organization-1"}, "uid": "cc00d9d8-91ac-4a28-9b03-1ab1dfa67523", "version": "434279", "external-name": "", "error": "cannot run refresh: refresh failed: Error updating resource: Could not read Metal Organization with ID : is not a valid UUID", "errorVerbose": "refresh failed: Error updating resource: Could not read Metal Organization with ID : is not a valid UUID\ncannot run refresh\ngithub.com/crossplane/upjet/pkg/controller.(*external).Observe\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/controller/external.go:215\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/reconciler/managed/reconciler.go:914\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1695"}

crossplane-p… │ 2024-07-28T21:34:43-04:00 DEBUG events cannot run refresh: refresh failed: Error updating resource: Could not read Metal Organization with ID : is not a valid UUID{"type": "Warning", "object": {"kind":"Organization","name":"tf-organization-1","uid":"cc00d9d8-91ac-4a28-9b03-1ab1dfa67523","apiVersion":"metal.equinix.jet.crossplane.io/v1alpha1","resourceVersion":"434279"}, "reason": "CannotObserveExternalResource"}

[displague]

On a Project resource, which does not use packngo, we observe a different error:

crossplane-p… │ 2024-07-28T21:51:44-04:00	DEBUG	provider-jet-equinix	Calling the inner handler for Create event.	{"gvk": "metal.equinix.jet.crossplane.io/v1alpha1, Kind=Project", "name": "upjet-example-project", "queueLength": 0}
crossplane-p… │ 2024-07-28T21:51:44-04:00	DEBUG	provider-jet-equinix	Reconciling	{"controller": "managed/metal.equinix.jet.crossplane.io/v1alpha1, kind=project", "request": {"name":"upjet-example-project"}}
crossplane-p… │ 2024-07-28T21:51:44-04:00	DEBUG	provider-jet-equinix	Reconciling	{"controller": "providerconfig/providerconfig.equinix.jet.crossplane.io", "request": {"name":"default"}}
crossplane-p… │ 2024-07-28T21:51:44-04:00	DEBUG	provider-jet-equinix	Running terraform	{"workspace": "/tmp/86feedb9-9a17-4162-9b32-3de074a8230e", "args": ["init", "-input=false"]}
crossplane-p… │ 2024-07-28T21:51:44-04:00	DEBUG	provider-jet-equinix	Reconciling	{"controller": "providerconfig/providerconfig.equinix.jet.crossplane.io", "request": {"name":"default"}}
crossplane-p… │ 2024-07-28T21:51:46-04:00	DEBUG	provider-jet-equinix	init ended	{"workspace": "/tmp/86feedb9-9a17-4162-9b32-3de074a8230e", "out": "\n\u001b[0m\u001b[1mInitializing the backend...\u001b[0m\n\n\u001b[0m\u001b[1mInitializing provider plugins...\u001b[0m\n- Finding equinix/equinix versions matching \"2.2.0\"...\n- Installing equinix/equinix v2.2.0...\n- Installed equinix/equinix v2.2.0 (signed by a HashiCorp partner, key ID \u001b[0m\u001b[1m1A65631C7288685E\u001b[0m\u001b[0m)\n\nPartner and community providers are signed by their developers.\nIf you'd like to know more about provider signing, you can read about it here:\nhttps://www.terraform.io/docs/cli/plugins/signing.html\n\nTerraform has created a lock file \u001b[1m.terraform.lock.hcl\u001b[0m to record the provider\nselections it made above. Include this file in your version control repository\nso that Terraform can guarantee to make the same selections by default when\nyou run \"terraform init\" in the future.\u001b[0m\n\n\u001b[0m\u001b[1m\u001b[32mTerraform has been successfully initialized!\u001b[0m\u001b[32m\u001b[0m\n\u001b[0m\u001b[32m\nYou may now begin working with Terraform. Try running \"terraform plan\" to see\nany changes that are required for your infrastructure. All Terraform commands\nshould now work.\n\nIf you ever set or change modules or backend configuration for Terraform,\nrerun this command to reinitialize your working directory. If you forget, other\ncommands will detect it and remind you to do so if necessary.\u001b[0m\n"}
crossplane-p… │ 2024-07-28T21:51:46-04:00	DEBUG	provider-jet-equinix	Running terraform	{"workspace": "/tmp/86feedb9-9a17-4162-9b32-3de074a8230e", "args": ["apply", "-refresh-only", "-auto-approve", "-input=false", "-lock=false", "-json"]}
crossplane-p… │ 2024-07-28T21:51:49-04:00	DEBUG	provider-jet-equinix	refresh ended	{"workspace": "/tmp/86feedb9-9a17-4162-9b32-3de074a8230e", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.8.2\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2024-07-28T21:51:46.860811-04:00\",\"terraform\":\"1.8.2\",\"type\":\"version\",\"ui\":\"1.2\"}\n{\"@level\":\"info\",\"@message\":\"equinix_metal_project.upjet-example-project: Refreshing state...\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2024-07-28T21:51:47.485657-04:00\",\"hook\":{\"resource\":{\"addr\":\"equinix_metal_project.upjet-example-project\",\"module\":\"\",\"resource\":\"equinix_metal_project.upjet-example-project\",\"implied_provider\":\"equinix\",\"resource_type\":\"equinix_metal_project\",\"resource_name\":\"upjet-example-project\",\"resource_key\":null},\"id_key\":\"id\"},\"type\":\"refresh_start\"}\n{\"@level\":\"info\",\"@message\":\"equinix_metal_project.upjet-example-project: Refresh complete\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2024-07-28T21:51:49.061690-04:00\",\"hook\":{\"resource\":{\"addr\":\"equinix_metal_project.upjet-example-project\",\"module\":\"\",\"resource\":\"equinix_metal_project.upjet-example-project\",\"implied_provider\":\"equinix\",\"resource_type\":\"equinix_metal_project\",\"resource_name\":\"upjet-example-project\",\"resource_key\":null},\"id_key\":\"id\"},\"type\":\"refresh_complete\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2024-07-28T21:51:49.069492-04:00\",\"changes\":{\"add\":0,\"change\":0,\"import\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 0 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2024-07-28T21:51:49.102880-04:00\",\"changes\":{\"add\":0,\"change\":0,\"import\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2024-07-28T21:51:49.102999-04:00\",\"outputs\":{},\"type\":\"outputs\"}\n"}
crossplane-p… │ 2024-07-28T21:51:49-04:00	DEBUG	provider-jet-equinix	Cannot observe external resource	{"controller": "managed/metal.equinix.jet.crossplane.io/v1alpha1, kind=project", "request": {"name":"upjet-example-project"}, "uid": "86feedb9-9a17-4162-9b32-3de074a8230e", "version": "436692", "external-name": "", "error": "cannot set critical annotations: cannot get external name: cannot find id in tfstate", "errorVerbose": "cannot find id in tfstate\ngithub.com/crossplane/upjet/pkg/config.init.func3\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/config/resource.go:103\ngithub.com/crossplane/upjet/pkg/resource.SetCriticalAnnotations\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/resource/lateinit.go:56\ngithub.com/crossplane/upjet/pkg/controller.(*external).Observe\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/controller/external.go:258\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/reconciler/managed/reconciler.go:914\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1695\ncannot get external name\ngithub.com/crossplane/upjet/pkg/resource.SetCriticalAnnotations\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/resource/lateinit.go:58\ngithub.com/crossplane/upjet/pkg/controller.(*external).Observe\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/controller/external.go:258\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/reconciler/managed/reconciler.go:914\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1695\ncannot set critical annotations\ngithub.com/crossplane/upjet/pkg/controller.(*external).Observe\n\tgithub.com/crossplane/upjet@v1.4.0/pkg/controller/external.go:260\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/reconciler/managed/reconciler.go:914\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1695"}
crossplane-p… │ 2024-07-28T21:51:49-04:00	DEBUG	events	cannot set critical annotations: cannot get external name: cannot find id in tfstate	{"type": "Warning", "object": {"kind":"Project","name":"upjet-example-project","uid":"86feedb9-9a17-4162-9b32-3de074a8230e","apiVersion":"metal.equinix.jet.crossplane.io/v1alpha1","resourceVersion":"436692"}, "reason": "CannotObserveExternalResource"}