Search endpoint is slow

[augustuswm]

Needs investigation.

Calling the search endpoint takes multiple seconds, whereas it should be around 300ms.

There are likely multiple issues affecting this, but to start with using time + curl across a few envs:

Local machine (Minnesota)

real	0m0.239s
user	0m0.082s
sys	0m0.000s

AWS VM (us-west-2)

real	0m0.231s
user	0m0.052s
sys	0m0.001s

Rack 3 VM - largely variable results

real	0m1.412s
user	0m0.067s
sys	0m0.008s

real	0m0.882s
user	0m0.056s
sys	0m0.019s

real	0m0.430s
user	0m0.066s
sys	0m0.008s

real	0m1.600s
user	0m0.070s
sys	0m0.005s

[augustuswm]

A large portion of the slowdown is related to the existing client re-establishing a new client and connection for each request. The large outliers though that I have still not tracked down.

[augustuswm]

We have seen similar behavior with requests for static route resources which suggests that the outliers are not rfd-api or vm specific. It is possible that we have some kind of bottleneck with our network appliances that sit in front of rack 3.

[augustuswm]

Independent of the above results, search responses return excess data and should be trimmed down to make sure we are only querying from our backed the exact necessary data.