"Error in if (is_http) { : argument is of length zero" in shinytest

[bersbersbers]

I am shinytesting a Shiny app, and while each test runs fine on its own, one specific test fails when run in a long sequence of some 20 other tests.

The error message is

Error in if (is_http) { : argument is of length zero

and is related to this:

httr/R/request.R

Lines 159 to 167 in 2cd5ed8

	url_scheme <- parse_url(resp$url)$scheme
	is_http <- tolower(url_scheme) %in% c("http", "https")
	if (is_http) {
	all_headers <- parse_http_headers(resp$headers)
	headers <- last(all_headers)$headers
	} else {
	all_headers <- NULL
	headers <- resp$headers
	}

I don't have a reprex yet, but I could investigate that url_scheme is NULL because resp$url is empty (I cannot yet say if it's character(0) or "", though, as they look the same in the one stop() output that I have). Interestingly, req$url looks very resonable:

http://127.0.0.1:8421/session/670db570-3e20-11ec-8cde-69c24c6e7fa3/screenshot

Here is some more output from

warning("url_scheme is NULL for request URL '", req$url, "' with response URL '", resp$url, "'. Request is ", paste(req), " and response is ", paste(resp))

url_scheme is NULL for request URL 'http://127.0.0.1:8729/session/d0d9d960-3e26-11ec-9ef8-7b66fe04ffba/screenshot' with response URL ''. Request is GEThttp://127.0.0.1:8729/session/d0d9d960-3e26-11ec-9ef8-7b66fe04ffba/screenshotc(Accept = "application/json", `Content-Type` = "application/json", `User-Agent` = "R webdriver")NULLlist(useragent = "libcurl/7.72.0 r-curl/4.3.2 httr/1.4.2", httpget = TRUE)NULLlist() and response is 0NAraw(0)NAc(redirect = 0, namelookup = 0, connect = 0, pretransfer = 0, starttransfer = 0, total = 0)as.raw(c(0x7b, 0x22, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x22, 0x3a, 0x22, 0x64, 0x30, 0x64, 0x39, 0x64, 0x39, 0x36, 0x30, 0x2d, 0x33, 0x65, 0x32, 0x36, 0x2d, 0x31, 0x31, 0x65, 0x63, 0x2d, 0x39, 0x65, 0x66, 0x38, 0x2d, 0x37, 0x62, 0x36, 0x36, 0x66, 0x65, 0x30, 0x34, 0x66, 0x66, 0x62, 0x61, 0x22, 0x2c, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x3a, 0x30, 0x2c, 0x22, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x3a, 0x22, 0x69, 0x56, 0x42, 

> intToUtf8(c(0x7b, 0x22, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x22, 0x3a, 0x22, 0x64, 0x30, 0x64, 0x39, 0x64, 0x39, 0x36, 0x30, 0x2d, 0x33, 0x65, 0x32, 0x36, 0x2d, 0x31, 0x31, 0x65, 0x63, 0x2d, 0x39, 0x65, 0x66, 0x38, 0x2d, 0x37, 0x62, 0x36, 0x36, 0x66, 0x65, 0x30, 0x34, 0x66, 0x66, 0x62, 0x61, 0x22, 0x2c, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x3a, 0x30, 0x2c, 0x22, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x3a, 0x22, 0x69, 0x56, 0x42))
[1] "{\"sessionId\":\"d0d9d960-3e26-11ec-9ef8-7b66fe04ffba\",\"status\":0,\"value\":\"iVB"
> 

So I while I am investigating while this request may be failing, I wonder if resp$url should ever be empty if req$url is a valid URL, and if so, if httr should handle that. I am aware of #620, but I feel this is a rather symptomatic fix (and not released yet, anyway).

[bersbersbers]

I am aware of #620, but I feel this is a rather symptomatic fix (and not released yet, anyway).

And I have tested it: #620 leads to

$ operator is invalid for atomic vectors>

probably because resp$headers and thus all_headers are NULL and either last(all_headers)$headers or headers$date fails.

By contrast, what solved the issue for me (including getting a correct screenshot saved, to my amazement) is [I cannot reproduce this]

  if (is.null(url_scheme)) {
    url_scheme <- ""
  }

In summary, I think httr should handle more NULL fields in req, and #620 does not cover all cases.

[bersbersbers]

Request:

structure(list(method = "GET", url = "http://127.0.0.1:7172/session/48203110-3e39-11ec-9f63-eb4efda64e22/screenshot", 
    headers = c(Accept = "application/json", `Content-Type` = "application/json", 
    `User-Agent` = "R webdriver"), fields = NULL, options = list(
        useragent = "libcurl/7.72.0 r-curl/4.3.2 httr/1.4.2", 
        httpget = TRUE), auth_token = NULL, output = structure(list(), class = c("write_memory", 
    "write_function"))), class = "request")

Response:

list(url = "", status_code = 0L, type = NA_character_, headers = raw(0), 
    modified = structure(NA_integer_, class = c("POSIXct", "POSIXt"
    )), times = c(redirect = 0, namelookup = 0, connect = 0, 
    pretransfer = 0, starttransfer = 0, total = 0), content = as.raw(c(0x7b, 
    0x22, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x49, 0x64, ...)))

And the content resolves to

{"sessionId":"48203110-3e39-11ec-9f63-eb4efda64e22","status":0,"value":"iVBORw..."}

where iVBORw is PNG and value is the complete (I believe) PNG in base64. At least value has length 250k, while the correct PNG file has 185k, very consistent with base64 overhead.

[bersbersbers]

I think the main problem that I am having with regard to this is that is does not reproduce consistently, so I cannot make any hard statements on whether #620 would solve this (I seem to remember it did not, but from looking at the code I am unable to say why it wouldn't) and/or whether

    url_scheme <- ""

works (I believe it should not, and require at least

    resp$headers <- NULL

as well). It may also be that my failing curl responses are a bit different from time to time.

At least, I think I have given enough information in #708 (comment) for you to see that if even the user does everything correct (i.e., does not forget a protocol or something), curl does return weird stuff sometimes which makes httr trip.

[hadley]

This sounds more like a bug in shinytest to me. Why are those fields NULL in the first place?