Upgrade OpenWPM to Firefox ESR 60

[englehardt]

Firefox 57+ will remove support for all non-WebExtension style extensions (which includes our extension, built using the add-on sdk). #143 will update us to Firefox 52 ESR, which is supported until June 2018. At that point, Firefox 59 will be the only available ESR version, which will no longer have WebExtensions support.

This means that we'll be stuck with FF 52 ESR until we've ported the extension. During my testing of #143 I also found that we'll be stuck with geckodriver v0.15 as geckodriver v0.16+ requires FF 53+. This is unfortunate as geckodriver is still under active development and older versions are likely to have missing features/bugs, though I haven't found any show-stoppers yet.

Doing a quick review it seems that all of the basic features we currently support (monitoring cookie access, requests/responses, and javascript calls) will still be possible. I think we will end up losing a bunch of the extra context available around HTTP requests (such as the call stack if the request was triggered in javascript, the type of element responsible for triggering the request, and so on). The javascript instrumentation will be unaffected as it doesn't use any native APIs. I haven't examined the cookie instrument yet. We will also likely need to completely rewrite the communication interface.

When it comes time to implement, this fork of OpenWPM might be a good starting point, though it appears to be abandoned.

[englehardt]

An alternative option may be to package the current add-on as a WebExtensions Experiment, which is aimed at allowing developers to access the same interfaces older add-ons could access to prototype new WebExtensions APIs. The linked site implies it will be limited to Nightly and Dev Edition, but it may end up that we can flip a pref in release (or maybe compile support into release). Whether it's worth going this route will depend on the amount of access we lose by moving to WebExtensions.

[englehardt]

More on the WebExtension Experiments approach:

The linked site implies it will be limited to Nightly and Dev Edition, but it may end up that we can flip a pref in release (or maybe compile support into release).

Reading the current docs it looks like the Nightly/Dev restriction for WebExtension Experiments is because of extension signing. As the docs state, unbranded builds of the release version of Firefox still allow unsigned extensions and ESR builds (which we use) will not enforce the signing requirement.

So it seems like the WebExtensions Experiment approach will allow us to continue to use the instrumentation with relatively little effort. A few notes:

• The extension will still require a bit of re-structuring. See https://github.com/web-ext-experiments for examples of experiment extensions. We'll have to ditch jpm.
• We should be able to load the extension the normal way after we package it into an .xpi. We just need to ensure xpinstall.signatures.required is set to False.
• It's unclear if the XPCOM APIs we rely on will remain stable now that other extensions won't be using them directly. I suspect the upgrade path of least resistance will be to keep everything that still works using the old APIs.

[englehardt]

ESR 52 is supported until August 21st, 2018, so it would be ideal to upgrade before then.

[englehardt]

It looks like bootstrap extensions will be removed in FF65 https://mail.mozilla.org/pipermail/dev-addons/2018-June/003831.html

[motin]

Quick summary of our discussion in June on this: The legacy add-ons SDK has been removed so we can't package the current add-on as a WebExtension Experiment without rewriting most of the code, and in doing so, it may be much easier to use the existing webRequest and cookies web extension API:s (like in https://github.com/groovecoder/data-leak).
We can still use WebExtension Experiments, but only where its necessary to use privileged code. Avoiding WebExtension Experiments altogether removes the need to sign the extension or run unbranded builds, allowing OpenWPM to stick with the official Firefox 60 ESR. Pros and cons everywhere :)

[englehardt]

Firefox 60 ESR does indeed still have the xpinstall.signatures.required option, which when set to False allows us to run unsigned add-ons. Thus we'll only need unbranded builds if we want to crawl with the current release version of Firefox.

[englehardt]

Ultimately we decided to move to WebExtensions using a couple experiment APIs to handle sockets and file IO. Now that the ff52/addon-sdk branch is over 400 commits behind the master branch, I think it's safe to close this. :)