Support for WebExtensions

[alexristich]

Hi,

I'm a developer working on Privacy Badger, and we've been looking at using OpenWPM to begin running some tests as we look at improving the heuristic detection algorithm. We've made the switch to WebExtensions (https://developer.mozilla.org/en-US/Add-ons/WebExtensions) for our Firefox extension, which should be live within the next week. Today I went ahead and attempted to run an initial test but found that OpenWPM doesn't seem to support WebExtensions:

raise AddonFormatError(str(e), sys.exc_info()[2])
AddonFormatError: ("[Errno 2] No such file or directory: '/tmp/tmp9uEcBI.privacy-badger-eff-latest.xpi/install.rdf'", <traceback object at 0x7f33c2be0290>)

Do you by chance have plans to support WebExtensions-based extensions in the near future?

[englehardt]

Hello,

Glad to hear your considering OpenWPM and thanks for the great work on Privacy Badger! OpenWPM's support for WebExtensions is dependent on the versions of Firefox and Selenium used within the platform.

We currently use version 45 of Firefox, which has "alpha" support for WebExtensions (see: https://blog.mozilla.org/addons/2015/12/21/webextensions-in-firefox-45-2/). What's the minimum required Firefox version for Privacy Badger? Upgrading Firefox will likely require a switch to geckodriver, see #93.

The error you're receiving sounds similar to this: SeleniumHQ/selenium#1181. The workaround given in that issue might work for you while OpenWPM is still dependent on FirefoxDriver. Once we switch over to geckodriver, I imagine WebExtensions will be supported natively.

[alexristich]

Good to know! Firefox 48 is the target, but I'll give that workaround a try and let you know how it goes.

[tommybananas]

Hey @alexristich,
My group is working on porting the Firefox extension based tools to WebExtensions, so any tips you have for getting Selenium/Firefox to work with your extension would be much appreciated!

[englehardt]

@tommybananas Great to hear! We'll eventually have to move to WebExtensions once the addon-sdk is removed from Firefox. We're avoiding it at the moment because of the context provided by XPCOM interfaces. Please keep us updated with your efforts -- it will be great to have a better idea of what's supported in WebExtensions.

In FF 48+ WebExtensions is officially "stable", and fixing #93 will allow us to support 48+.

[englehardt]

My interpretation of #1181 posted above is as follows:

You need to create an extensions directory within the Firefox profile directory and copy your extension's xpi into that directory. You must do this prior to launching the browser with Selenium, as Firefox will load the extensions from that directory during startup.

In OpenWPM, we create the Firefox profile directory through selenium here. The browser_profile_path variable holds the absolute path to the temporary directory the Selenium/Firefox instance uses for that browser's profile. We launch the Firefox instance later in that same method here, so you'll need to create the extensions directory and copy your xpi in after the profile is created but before initializing the browser.

I recommend creating a new default browser configuration parameter here by adding webextensions: [], to the current file. In your crawl script you can then add the full path of WebExtensions-based extension you want to load to that list. For example in demo.py you could do the following:

# Update browser configuration (use this for per-browser settings)
for i in xrange(NUM_BROWSERS):
    browser_params[i]['http_instrument'] = True # Record HTTP Requests and Responses
    browser_params[i]['disable_flash'] = False #Enable flash for all three browsers
    browser_params[i]['webextensions'] = ['/home/example/full/path/to/extension.xpi'] # Load WebExtensions
browser_params[0]['headless'] = True #Launch only browser 0 headless

Then in deploy_firefox.py you can add the following:

if len(browser_params['webextensions']) > 0:
    extension_dir = os.path.join(browser_profile_path,'extensions')
    os.makedirs(extension_dir)
    for path in browser_params['webextensions']:
        logger.debug("BROWSER %i: Copying WebExtension %s to %s" % (browser_params['crawl_id'], path, extension_dir))
        shutil.copyfile(path, extension_dir)

I would add this immediately prior to launching the webdriver. Note that I didn't test this so I'm not sure if it will work. I hope this helps!

[alexristich]

Thanks! Sorry, been swamped this week, but I will get on this first thing Monday morning and see if I can get things working. I'll be in touch soon! Super appreciate your detailed help.

[gunesacar]

This blog post says extensions other than WebExtensions will not work after November 2017:
https://blog.mozilla.org/addons/2017/01/20/migrating-to-webextensions-port-your-stored-data/

[alexristich]

Took me about forever, but I finally had a chance to give this a shot. Doesn't seem to be working though. When running it with the suggested modifications I get an error on the first iteration saying Crash in driver, restarting browser manager with the corresponding error: IOError: [Errno 21] Is a directory: ///path/to/extension_dir///.
On the second and subsequent iterations the error is: OSError: [Errno 17] File exists: ///path/to/extension_dir///

I tried fiddling around for a bit but wasn't successful in fixing it. Thought I'd stop by to see if you might have any ideas on what would be causing this.

[englehardt]

This sounds like an issue while copying the XPI, some type of directory / file mismatch. Would you mind to share your changes? Either as a fork or a patch file from git?

[alexristich]

Sure! Thanks for the quick reply by the way :) It took me a while to remember that I had to generate a new Personal Access Token for GitHub to be able to push from another machine using my 2FA enabled account, but after a brief refresher I've managed to push it to a forked repo here: https://github.com/alexristich/OpenWPM

[englehardt]

Sorry I haven't had time to look through your code. I suggest checking out branch in #143. Firefox 52 is supported, so you may have more luck getting that to work than struggling with the workaround I suggested.

[alexristich]

Awesome, thanks for the heads up! Will hopefully have time to fiddle around with this in the next week or two. I'll follow up there if I encounter any issues!