Documentation: add recommendation that jvm-option and :SystemEmail options match to avoid spam filters

[donsizemore]

IQSS and Odum have each run into trouble with e-mail verification messages sent from Dataverse instances getting chomped by spam filters. In Odum's case, the token-èd link was getting stripped so end users only received "Please contact us if you did not intend this change or if you need assistance." Changing the mail-resource "from" field in domain.xml to match the value set for :SystemEmail seemed to help IQSS.

Though dealing with junk mail and mail filters is outside the scope of Dataverse's documentation, it might be helpful for the installation/configuration guide to note the above. Consider this self-assigned (pull request forthcoming; the links in Odum's outgoing messages are still getting stripped so Odum may have more to tweak).

[pdurbin]

In Odum's case, the token-èd link was getting stripped so end users only received "Please contact us if you did not intend this change or if you need assistance."

This reminds me a lot of #3407.

[donsizemore]

Can I claim that I hadn't read the entire title of #3407? I promise I did a search. Anyway, UNC has us on three Office365 whitelists (spam, spoof, fraud) but the link is still viewed as evil. My Gmail accounts receive the e-mail in its entirety. Still working with UNC to resolve this as Thu-Mai hopes to require e-mail verification as Dataverse makes more use of it.

[pdurbin]

Thu-Mai hopes to require e-mail verification as Dataverse makes more use of it

#3300 represents our plans to have email verification mean something. Maybe you aren't allowed to publish if your email hasn't been verified or something. Or maybe we simply don't send you notifications if you haven't verified your email. We can discuss what the consequences should be there.

@donsizemore you and @kcondon were working on email trouble in #4206 and I guess this issue #4210 is simply about documenting the fix. Do you feel like making a pull request? Here's what it says as of 4.8.1: https://github.com/IQSS/dataverse/blob/v4.8.1/doc/sphinx-guides/source/installation/config.rst#systememail

[donsizemore]

@pdurbin oh. whoops. so i duplicated #3407 and #4206. I'm still working with UNC/Microsoft on our end; if I learn anything more regarding Dataverse configuration I'll include it in the PR.

[djbrooke]

@donsizemore - thanks for taking a look at this!

[pdurbin]

mail-resource "from" field in domain.xml

Just to make this more concrete, here's what the "from" filed looks like on my test server:

grep @ /usr/local/glassfish4/glassfish/domains/domain1/config/domain.xml

<mail-resource host="localhost" from="do-not-reply@dev1.dataverse.org" user="dataversenotify" jndi-name="mail/notifyMailSession"></mail-resource>

It looks like "do-not-reply" comes from scripts/installer/glassfish-setup.sh:

./asadmin $ASADMIN_OPTS create-javamail-resource --mailhost "$SMTP_SERVER" --mailuser "dataversenotify" --fromaddress "do-not-reply@${HOST_ADDRESS}" mail/notifyMailSession

It sounds like the installer shouldn't put that "do-not-reply" address in there in the first place. According to http://guides.dataverse.org/en/4.8/installation/installation-main.html#running-the-dataverse-installer the installer asks for the "Administrator email address for this Dataverse" so I assume that is the value that should be put in both the JVM mail-resource and then database :SystemEmail. Someone please tell me if I'm confused. Thanks.

[donsizemore]

@pdurbin excellent point. Feeding the jvm-option the administrator e-mail address might only be a problem if said administrator's personal address were to become Shibbolized, but I imagine neither option's address would show up in the authenticateduser table (no other conflicts)?

[pdurbin]

@donsizemore part of the confusion here is the meaning of "Administrator email address for this Dataverse" from http://guides.dataverse.org/en/4.8/installation/installation-main.html#running-the-dataverse-installer and https://github.com/IQSS/dataverse/blob/v4.8.1/scripts/installer/install#L103

Which of the following are we talking about?

• The email address of the dataverseAdmin superuser account? (This is set in https://github.com/IQSS/dataverse/blob/v4.8.1/scripts/api/data/user-admin.json as described at http://guides.dataverse.org/en/4.8/installation/installation-main.html#superuser-account )
• The email address behind the "Support" link in the header of of an installation of Dataverse?

These are separate email addresses.

From a quick look at https://github.com/IQSS/dataverse/blob/v4.8.1/scripts/installer/install#L1278 it looks like

"Administrator email address for this Dataverse"

should be changed to

"System/Support email address for this Dataverse"

Otherwise, it's confusing.

I guess we could have the installer prompt for both email addresses but don't let me turn a small chunk into a big chunk. @donsizemore if you just want to make a small tweak to the docs to explain the situation, it would be incremental progress!

[landreev]

Joining the discussion a bit late, but a couple of things:

The "from=" parameter, in the "mail-resource" section of the domain.xml; that in turn is set by the installer to do-not-reply@${HOST_ADDRESS}:

It's not clear if we need this parameter at all. It is a leftover from some old days, when we specifically didn't want people to be able to reply to these email. Note that, currently, this parameter is NOT actually being used for the "From:" field of the email. Instead we are setting it in the Java code, in MailServiceBean.java:

msg.setFrom(systemAddress);

and systemAddress is the value of the systemEmail setting.

The parameter set in the domain.xml is not completely ignored - it appears that it ends up being used at least for generating the id of the email (just from looking at the raw headers of a Dataverse-sent email). But I don't think it has any effect on whether the email gets relayed and delivered, or rejected and/or assigned spam points by filters... So it may be safe to just get rid of the parameter completely (I'll test that). Once again, for all practical purposes it is the value of the systemEmail that's important - that's what the recipients of the email see, etc.

[landreev]

Kevin and I were talking about/experimenting with these email settings last week; some of the results we were seeing were somewhat counter-intuitive. It appears that it's more complicated, than just saying that the systemEmail setting must match the HOST_ADDRESS in the domain.xml. I mean, if the host name in systemEmail matches the address of your server, than yes, it should work - your email will be coming from where your "from:" header says, and all the relays and spam filters should be happy. But this straightforward case is not going to satisfy everybody. Some dataverse owners will always want to use outside mail servers. So the full answer that we document in the guide should say something like, if the systemEmail you want to use is not in the same domain as your server, it should be in a domain that your mail/smtp server (specified in the mail-resource section of the domain.xml) is configured to relay mail for.

For example: We run Dataverse on dataverse.org; we configure it to use "mail.hmdc.harvard.edu" as the smtp server. mail.hmdc.harvard.edu is the registered MX record for dataverse.org. So we should be able to use support@hmdc.harvard.edu or support@dataverse.org as our systemEmail, both should work.

Explaining it all in the guide clearly may be a bit tricky.

[pdurbin]

We discussed this issue in standup yesterday and decided to move it out of Development at https://waffle.io/IQSS/dataverse . @donsizemore you are very welcome to make a pull request and we'd move this issue straight into Code Review. Probably anything you add would be an improvement but we hope that all the discussion here hasn't confused the matter. Please let us know if you need anything. Thanks!

[donsizemore]

@pdurbin I'm still going back and forth with our campus mail admins =)

[pdurbin]

@donsizemore ok, no rush. At least we've started to figure out this mail delivery issue. Thanks.

[pdurbin]

@donsizemore we haven't talked about this issue in a while. Should it still be assigned to you? If not we can throw it back on the open issue heap. 😄

[mheppler]

Annual community mtg check in on this issue... @donsizemore? @landreev? @pdurbin? Bueller?

[donsizemore]

I'm happy to make a pull request to address the ticket title? The original issue I was hitting was a bug in Dataverse which apparently won't be fixed now that Shibboleth logins are assumed to be verified (though most of Odum's aren't... separate issue).

[poikilotherm]

@pdurbin asked me to create a backlink to #7424 here.

Looking at https://github.com/eclipse-ee4j/angus-mail/blob/3b3df4f82fc29c64bfc8fb4e510539b6ad12fbc2/providers/smtp/src/main/java/org/eclipse/angus/mail/smtp/SMTPTransport.java#L1749-L1763, it is pretty clear that if we send every of our messages with a from set in them, this will always have precedence over anything we do with "mail.from".

With #7424 I will simply go ahead and remove that setting, as we have been requiring people to set the database settings since ever.

[pdurbin]

Resolved by this pull request:

• Enable mail MTA/SMTP configuration using MicroProfile Config #9939