Gmail SMTP Mailer

[mudmin]

I've never been able to nail down exactly what causes the freakouts. It's why I sort of set it up one way and then recommended that everyone used gmail.

I'm open to suggestions.

[Brandin]

I don't even know what to suggest honestly

I don't understand why I can't connect to the Gmail SMTP server from my prod server but not my localhost...it makes no sense!

It must be a server configuration thing.

I'm going to do some research and see what I can figure out

[mudmin]

I've actually never been able to send mail from my localhost. That's why there is an extension called phpfakesendmail or something like that. It's probably a built in firewall thing.

[tilmanharms]

Hello,

Firstly, I'm a new user of UserSpice, checking it out for some admin/monitoring applications and I LOVE it! It's been going real well so far, but I am having a very weird issue (seemingly related to this thread) that no amount of googling seems to yield a solution to. I'm using the standard gmail smtp setup as described. Although I changed to my own gmail credentials at some point, the issue is the same for both sets of credentials. (I have also sent via another smtp service with the same result).

So the problem is this:

In all cases, when sending test mails (or constructed mails) I can send emails successfully to my iCloud email address. However, when I send to either my gmail.com address or my work address (domain hosted by networksolutions.com), the mail function sends successfully, but the email never arrives. There is no error, but the email arrives on iCloud and nowhere else.

I've tried the above solution, commenting out the isSMTP line, with no change to the result. Running with the isSMTP option switched on results in a time out.

And finally, before posting this I tried one more thing: I moved the entire UserSpice setup from the private RedHat server located in Latin America to a personal SiteGround-hosted domain of mine. (Kudos to you guys, by the way, for how easy that was! And to my very great surprise, on SiteGround it suddenly WORKED!

So I compared the contents of the $mail object just after the send

Code:

$result = $mail->send();

in the helpers.php file and these were the only differences:

Successful (Chicago) $mail:
Message-ID is populated
lastMessageID:protected is populated

Failed (Latin America) $mail:
Message-ID is empty
lastMessageID:protected is empty

Code:

unique:protected

and

Code:

boundary:protected

were populated in both cases, but their values differed.

My only thought now is that somebody in the chain is blocking the message based on where it's coming from. Unfortunately I have NO idea how to determine or solve that. I realize that this has gone beyond the scope of your software, but if you are so inclined, any assistance or pointers will be GREATLY appreciated!

[mudmin]

Thanks so much for the detailed post.

Unfortunately it's super common for servers to all block each other when it comes to email generated from web sites...and with good reason...that's where the bulk of the SPAM comes from. I'm hoping that someone smarter than me can offer some insight. @Brandin.

This is also the reason why I'm working on building integrations with things like mailchimp etc for sending mail. Sending it from a server is just always going to be tricky.

[Brandin]

Now this I can actually speak to! Recently the provider I've been with for years moved my server from one location to another, changing my IPs. It just so happened the person that had the IP range before me, spammed the CRAP out of people and it got blocked by the most popular security firms that watch IP address for spamming (I can't think of any off the top of my head-but its a thing), this is a great resource to check that:
https://mxtoolbox.com/blacklists.aspx

If that is fine-I would say the next thing to look at is Server Configuration-is your server stopping the message from going but not producing an error? How do you have your Latin America server setup? A Control Panel setup or you installed Apache, SMTP, etc yourself? How was the Chicago server setup in comparison? These things might seem like they don't have any merit, but I've learned server configuration plays a huge role in how UserSpice can actually operate.

The concerning part I find is that you posted about the Message ID being empty...that tells me, that even when you get a successful note, it has no idea how to trace what happened, and that means it died---somewhere.

I know this post probably doesn't help a whole lot, but we will need to dig in a lot deeper on the routes that your emails have to take to get to Gmail for example. I wonder if the PHP Mailer-or your built in mailer service has some Logging Service to see what is happening-and finding out when it dies? I'm going to do some more research, but you got me stumped!

And although this is outside the scope of our project, I appreciate you bringing this to light in such an in-depth and well-tested way. When you do this, I'm more than positive Dan, I and other community members have no problem helping, because we know exactly where you left off, and how we can help debug this. Your post is more than appreciated, and we will do the best to help you!

Brandin.

[tilmanharms]

Thanks @Brandin and @mudmin for the fast and detailed responses! I will go through them in detail tomorrow. In the meantime I have managed to "correct" the missing message-ID value by replacing less and greater than signs in the sprintf formatting string in 'users/vendor/phpmailer/phpmailer/src/PHPMailer.php':

Code:

2288             //$this->lastMessageID = sprintf('<{3bc1fe685386cc4c3ab89a3f76566d8931e181ad17f08aed9ad73b30bf28114d}s@{3bc1fe685386cc4c3ab89a3f76566d8931e181ad17f08aed9ad73b30bf28114d}s>', $this->uniqueid, $this->serverHostname());

Code:

2289             $this->lastMessageID = sprintf('<{3bc1fe685386cc4c3ab89a3f76566d8931e181ad17f08aed9ad73b30bf28114d}s@{3bc1fe685386cc4c3ab89a3f76566d8931e181ad17f08aed9ad73b30bf28114d}s>', $this->uniqueid, $this->serverHostname());

I'm not sure why php sprintf() does this on the RedHat server (PHP 5.6.27) and not on the SiteGround account (PHP 7.0.27), if it's the version or a setting... Unfortunately this hasn't yet fixed the fact that emails are not arriving at my gmail account.

[Brandin]

I'm glad to hear you found a resolution for one problem...this could get us further. Where are you sending emails from that they're not arriving? This is directly from the email test page?

[tilmanharms]

Yes, just from the page at 'users/email_test.php'.

[tilmanharms]

@Brandin, thanks for the mxtoolbox link! That is really cool.

Please disregard everything I've said about the message ID and sprintf. I didn't use htmlspecialchars() for displaying the $mail variable, so the message-ID string was being treated as a tag by my browser. In some instances...

The ip has come up clean on the blacklist search and now that I'm using htmlspecialchars(), the $mail variables are identical between the servers (except of course the unique fields).

To answer your question about the Latin American setup, it's a RedHat Linux server with Apache (2.2.15) and PHP (5.6.27) installed from packages.

Finally, I've done some more googling and have managed to reduce my ignorance about email by the tiniest little bit... I was comparing the email headers of the email I sent that made it through to from Latin America to the one from SiteGround to another one from a mass mailer and found a suspicious discrepancy in a 'Received-SPF' header, which vaguely reminded me about something I read in a mass mailer quick start somewhere: TXT DNS record checking.

<pre>

Code:

ARC-Authentication-Results: i=1; mx.google.com;
       <strong>spf=softfail (google.com: domain of transitioning XXXXXXXXX@gmail.com does not designate XXX.XXX.XXX.XXX as permitted sender)</strong> smtp.mailfrom= XXXXXXXXX@gmail.com;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=icloud.com

</pre>



In short it brought me to this:

http://www.x-pose.org/2013/10/22/how-to-...ed-sender/

It makes sense to me that perhaps Gmail has a tougher policy on softfails (not softballs) in place than iCloud. So for now I am setting up the TXT record on my SiteGround domain and using that domain in the sender address of messages sent from my setup in Latin America.

Sorry, this is becoming very long winded about something that's not your problem, but I'll admit that I'm enjoying chatting to you about it (and it seems to be helping in clarifying my thoughts too)