Friday, April 24, 2015

Exchange 2010 calendar emailing people for no reason

Problem: Exchange 2010 is sending out calendar invite declined and accepted replies to seemingly random people.  In my situation we have an HR.Office@contoso.com email address.  Of course HR staff come and go about like the leaves on trees.  They come and go and the HR.Office@contoso.com email address stays the same.  Well mostly the same.  Delegates to the calendar get added (and not by me, that is purely a user function as far as I am concerned).  Users being users they don't ever go in and clean anything up when someone moves on.  I am certain they think the Exchange Admin should be earning that fat check by reading minds and cleaning all that up.

First things first.
I started down this path thanks to two blog posts:
First I found this one that got me on the right path:
Delete Corrupted, Hidden or Stale rules from mailbox with MFCMapi

Then I found this one the next day that really helped me out:
Using MFCMAPI to delete delegate rules from mailbox

Of course I also need to point you to the tool to do all this with:
MFCMAPI

The ticket I got stated that Beth and Mary were getting responses to calendar invites to the HR.Office@contoso.com.  They said that this has been happening for a long time and they are finally getting tired of getting them.  It appears that this could have been going on for years.  According to what I have heard both of those women worked in HR before I even showed up to the organization. Uma, who is the new HR director, copied and pasted the text of the calendar reply into the ticket.

The first thing that I think is that there is a rule setup to forward those replies to Beth and Mary.  Turns out I'm right, but it's a long road to get there.  To prove that I'm right I go to the Message Tracking Log Explorer.




As you can see Uma Declines the calendar invite and the response goes to HR.Office@contoso.com.  Also according to a mailbox rule Uma@contoso.com sends the response to Beth@contoso.com and Mary@contoso.com.

At this point I get myself full access to both Uma@contoso.com and HR.Office@contoso.com.  I try to look at the mailbox rules from my Outlook client.  I was only able to see my own.  So I search for a way to look at mailbox rules via Powershell.  I find the command:

Get-InboxRule

I try:

Get-InboxRule Uma@contoso.com
and
Get-InboxRule HR.Office@contoso.com

for both I get the following error:
The specified mailbox "contoso.com/Accounts/domainadmin" doesn't exist. Reason: contoso.com/Accounts/domainadmin isn't a mailbox user.
    + CategoryInfo          : NotSpecified: (0:Int32) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : AD6EC4C4,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com


After digging a bit I figure out that I have to use:

Get-Mailbox Uma@contoso.com | Get-InboxRule

This command will actually show me all the rules that are enabled on the mailbox.  Well kinda, as I found out later it doesn't.  Also something of note.  You will get the following error (or, um, helpful output from Powershell) if the user doesn't have any mailbox rules setup:
The operation couldn't be performed because 'contoso.com/DifferentAccounts/Lastname, Uma\' couldn'
t be found.
    + CategoryInfo          : NotSpecified: (:) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : 112BED4B,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com


Running Get-Mailbox HR.Office@contoso.com | Get-InboxRule returns me normal Powershell output:
Name                          Enabled                       Priority                      RuleIdentity
----                          -------                       --------                      ------------
Clear categories on mail (... True                          1                             16791953785139232769

Clearly nothing helpful.  A bit disgruntled I do a lot of Google searching for calendar invite sending emails and mailboxrule.  It is at this point I find that there are hidden mailbox rules in Exchange.  Oh joy!  Oh wonderful!  Even better is that there isn't any good way to figure out what is going on because the are not only hidden from the user, they are hidden from the Exchange admin too.  You can't surface them in Powershell.

At this point I start a discovery search on my four mailboxes so that I can actually get to the mail items and look at them.  Going through the mailbox for Uma and HR.Office shows no sent item that goes to Beth or Mary.  Aside from going to their offices and looking through their deleted item recovery I am not going to be able to see an actual mail item.  I start the discovery search and when it completes I drill down to the deleted item and double click on it.  Once in the item I go to file -> properties to look at the headers:


Received: from exchange.contoso.com by exchange.contoso.com with Microsoft
 SMTP Server id 14.03.0224.002; Wed, 22 Apr 2015 08:18:51 -0600
Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: "Lastname, Uma" <Uma@contoso.com>
To: "Person, Gone"
    <IMCEAEX-_o=contoso_cn=DisabledAccounts_cn=GONE@contoso.local>, "Bright, Mary" <Mary@contoso.com>, "Short, Beth" <Beth@contoso.com>
Subject: Declined: Testing 2
Thread-Topic: Testing 2
Thread-Index: RandomLooking+SetofCharacters=
Date: Wed, 22 Apr 2015 14:18:50 +0000
Accept-Language: en-US
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: exchange.contoso.com
X-MS-Has-Attach:
X-Auto-Response-Suppress: All
X-MS-Exchange-Organization-SCL: -1
X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com
X-MS-TNEF-Correlator: GUIDLookingThing
x-originating-ip: [192.168.15.254]
MIME-Version: 1.0
Message-Id: <longrandomhexnumber@exchange.contoso.com>




This header provided me the key, but I didn't know it yet:

X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com

That is the mailbox that has the hidden mailbox rule on it that needed to be deleted to stop the mail from going to the "random" people.  (Quick note:  I called Microsoft support and talked to someone in a foreign country.  He had a very difficult time figuring out what the issue even was.  I spent 45 minutes on the phone trying to explain to him what was happening and when it was quitting time for me he still didn't sound like he could understand what was happening when we were going to pick up working on this the next day.  The rant here is that Microsoft support does not appear to employ people familiar with the nuts and bolts of the products they are supposed to support.  I really don't care where in the world they are.  I care that they seem lost when I am trying to explain what the problem is.)

When I get out of the shower the next morning I am convinced that the heart of the problem are those pesky hidden mailbox rules.  I go back to work and start digging using MFCMAPI.  What I find is that I need to delete that hidden rule on the HR.Office mailbox. Run it on your local desktop.



 Open MFCMAPI:


Logon:

Select Outlook Profile:

Double click the HR.Office mailbox:

A new window will open.  Expand Root Container -> Top of Information Store -> Inbox:
(If Top of Information Store is not there and you have IPM Subtree then your local Outlook in in cached mode.  You need to change this.  If you don't know what I'm talking about then you should stop what you're doing and not do anything else.)

Right click Inbox -> Other tables -> Rules table:

Find the hidden mailbox rule:



Delete the hidden mailbox rule:

That pretty much solved the problem that I had.  Of course I don't know yet if there are other problems that this caused.  I don't believe so.  If you're Exchange Admin I am sure you are fully aware to be afraid of doing some of this stuff.  Of course I also remember that at the last job this same problem cropped up and no one knew how to solve it.