Exchange 2010 calendar emailing people for no reason

Problem: Exchange 2010 is sending out calendar invite declined and accepted replies to seemingly random people.  In my situation we have an HR.Office@contoso.com email address.  Of course HR staff come and go about like the leaves on trees.  They come and go and the HR.Office@contoso.com email address stays the same.  Well mostly the same.  Delegates to the calendar get added (and not by me, that is purely a user function as far as I am concerned).  Users being users they don't ever go in and clean anything up when someone moves on.  I am certain they think the Exchange Admin should be earning that fat check by reading minds and cleaning all that up.

First things first.
I started down this path thanks to two blog posts:
First I found this one that got me on the right path:

Delete Corrupted, Hidden or Stale rules from mailbox with MFCMapi

Then I found this one the next day that really helped me out:
Using MFCMAPI to delete delegate rules from mailbox

Of course I also need to point you to the tool to do all this with:
MFCMAPI

The ticket I got stated that Beth and Mary were getting responses to calendar invites to the HR.Office@contoso.com.  They said that this has been happening for a long time and they are finally getting tired of getting them.  It appears that this could have been going on for years.  According to what I have heard both of those women worked in HR before I even showed up to the organization. Uma, who is the new HR director, copied and pasted the text of the calendar reply into the ticket.

The first thing that I think is that there is a rule setup to forward those replies to Beth and Mary.  Turns out I'm right, but it's a long road to get there.  To prove that I'm right I go to the Message Tracking Log Explorer.

As you can see Uma Declines the calendar invite and the response goes to HR.Office@contoso.com.  Also according to a mailbox rule Uma@contoso.com sends the response to Beth@contoso.com and Mary@contoso.com.

At this point I get myself full access to both Uma@contoso.com and HR.Office@contoso.com.  I try to look at the mailbox rules from my Outlook client.  I was only able to see my own.  So I search for a way to look at mailbox rules via Powershell.  I find the command:

Get-InboxRule

I try:

Get-InboxRule Uma@contoso.com
and
Get-InboxRule HR.Office@contoso.com

for both I get the following error:
The specified mailbox "contoso.com/Accounts/domainadmin" doesn't exist. Reason: contoso.com/Accounts/domainadmin isn't a mailbox user.
    + CategoryInfo          : NotSpecified: (0:Int32) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : AD6EC4C4,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com

After digging a bit I figure out that I have to use:

Get-Mailbox Uma@contoso.com | Get-InboxRule

This command will actually show me all the rules that are enabled on the mailbox.  Well kinda, as I found out later it doesn't.  Also something of note.  You will get the following error (or, um, helpful output from Powershell) if the user doesn't have any mailbox rules setup:
The operation couldn't be performed because 'contoso.com/DifferentAccounts/Lastname, Uma\' couldn'
t be found.
    + CategoryInfo          : NotSpecified: (:) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : 112BED4B,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com

Running Get-Mailbox HR.Office@contoso.com | Get-InboxRule returns me normal Powershell output:
Name                          Enabled                       Priority                      RuleIdentity
----                          -------                       --------                      ------------
Clear categories on mail (... True                          1                             16791953785139232769

Clearly nothing helpful.  A bit disgruntled I do a lot of Google searching for calendar invite sending emails and mailboxrule.  It is at this point I find that there are hidden mailbox rules in Exchange.  Oh joy!  Oh wonderful!  Even better is that there isn't any good way to figure out what is going on because the are not only hidden from the user, they are hidden from the Exchange admin too.  You can't surface them in Powershell.

At this point I start a discovery search on my four mailboxes so that I can actually get to the mail items and look at them.  Going through the mailbox for Uma and HR.Office shows no sent item that goes to Beth or Mary.  Aside from going to their offices and looking through their deleted item recovery I am not going to be able to see an actual mail item.  I start the discovery search and when it completes I drill down to the deleted item and double click on it.  Once in the item I go to file -> properties to look at the headers:


Received: from exchange.contoso.com by exchange.contoso.com with Microsoft
 SMTP Server id 14.03.0224.002; Wed, 22 Apr 2015 08:18:51 -0600
Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: "Lastname, Uma" <Uma@contoso.com>
To: "Person, Gone"
    <IMCEAEX-_o=contoso_cn=DisabledAccounts_cn=GONE@contoso.local>, "Bright, Mary" <Mary@contoso.com>, "Short, Beth" <Beth@contoso.com>
Subject: Declined: Testing 2
Thread-Topic: Testing 2
Thread-Index: RandomLooking+SetofCharacters=
Date: Wed, 22 Apr 2015 14:18:50 +0000
Accept-Language: en-US
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: exchange.contoso.com
X-MS-Has-Attach:
X-Auto-Response-Suppress: All
X-MS-Exchange-Organization-SCL: -1
X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com
X-MS-TNEF-Correlator: GUIDLookingThing
x-originating-ip: [192.168.15.254]
MIME-Version: 1.0
Message-Id: <longrandomhexnumber@exchange.contoso.com>




This header provided me the key, but I didn't know it yet:

X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com

That is the mailbox that has the hidden mailbox rule on it that needed to be deleted to stop the mail from going to the "random" people.  (Quick note:  I called Microsoft support and talked to someone in a foreign country.  He had a very difficult time figuring out what the issue even was.  I spent 45 minutes on the phone trying to explain to him what was happening and when it was quitting time for me he still didn't sound like he could understand what was happening when we were going to pick up working on this the next day.  The rant here is that Microsoft support does not appear to employ people familiar with the nuts and bolts of the products they are supposed to support.  I really don't care where in the world they are.  I care that they seem lost when I am trying to explain what the problem is.)

When I get out of the shower the next morning I am convinced that the heart of the problem are those pesky hidden mailbox rules.  I go back to work and start digging using MFCMAPI.  What I find is that I need to delete that hidden rule on the HR.Office mailbox. Run it on your local desktop.



 Open MFCMAPI:

Logon:

Select Outlook Profile:

Double click the HR.Office mailbox:

A new window will open.  Expand Root Container -> Top of Information Store -> Inbox:
(If Top of Information Store is not there and you have IPM Subtree then your local Outlook in in cached mode.  You need to change this.  If you don't know what I'm talking about then you should stop what you're doing and not do anything else.)

Right click Inbox -> Other tables -> Rules table:

Find the hidden mailbox rule:

Delete the hidden mailbox rule:

That pretty much solved the problem that I had.  Of course I don't know yet if there are other problems that this caused.  I don't believe so.  If you're Exchange Admin I am sure you are fully aware to be afraid of doing some of this stuff.  Of course I also remember that at the last job this same problem cropped up and no one knew how to solve it.

Exchange 2010 WinRM error

I upgraded my Exchange boxes from Exchange 2010 SP1 to Exchange 2010 SP3 last night.  On the last mailbox server in the DAG I was unable to open the EMC or EMS.  I don't have the exact error but this is mostly what the EMS was throwing:  

"The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid."

The EMC was giving a kerberos error.  I couldn't copy and paste it into anything and could only click retry, so I don't have the error because I didn't write it down.

After reinstalling the WinRM IIS thing about 10 times and running quickconfig (qc, not all typed out) and going through every single setting in IIS and comparing the broken DAG machine with the working DAG machine I was stuck.  (And I'm sure you know exactly what I'm talking about because it's always the same answers to the same thing that never works but you try it 3 or 4 times because you don't know what else to do but you have to do something so...)

This blog pointed me in the right direction:

http://trikks.wordpress.com/2012/08/13/exchange-error-of-death-the-winrm-client-cannot-process-the-request-it-cannot-determine-the-content-type-of-the-http-response-from-the-destination-computer/

As it turns out the following file had errors in it:

E:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\PowerShell\web.config

On the server that was broken there were multiple entries with a path like the following:

<codebase href="file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Configuration.RedirectionModule.dll" version="14.0.0.0">

On my good server it had the same line as:

<codebase href="file:///E:\Program Files\Microsoft\Exchange Server\V14\bin\Microsoft.Exchange.Configuration.RedirectionModule.dll" version="14.0.0.0">

I renamed the broken web.config file and copied over the working one from my other box and everything just worked from that point.


This tool was no help to me on this error (probably because Exchange powershell was broken?):

http://blogs.technet.com/b/exchange/archive/2010/12/07/3411644.aspx

I hope this helps someone.

{edit - corrected typo in a sentence}

It's been a while...

It's been a long time since I've posted on here. We have our deployment working and it's been stable for about a year and a half now I guess. The new MDT 2010 is out now. I never looked at any of the betas for it because I did not want to disturb my current deployment point. I recently downloaded the 2010 (RTM?) and looked at it and it is going to take me a little bit to digest it.

The first thing that I can say about it is that I cannot wait to play with it. The MS folks have added quite a bit of nice features it looks like to me. I wasn't bright enough to understand how to setup one deployment point and have it be a master for some other ones (for lack of a better term). It appears that is easy in 2010 (and was supposedly in earlier ones, but not easy). Following that same way of doing things having a dev and prod deployment looks to be tons easier now too.

There are lots of improvements so if you haven't looked at MDT 2010 you really should take a peek. As I get to start working with it over the next little bit I will get back to updating this blog with the things that I figure out.

Production Deployment

We started our production deployment here on May 12, and all in all I am pretty happy. There have been a couple "gotchas", but after about a year of working on this thing I have to say it has went very smoothly. One of our biggest issues here is bandwidth. Going from XP to Vista our deployment has, in most instances, trippled in size. With XP we were about 10G in size, and with Vista we are at about 30G. The problem with that is that our network infrastructure is inconsistent at best. There is nothing I can do about that and so that's the limitation that I have to deal with.

The first deployment that we did we did on an 8-port (that's what we had at the time) Gbit switch. That allowed us to do six machines at a time. Two ports were used by the net connection for windows updates and the nas for building from. Those six machines took three hours. Next we put twelve machines on a 100Mb switch. Those machines took four and a half hours. That evening we plugged the nas into the wall and started the rest of our machines (about 25). Sorry I don't have exact figures on this, but by the next morning several machines failed asking for the CD, I think 3 just failed with some deployment error, and 4-6 were still building.

So after looking at this I tried to think of different things to try to get things done faster. First off, I have to say that I am pretty happy with the loading on our nas. I don't know how well it should load, but I know that we were never able to load our old server pulling down ghost images like we are able to do on the nas (If we got more than 5 or 6 ghost sessions going some would start to fail). We are using a Qnap TS-409 Pro nas. One of the things that I thought about doing is copying the application install files down to the local machine and then have the task sequence fire them off the local hdd. Currently the installs are fired off the server/nas. I don't know if that would work in our environment because until the past year we always got the smallest hdd we could from the vendor because we did not really need any bigger drives. I also thought about other temporary network topologies, but nothing really stands out as a solution. So in the end we just opted for a 24-port Gbit switch and let em grind.

The biggest time hog for deployment/installing is Visual Studio 2008. The old version took forever to install. I think this one takes twice as long to install! This install just drives me insane. I have sat and watched the resource monitor for this install and it is just slow. First off the it looks like msiexec.exe needs to be ported over to being a multithreaded app. For most all of the VS install the processor is pegged right at 50%. (I've tried looking to see if there is any documentation on Windows Installer being either single threaded or multithreaded and I could not find anything, so I am assuming it is single threaded.) Office 2007 and CS3 take some time too, but they seem to be a fair amount better about dealing with network/cpu/hdd/memory usage for their installs. VS just seems to have a one track mind though. It is either using network or cpu or hdd, but when one spikes all the others fall off.

As for any errors that I've seen the most common one is a program give me an exit code of 13 and I cannot find a way for the deployment wizzard to accept that return code as ok. Other machines have failed up doing windows updates, which to me is not a show stopper. They will get their updates soon enough. One or two errors happened that caused us to just wipe the drive and start over. I don't remember the exact errors, but they were pretty odd ones about installing Vista and wiping the hdd did the trick.

Quick and dirty "spare" deployment point

In our environment we have some "problem" labs that we have to deploy Vista into. For one reason or another the network will not really handle deploying several machines at a time. Trying to remedy this solution I got us a nas to work off of. This presents some interesting challenges.

The deployment tool isn't really setup to handle a dual server type setup very well (if at all). I might be missing something that would make this really easy, but I don't think so. Other blogs I have read talk about dns round-robbin and load balancing and the like, but none of those scenarios really address the problems that I face -- slow connections to the server room, either because of a wan-link or a flakey network. At any rate I got us a small nas so that I would have a local resource we could build from in any lab.

What I did was create shares on the nas that are named the same as the ones that are on our primary server. I use two different shares for the deployment and the applications. I then copied all the files over from each share to the corresponding one I made on the nas. After doing this I had to modify a few things to point to the nas.

One of the first things that I had to modify was the bootstrap.ini file to point to my nas -- but that was easier said than done. There might be an easier way to do it than what I put here, but this is what I did. I took the LTI iso and mounted it so I could read/write to it. I then had to open the boot.wim in the Sources folder that is in the iso (for the time being I'm going to leave it to your google-foo to figure that out -- if anyone asks I'll write that up later). I edited the bootstrap.ini in there and saved all of it back out and burned a boot cd.

From there I had to modify serveral files on the nas to point to the nas:

\Control\Applications.xml
and
\Control\*\TS.xml

* is the task sequence number of any task sequence you might use from the nas

I did all of this outside of the deployment tool because I did not want to make two deployment points and I didn't want to duplicate task sequences and have to edit both of them for each change I make (well I have to edit it anyway, but I would rather use a txt editor to do a search and replace rather than using the task sequence GUI).

That's my quick and dirty way to use another server to deploy from. Good luck.

Using Robocopy

In part of my deployment I use robocopy to copy over a batch file for later use by the system. I am now to the point in my deployment where I am working on the "little" things and getting the kinks out of them. After my task sequence ran robocopy it would fail because robocopy returned an exit code of 1. After doing a bit of digging I found that means everything worked out just fine. So if you are using robocopy in any of your task sequences you will want to find that/those step(s) and go to the options tab and edit a few things. I put a check in front of "Continue on error" so that copying over one file is not going to derail my deployment. I also added in a 1 in the "Success codes:" box. I have not ever touched this box before, but I have faith that is going to solve my problem with robocopy. Be aware that you may have to put other robocopy "success" codes in to make your particular deployment not bark at you.

Booting off USB

Found a little note in the MSDT help files on booting off USB. I had a couple little gotchas when doing this, and I'll let you know what they are. One of the gotchas is in their instructions so I'll give you my instructions (this wipes your flash drive, just so you know).

1 Open admin cmd prompt.
2 Run "diskpart".
3 Run diskpart command "list disk".
4 Make note of the disk number your flash drive is.
5 Run diskpart command "select disk (drive number)".
6 Run diskpart command "clean".
7 Run diskpart command "create partition primary".
8 Run diskpart command "select partition 1".
9 Run diskpart command "active".
10 Run diskpart command "format fs=fat32".
11 Run diskpart command "assign"
12 Run diskpart command "exit"


Then the instructions tell you to use xcopy to copy the contents of your boot CD to the flash drive. Xcopy would not work for me. I just copied the contents of the CD over to the flash drive and it seems to work just fine.

Remember to disconnect your flash drive before your machine needs to reboot or your sequence will fail.