Wednesday, December 9, 2015

Edge transport doesn't talk to 2013 but works just fine with 2010

I could not get my Exchange 2010 Edge Transport to talk to my new Exchange 2013 server.  The Edge Transport continued to function just fine with Exchange 2010.  All the mail from my new 2013 box that was supposed to be going through the Edge Transport was just sitting in the mail queue.  I could see MSExchange EdgeSync 1032 errors in the logs.  I was also getting MSexchangeTransport 12023 errors in the event logs.  These two issues are related but I didn't know it last week when I found out that I was queuing mail.  I had to make a workaround to get mail flowing.  It wasn't until today that I fixed one and that lead me to a useful error of MSExchange EdgeSync 1033 that gave a certificate error.

I had to run
Enable-ExchangeCertificate -Thumbprint <thumbprint> -Services SMTP
to clear MSexchangeTransport 12023 error.  Using the ECP to set the "already set" SMTP flag did not work and no error or information was shown.  Running that command from EMS asked if I wanted to replace the default SMTP certificate. I answered yes and it told me there was an Edge Transport problem and to resubscribe.  Below is a partial screenshot.






I did the resubscription and all is good now.  Now I just have to tear down my workaround. 

Tuesday, December 1, 2015

Site to Zone Assignment list and IEHarden

I am going to apologize for knowingly posting something incomplete on here.  Was at work late working on this issue and I'm in zombie mode today.

If your Site to Zone Assignment list via GPO seems not to be working, and especially if it is working for some users and not other users it could be a registry setting.  If a user profile is made before changing the IE Enhanced Security Mode it will inherit the registry setting.  Then you change the setting for users via Server Manger -> "Configure IE ESC" new profiles are created without the registry setting.

I connected via remote registry and under the user section for my user sid I went to
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap
and found the value IEHarden set to 1.  If I set it to 0 and restarted IE it would do what I needed it to and expected it to.

I used this to solve my problems with the error of "an add-on for this website failed to run" and wouldn't start Java for a particular set of allowed sites.

Tuesday, October 20, 2015

Upgrading Dirsync to Azure Active Directory Connect (AADC) a cautionary tale

I usually try to be pretty thorough about things when I post.  I wasn't planning on posting a here's how you upgrade post so I have no screen shots or errors. Basically the upgrade is next next next.  The one problem that I had was that I needed to change the schedule for the sync process.  This is now a scheduled task that is automagically setup in AADC.

When you change the schedule DO NOT JUST CHANGE THE PASSWORD ON THE CREATED SERVICE ACCOUNT.  Make a new service account, or use another account to edit the scheduled task and run it.  If you want to change the password on the created service account it will mess up the encryption keys in AADC and you will have to jump through some hoops.  It is not a huge deal, but it takes way way less time to just make or use another account to run the task.  Just save yourself some time.

Tuesday, October 13, 2015

Forced reboots from WSUS

So lets say that you are oh I don't know upgrading Dirsync to the new hotness.  You do your research and everyone says it's just next next next and then happiness.  You don't even have to install any prereqs it will do that all for you.  (I'm looking at you https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnect-dirsync-upgrade-get-started/)  Well you have to update powershell and .NET.  You do that and reboot a couple times and then start the new Azure Active Directory Connect install.  It just all works.

Until you're in the middle of a full sync process and WSUS says hey you have new software that needs patches!  Reboot now!  I don't care what database operations you are doing that I am going to mess up!  BOW TO MY EVERY WHIM!

So I Google blocking reboots due to an update and every single suggestion I find says change this and reboot and you're golden.  What part of blocking a reboot do you not understand here people?

So I rush to download Process Explorer onto my machine.  I do the process find over the WSUS I'M GOING TO DESTROY YOU popup (I think it might actually say something like your machine is going to reboot in X time would you like to reboot now).  I right click that and go to the Threads tab and Suspend each thread.  That appears to have saved my hour of database operations from being interrupted.  I'll go in and unsuspend all that after my full sync is done and then let the server reboot for updates ON MY SCHEDULE.

Thursday, October 8, 2015

Installing Exchange 2013 (Something Went Wrong)

I'm not going to go into grand detail today.  Just a few quick notes on installing Exchange 2013 for the first time.  I have 2010 already in my org.

1) You can just install from the latest CU rollup.  I installed directly from 2013 CU9.

2) Install your first server as enterprise admin even if you have previously done the AD prep steps.  The first install of a 2013 server likes to create a user in root domain of your forest.  My Exchange servers live in a child domain and I got access denied when doing some user actions for the SystemMailbox.

3) When you go to access the ECP page you pretty much have to add ?ExchClientVer=15 at the end of the URL to get to see the ECP (https://mail.contoso.com/ecp/?ExchClientVer=15).  If not it just redirects you to a something went wrong page.  This appears to be because of the bug in the installer below.

4) Apparently there is a bug in some of the later CU installation processes.  After finalizing the install of 2013 and setting up the virtual directories I migrated a mailbox.  That mailbox was completely inaccessible. Outlook couldn't open it saying a folder was unavailable and OWA just gave the something went wrong error.  I was not able to even access https://server.contoso.com/autodiscover/autodiscover.xml.  It gave an error about
Parser Error Message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
Line 41:         <add assembly="Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, publicKeyToken=31bf3856ad364e35" />

 The application logs were filled with ASP.NET 4.0.30319.0 Event code: 3008.

From what I found on a couple other blogs is that MS split a file into two and the installer doesn't properly create the second file in all directories as it should in later CU updates.

Copy the sharedwebconfig.config file from:
E:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy
To:
E:\Program Files\Microsoft\Exchange Server\V15\ClientAccess

I'll now post the full error messages for completeness.

From IE:

Server Error in '/Autodiscover' Application.

Configuration Error

Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately. 

Parser Error Message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Source Error: 


Line 39:         <!--add assembly="Microsoft.Exchange.Common, Version=8.0.00207.0, Culture=neutral, publicKeyToken=31bf3856ad364e35"/-->
Line 40:         <!--add assembly="Microsoft.Exchange.Data.Common, Version=8.0.00207.0, Culture=neutral, publicKeyToken=31bf3856ad364e35"/-->
Line 41:         <add assembly="Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, publicKeyToken=31bf3856ad364e35" />
Line 42:       </assemblies>
Line 43:     </compilation>

Source File: E:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Autodiscover\web.config    Line: 41 

Assembly Load Trace: The following information can be helpful to determine why the assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' could not be loaded.


WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].


Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.34248



From Application Event Log:

 Event code: 3008
Event message: A configuration error has occurred.
Event time: 10/7/2015 4:12:01 PM
Event time (UTC): 10/7/2015 10:12:01 PM
Event ID: b89e6ff51088402180f450f7ad847cd1
Event sequence: 1
Event occurrence: 1
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/2/ROOT/Rpc-7-130887295216281384
    Trust level: Full
    Application Virtual Path: /Rpc
    Application Path: C:\Windows\System32\RpcProxy\
    Machine name: HOSTNAME

Process information:
    Process ID: 10544
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM

Exception information:
    Exception type: ConfigurationErrorsException
    Exception message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. (C:\Windows\System32\RpcProxy\web.config line 37)
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
   at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
   at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
   at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
   at System.Web.Compilation.BuildManager.ExecutePreAppStart()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)

Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.Load(String assemblyString)
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)



Request information:
    Request URL: https://FQDN:444/rpc/rpcproxy.dll?
FQDN:6001
    Request path: /rpc/rpcproxy.dll
    User host address: 
    User:
    Is authenticated: False
    Authentication Type:
    Thread account name: NT AUTHORITY\SYSTEM

Thread information:
    Thread ID: 28
    Thread account name: NT AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:    at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
   at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
   at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
   at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
   at System.Web.Compilation.BuildManager.ExecutePreAppStart()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)


Custom event details:

 

Wednesday, July 15, 2015

Exchange 2010 SP3 /mRecoverServer (the short version)

We "have" a HA Exchange 2010 setup.  The 2 x CAS/HT servers are Hyper-V and WLB.  The 2 x MBX/HT servers are clustered physical boxes.  At one point Outlook client connections just started dropping from Exchange seemingly at random.  After trying tons of things I found that after I rebooted one of the hosts it would calm down for a while.  Eventually I turned off the box and everything has been great since.  I have no clue why it was doing that.  I believe that something got borked inside Exchange on that box.

After having that box off for a while it was decided that I start testing Exchange 2013 on it.  I went to uninstall Exchange (once I was out of a critical work period) and it gave me all kinds of errors and would not uninstall gracefully.  I ran a setup uninstall command and that got me further but eventually failed.  That box was certainly in a failed state.

I knew that I couldn't just format c: and put 2013 on it and go on with my life as a higher up suggested to me.  I had to gracefully uninstall Exchange off that box.  (Well I didn't have to, but I *DID NOT* want to launch ADSIedit at all!)  I started doing research and found the recoverserver switch.  I really didn't want to format c: either and spend a day installing Windows and getting patches and doing all that fun stuff.  I figured that I could just say maybe delete a Microsoft directory under Program Files and recover from there.  Here is my trials and tribulations of a failed Exchange 2010 SP3 uninstall and the attempts to use recoverserver to get it back installed again:

GUI install certainly didn't work.  We all knew that would be the case.  So after a bit of searching I found the recoverserver command (which failed):

I also need to point out that I had problems running the install because I had all the services set to Disabled.  The installer needs to be able to start services or it will fail out.  

Error #1:

Setup /M:RecoverServer /TargetDir "E:\Program Files\Microsoft\Exchange Server\V14" /donotstartTransport

Error text:
The parameters specified are either missing additional required parameters or are not valid together.
To list the available command-line parameters, type Setup /?



I couldn't spot the subtle error.  Finally figured out the switch TargetDir needs a : between it and the target.

Working command:
Setup /m:RecoverServer /donotstartTransport /TargetDir:"E:\Program Files\Microsoft\Exchange Server\V14"

Error #2:

Error text:
The previous installation path could not be found in the registry. Only disaster recovery mode is available.
The previously installed version could not be determined from the registry. Only disaster recovery mode is available.

Exchange Server setup encountered an error.



Fix:
looked at registry path HKLM\software\Microsoft\Exchange server\V14
- deleted Key "Midfilecopy" and "Setup" under V14
DID NOT REBOOT THE SERVER

Source: https://social.technet.microsoft.com/Forums/exchange/en-US/209604ce-2f74-41b1-b0eb-c5ad19ece305/failed-to-upgrad-exchange-2010-sp1-the-previous-installation-path-could-not-be-found-in-the?forum=exchange2010



Error #3:




Error text:
Setup previously failed while performing the action "BuildToBuildUpgrade". You can't resume setup by performing the action "DisasterRecovery".

Exchange Server setup encountered an error.

Fix:
Under the registry Key that corresponds to the "installed" roles:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\<"installed" roles>

Delete Value "Action" with Data of "Install"
(If you want to skip past Error #4 then do the following while you're here:
Delete Value "Watermark" with Data of "Powershellsomethingsomethingsomething")

Source: http://baudlabs.com/setup-previously-failed-while-performing-the-action-install-error-during-exchange-2010-sp2-upgrade/

Error #4:

Error text:
A Setup failure previously occurred while installing the HubTransport role. Either run Setup again for just this role, or remove the role using Control Panel.

Fix:
You know that place you were just in?
Under the registry Key that corresponds to the "installed" roles:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\<"installed" roles>
Delete Value "Watermark" with Data of "Powershellsomethingsomethingsomething"

Source: https://myrefspot.wordpress.com/2013/01/10/exchange-2010-server-recovery-failed-on-hub-transport-role/

Error #5:

Error text:
Installing product C:\Temp\Exchange SP3 Install\exchangeserver.msi failed.
Fatal error during installation. Error code is 1603.

Fix:
I copied the install files over from a central location.  In the Updates folder under the extracted SP3 setup files at one point someone (probably me) placed the CU2 update in that directory.  It took me forever to figure this one out.  I deleted the file and the 1603 error went away.  It's always a good idea to just redownload and reextract the SP3 file to the machine you are working on in case some idiot messes up your source files directory.

Error #6:

Error text:
[ERROR] The virtual directory 'PowerShell-Proxy' already exists under 'FQDN.Server.Name/Default Web Site'.

Fix:
Went into IIS and deleted the PowerShell-Proxy virtual directory.

Had to remove the setup Keys and Values from under the registry as above. 
Had to reboot after this step since it got far enough into the install. 

Error #7:

Error text:
The following error was generated when "$error.Clear();
$wevtutil= join-path (join-path $env:SystemRoot system32) wevtutil.exe;
$manifestPath = [System.IO.Path]::Combine($RoleInstallPath, "Scripts\TSCrimsonManifest.man");
Start-SetupProcess -Name:"$wevtutil" -Args:"im `"$manifestPath`" "
        " was run: "Process execution failed with exit code 87.".
 
Fix:
Delete the registry Key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\Microsoft-Exchange-Troubleshooters/Operational

Had to remove the PowerShell-Proxy virtual directory.
Had to remove the setup Keys and Values from under the registry as above. 
Had to reboot after this step since it got far enough into the install. 

Source: https://social.technet.microsoft.com/Forums/exchange/en-US/17619f0e-bd05-4e40-9f60-49fc1bbb1602/problems-installing-mailbox-role-on-exchange-server-2010-sp3?forum=exchange2010

Summary:

In the registry:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\
Delete Keys: "Midfilecopy" and "Setup"
Under the 'server roles' Keys delete any Values that are "Action" and "Watermark"

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\
Delete Key: Microsoft-Exchange-Troubleshooters/Operational

In IIS:
Delete the PowerShell-Proxy virtual directory

Make sure your install files are good and do not have any extra stuff included in the folders.

Run an administrative command prompt then start setup from within the extracted SP3 folder making note of the subtleties of the command syntax:
Setup /m:RecoverServer /donotstartTransport /TargetDir:"E:\Program Files\Microsoft\Exchange Server\V14"

Friday, April 24, 2015

Exchange 2010 calendar emailing people for no reason

Problem: Exchange 2010 is sending out calendar invite declined and accepted replies to seemingly random people.  In my situation we have an HR.Office@contoso.com email address.  Of course HR staff come and go about like the leaves on trees.  They come and go and the HR.Office@contoso.com email address stays the same.  Well mostly the same.  Delegates to the calendar get added (and not by me, that is purely a user function as far as I am concerned).  Users being users they don't ever go in and clean anything up when someone moves on.  I am certain they think the Exchange Admin should be earning that fat check by reading minds and cleaning all that up.

First things first.
I started down this path thanks to two blog posts:
First I found this one that got me on the right path:
Delete Corrupted, Hidden or Stale rules from mailbox with MFCMapi

Then I found this one the next day that really helped me out:
Using MFCMAPI to delete delegate rules from mailbox

Of course I also need to point you to the tool to do all this with:
MFCMAPI

The ticket I got stated that Beth and Mary were getting responses to calendar invites to the HR.Office@contoso.com.  They said that this has been happening for a long time and they are finally getting tired of getting them.  It appears that this could have been going on for years.  According to what I have heard both of those women worked in HR before I even showed up to the organization. Uma, who is the new HR director, copied and pasted the text of the calendar reply into the ticket.

The first thing that I think is that there is a rule setup to forward those replies to Beth and Mary.  Turns out I'm right, but it's a long road to get there.  To prove that I'm right I go to the Message Tracking Log Explorer.




As you can see Uma Declines the calendar invite and the response goes to HR.Office@contoso.com.  Also according to a mailbox rule Uma@contoso.com sends the response to Beth@contoso.com and Mary@contoso.com.

At this point I get myself full access to both Uma@contoso.com and HR.Office@contoso.com.  I try to look at the mailbox rules from my Outlook client.  I was only able to see my own.  So I search for a way to look at mailbox rules via Powershell.  I find the command:

Get-InboxRule

I try:

Get-InboxRule Uma@contoso.com
and
Get-InboxRule HR.Office@contoso.com

for both I get the following error:
The specified mailbox "contoso.com/Accounts/domainadmin" doesn't exist. Reason: contoso.com/Accounts/domainadmin isn't a mailbox user.
    + CategoryInfo          : NotSpecified: (0:Int32) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : AD6EC4C4,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com


After digging a bit I figure out that I have to use:

Get-Mailbox Uma@contoso.com | Get-InboxRule

This command will actually show me all the rules that are enabled on the mailbox.  Well kinda, as I found out later it doesn't.  Also something of note.  You will get the following error (or, um, helpful output from Powershell) if the user doesn't have any mailbox rules setup:
The operation couldn't be performed because 'contoso.com/DifferentAccounts/Lastname, Uma\' couldn'
t be found.
    + CategoryInfo          : NotSpecified: (:) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : 112BED4B,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com


Running Get-Mailbox HR.Office@contoso.com | Get-InboxRule returns me normal Powershell output:
Name                          Enabled                       Priority                      RuleIdentity
----                          -------                       --------                      ------------
Clear categories on mail (... True                          1                             16791953785139232769

Clearly nothing helpful.  A bit disgruntled I do a lot of Google searching for calendar invite sending emails and mailboxrule.  It is at this point I find that there are hidden mailbox rules in Exchange.  Oh joy!  Oh wonderful!  Even better is that there isn't any good way to figure out what is going on because the are not only hidden from the user, they are hidden from the Exchange admin too.  You can't surface them in Powershell.

At this point I start a discovery search on my four mailboxes so that I can actually get to the mail items and look at them.  Going through the mailbox for Uma and HR.Office shows no sent item that goes to Beth or Mary.  Aside from going to their offices and looking through their deleted item recovery I am not going to be able to see an actual mail item.  I start the discovery search and when it completes I drill down to the deleted item and double click on it.  Once in the item I go to file -> properties to look at the headers:


Received: from exchange.contoso.com by exchange.contoso.com with Microsoft
 SMTP Server id 14.03.0224.002; Wed, 22 Apr 2015 08:18:51 -0600
Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: "Lastname, Uma" <Uma@contoso.com>
To: "Person, Gone"
    <IMCEAEX-_o=contoso_cn=DisabledAccounts_cn=GONE@contoso.local>, "Bright, Mary" <Mary@contoso.com>, "Short, Beth" <Beth@contoso.com>
Subject: Declined: Testing 2
Thread-Topic: Testing 2
Thread-Index: RandomLooking+SetofCharacters=
Date: Wed, 22 Apr 2015 14:18:50 +0000
Accept-Language: en-US
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: exchange.contoso.com
X-MS-Has-Attach:
X-Auto-Response-Suppress: All
X-MS-Exchange-Organization-SCL: -1
X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com
X-MS-TNEF-Correlator: GUIDLookingThing
x-originating-ip: [192.168.15.254]
MIME-Version: 1.0
Message-Id: <longrandomhexnumber@exchange.contoso.com>




This header provided me the key, but I didn't know it yet:

X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com

That is the mailbox that has the hidden mailbox rule on it that needed to be deleted to stop the mail from going to the "random" people.  (Quick note:  I called Microsoft support and talked to someone in a foreign country.  He had a very difficult time figuring out what the issue even was.  I spent 45 minutes on the phone trying to explain to him what was happening and when it was quitting time for me he still didn't sound like he could understand what was happening when we were going to pick up working on this the next day.  The rant here is that Microsoft support does not appear to employ people familiar with the nuts and bolts of the products they are supposed to support.  I really don't care where in the world they are.  I care that they seem lost when I am trying to explain what the problem is.)

When I get out of the shower the next morning I am convinced that the heart of the problem are those pesky hidden mailbox rules.  I go back to work and start digging using MFCMAPI.  What I find is that I need to delete that hidden rule on the HR.Office mailbox. Run it on your local desktop.



 Open MFCMAPI:


Logon:

Select Outlook Profile:

Double click the HR.Office mailbox:

A new window will open.  Expand Root Container -> Top of Information Store -> Inbox:
(If Top of Information Store is not there and you have IPM Subtree then your local Outlook in in cached mode.  You need to change this.  If you don't know what I'm talking about then you should stop what you're doing and not do anything else.)

Right click Inbox -> Other tables -> Rules table:

Find the hidden mailbox rule:



Delete the hidden mailbox rule:

That pretty much solved the problem that I had.  Of course I don't know yet if there are other problems that this caused.  I don't believe so.  If you're Exchange Admin I am sure you are fully aware to be afraid of doing some of this stuff.  Of course I also remember that at the last job this same problem cropped up and no one knew how to solve it.