Showing posts with label Exchange 2010. Show all posts
Showing posts with label Exchange 2010. Show all posts

Tuesday, January 17, 2017

Exchange contact NDR IMCEAEX NT5 00000000000000000000000000000000

Another bare bones post here.  I wish I actually understood WHY this happens.  What I do know is this does happen and here is my fix.

If you get a contact from another org you might end up seeing NDR errors that look like the following:

IMCEAEX-_o=NT5_ou=00000000000000000000000000000000_cn=...

When you get the NDR back it looks like the correct address is filled in in the message headers.  Only when you put your cursor over the person's name does the IMCEAEX address pop out.  This is what it looks like:

Delivery has failed to these recipients or groups:
LastName, FirstName
The email address you entered couldn't be found. Please check the recipient's email address and try to resend the message. If the problem continues, please contact your helpdesk.


Diagnostic information for administrators:
Generating server: mail.contoso.com
IMCEAEX-_o=NT5_ou=00000000000000000000000000000000_cn={32 char number}@mail.contoso.com
Remote Server returned '550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found'
Original message headers: ...

I don't know why, but appearntly it has something to do with a bad LegacyExchangeDN?

To fix it I run this command daily against my OU that the contacts are in:
Get-MailContact -OrganizationalUnit "OU=MailContacts,OU=OffSite,DC=ad,DC=mydomain,DC=com" | Update-Recipient



You may need to rebuild the OAB after doing this to solve the problem *right now* but I don't know.

Wednesday, July 15, 2015

Exchange 2010 SP3 /mRecoverServer (the short version)

We "have" a HA Exchange 2010 setup.  The 2 x CAS/HT servers are Hyper-V and WLB.  The 2 x MBX/HT servers are clustered physical boxes.  At one point Outlook client connections just started dropping from Exchange seemingly at random.  After trying tons of things I found that after I rebooted one of the hosts it would calm down for a while.  Eventually I turned off the box and everything has been great since.  I have no clue why it was doing that.  I believe that something got borked inside Exchange on that box.

After having that box off for a while it was decided that I start testing Exchange 2013 on it.  I went to uninstall Exchange (once I was out of a critical work period) and it gave me all kinds of errors and would not uninstall gracefully.  I ran a setup uninstall command and that got me further but eventually failed.  That box was certainly in a failed state.

I knew that I couldn't just format c: and put 2013 on it and go on with my life as a higher up suggested to me.  I had to gracefully uninstall Exchange off that box.  (Well I didn't have to, but I *DID NOT* want to launch ADSIedit at all!)  I started doing research and found the recoverserver switch.  I really didn't want to format c: either and spend a day installing Windows and getting patches and doing all that fun stuff.  I figured that I could just say maybe delete a Microsoft directory under Program Files and recover from there.  Here is my trials and tribulations of a failed Exchange 2010 SP3 uninstall and the attempts to use recoverserver to get it back installed again:

GUI install certainly didn't work.  We all knew that would be the case.  So after a bit of searching I found the recoverserver command (which failed):

I also need to point out that I had problems running the install because I had all the services set to Disabled.  The installer needs to be able to start services or it will fail out.  

Error #1:

Setup /M:RecoverServer /TargetDir "E:\Program Files\Microsoft\Exchange Server\V14" /donotstartTransport

Error text:
The parameters specified are either missing additional required parameters or are not valid together.
To list the available command-line parameters, type Setup /?



I couldn't spot the subtle error.  Finally figured out the switch TargetDir needs a : between it and the target.

Working command:
Setup /m:RecoverServer /donotstartTransport /TargetDir:"E:\Program Files\Microsoft\Exchange Server\V14"

Error #2:

Error text:
The previous installation path could not be found in the registry. Only disaster recovery mode is available.
The previously installed version could not be determined from the registry. Only disaster recovery mode is available.

Exchange Server setup encountered an error.



Fix:
looked at registry path HKLM\software\Microsoft\Exchange server\V14
- deleted Key "Midfilecopy" and "Setup" under V14
DID NOT REBOOT THE SERVER

Source: https://social.technet.microsoft.com/Forums/exchange/en-US/209604ce-2f74-41b1-b0eb-c5ad19ece305/failed-to-upgrad-exchange-2010-sp1-the-previous-installation-path-could-not-be-found-in-the?forum=exchange2010



Error #3:




Error text:
Setup previously failed while performing the action "BuildToBuildUpgrade". You can't resume setup by performing the action "DisasterRecovery".

Exchange Server setup encountered an error.

Fix:
Under the registry Key that corresponds to the "installed" roles:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\<"installed" roles>

Delete Value "Action" with Data of "Install"
(If you want to skip past Error #4 then do the following while you're here:
Delete Value "Watermark" with Data of "Powershellsomethingsomethingsomething")

Source: http://baudlabs.com/setup-previously-failed-while-performing-the-action-install-error-during-exchange-2010-sp2-upgrade/

Error #4:

Error text:
A Setup failure previously occurred while installing the HubTransport role. Either run Setup again for just this role, or remove the role using Control Panel.

Fix:
You know that place you were just in?
Under the registry Key that corresponds to the "installed" roles:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\<"installed" roles>
Delete Value "Watermark" with Data of "Powershellsomethingsomethingsomething"

Source: https://myrefspot.wordpress.com/2013/01/10/exchange-2010-server-recovery-failed-on-hub-transport-role/

Error #5:

Error text:
Installing product C:\Temp\Exchange SP3 Install\exchangeserver.msi failed.
Fatal error during installation. Error code is 1603.

Fix:
I copied the install files over from a central location.  In the Updates folder under the extracted SP3 setup files at one point someone (probably me) placed the CU2 update in that directory.  It took me forever to figure this one out.  I deleted the file and the 1603 error went away.  It's always a good idea to just redownload and reextract the SP3 file to the machine you are working on in case some idiot messes up your source files directory.

Error #6:

Error text:
[ERROR] The virtual directory 'PowerShell-Proxy' already exists under 'FQDN.Server.Name/Default Web Site'.

Fix:
Went into IIS and deleted the PowerShell-Proxy virtual directory.

Had to remove the setup Keys and Values from under the registry as above. 
Had to reboot after this step since it got far enough into the install. 

Error #7:

Error text:
The following error was generated when "$error.Clear();
$wevtutil= join-path (join-path $env:SystemRoot system32) wevtutil.exe;
$manifestPath = [System.IO.Path]::Combine($RoleInstallPath, "Scripts\TSCrimsonManifest.man");
Start-SetupProcess -Name:"$wevtutil" -Args:"im `"$manifestPath`" "
        " was run: "Process execution failed with exit code 87.".
 
Fix:
Delete the registry Key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\Microsoft-Exchange-Troubleshooters/Operational

Had to remove the PowerShell-Proxy virtual directory.
Had to remove the setup Keys and Values from under the registry as above. 
Had to reboot after this step since it got far enough into the install. 

Source: https://social.technet.microsoft.com/Forums/exchange/en-US/17619f0e-bd05-4e40-9f60-49fc1bbb1602/problems-installing-mailbox-role-on-exchange-server-2010-sp3?forum=exchange2010

Summary:

In the registry:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\
Delete Keys: "Midfilecopy" and "Setup"
Under the 'server roles' Keys delete any Values that are "Action" and "Watermark"

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\
Delete Key: Microsoft-Exchange-Troubleshooters/Operational

In IIS:
Delete the PowerShell-Proxy virtual directory

Make sure your install files are good and do not have any extra stuff included in the folders.

Run an administrative command prompt then start setup from within the extracted SP3 folder making note of the subtleties of the command syntax:
Setup /m:RecoverServer /donotstartTransport /TargetDir:"E:\Program Files\Microsoft\Exchange Server\V14"

Friday, April 24, 2015

Exchange 2010 calendar emailing people for no reason

Problem: Exchange 2010 is sending out calendar invite declined and accepted replies to seemingly random people.  In my situation we have an HR.Office@contoso.com email address.  Of course HR staff come and go about like the leaves on trees.  They come and go and the HR.Office@contoso.com email address stays the same.  Well mostly the same.  Delegates to the calendar get added (and not by me, that is purely a user function as far as I am concerned).  Users being users they don't ever go in and clean anything up when someone moves on.  I am certain they think the Exchange Admin should be earning that fat check by reading minds and cleaning all that up.

First things first.
I started down this path thanks to two blog posts:
First I found this one that got me on the right path:
Delete Corrupted, Hidden or Stale rules from mailbox with MFCMapi

Then I found this one the next day that really helped me out:
Using MFCMAPI to delete delegate rules from mailbox

Of course I also need to point you to the tool to do all this with:
MFCMAPI

The ticket I got stated that Beth and Mary were getting responses to calendar invites to the HR.Office@contoso.com.  They said that this has been happening for a long time and they are finally getting tired of getting them.  It appears that this could have been going on for years.  According to what I have heard both of those women worked in HR before I even showed up to the organization. Uma, who is the new HR director, copied and pasted the text of the calendar reply into the ticket.

The first thing that I think is that there is a rule setup to forward those replies to Beth and Mary.  Turns out I'm right, but it's a long road to get there.  To prove that I'm right I go to the Message Tracking Log Explorer.




As you can see Uma Declines the calendar invite and the response goes to HR.Office@contoso.com.  Also according to a mailbox rule Uma@contoso.com sends the response to Beth@contoso.com and Mary@contoso.com.

At this point I get myself full access to both Uma@contoso.com and HR.Office@contoso.com.  I try to look at the mailbox rules from my Outlook client.  I was only able to see my own.  So I search for a way to look at mailbox rules via Powershell.  I find the command:

Get-InboxRule

I try:

Get-InboxRule Uma@contoso.com
and
Get-InboxRule HR.Office@contoso.com

for both I get the following error:
The specified mailbox "contoso.com/Accounts/domainadmin" doesn't exist. Reason: contoso.com/Accounts/domainadmin isn't a mailbox user.
    + CategoryInfo          : NotSpecified: (0:Int32) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : AD6EC4C4,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com


After digging a bit I figure out that I have to use:

Get-Mailbox Uma@contoso.com | Get-InboxRule

This command will actually show me all the rules that are enabled on the mailbox.  Well kinda, as I found out later it doesn't.  Also something of note.  You will get the following error (or, um, helpful output from Powershell) if the user doesn't have any mailbox rules setup:
The operation couldn't be performed because 'contoso.com/DifferentAccounts/Lastname, Uma\' couldn'
t be found.
    + CategoryInfo          : NotSpecified: (:) [Get-InboxRule], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : 112BED4B,Microsoft.Exchange.Management.RecipientTasks.GetInboxRule
    + PSComputerName        : exchange.contoso.com


Running Get-Mailbox HR.Office@contoso.com | Get-InboxRule returns me normal Powershell output:
Name                          Enabled                       Priority                      RuleIdentity
----                          -------                       --------                      ------------
Clear categories on mail (... True                          1                             16791953785139232769

Clearly nothing helpful.  A bit disgruntled I do a lot of Google searching for calendar invite sending emails and mailboxrule.  It is at this point I find that there are hidden mailbox rules in Exchange.  Oh joy!  Oh wonderful!  Even better is that there isn't any good way to figure out what is going on because the are not only hidden from the user, they are hidden from the Exchange admin too.  You can't surface them in Powershell.

At this point I start a discovery search on my four mailboxes so that I can actually get to the mail items and look at them.  Going through the mailbox for Uma and HR.Office shows no sent item that goes to Beth or Mary.  Aside from going to their offices and looking through their deleted item recovery I am not going to be able to see an actual mail item.  I start the discovery search and when it completes I drill down to the deleted item and double click on it.  Once in the item I go to file -> properties to look at the headers:


Received: from exchange.contoso.com by exchange.contoso.com with Microsoft
 SMTP Server id 14.03.0224.002; Wed, 22 Apr 2015 08:18:51 -0600
Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: "Lastname, Uma" <Uma@contoso.com>
To: "Person, Gone"
    <IMCEAEX-_o=contoso_cn=DisabledAccounts_cn=GONE@contoso.local>, "Bright, Mary" <Mary@contoso.com>, "Short, Beth" <Beth@contoso.com>
Subject: Declined: Testing 2
Thread-Topic: Testing 2
Thread-Index: RandomLooking+SetofCharacters=
Date: Wed, 22 Apr 2015 14:18:50 +0000
Accept-Language: en-US
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: exchange.contoso.com
X-MS-Has-Attach:
X-Auto-Response-Suppress: All
X-MS-Exchange-Organization-SCL: -1
X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com
X-MS-TNEF-Correlator: GUIDLookingThing
x-originating-ip: [192.168.15.254]
MIME-Version: 1.0
Message-Id: <longrandomhexnumber@exchange.contoso.com>




This header provided me the key, but I didn't know it yet:

X-MS-Exchange-Inbox-Rules-Loop: HR.Office@contoso.com

That is the mailbox that has the hidden mailbox rule on it that needed to be deleted to stop the mail from going to the "random" people.  (Quick note:  I called Microsoft support and talked to someone in a foreign country.  He had a very difficult time figuring out what the issue even was.  I spent 45 minutes on the phone trying to explain to him what was happening and when it was quitting time for me he still didn't sound like he could understand what was happening when we were going to pick up working on this the next day.  The rant here is that Microsoft support does not appear to employ people familiar with the nuts and bolts of the products they are supposed to support.  I really don't care where in the world they are.  I care that they seem lost when I am trying to explain what the problem is.)

When I get out of the shower the next morning I am convinced that the heart of the problem are those pesky hidden mailbox rules.  I go back to work and start digging using MFCMAPI.  What I find is that I need to delete that hidden rule on the HR.Office mailbox. Run it on your local desktop.



 Open MFCMAPI:


Logon:

Select Outlook Profile:

Double click the HR.Office mailbox:

A new window will open.  Expand Root Container -> Top of Information Store -> Inbox:
(If Top of Information Store is not there and you have IPM Subtree then your local Outlook in in cached mode.  You need to change this.  If you don't know what I'm talking about then you should stop what you're doing and not do anything else.)

Right click Inbox -> Other tables -> Rules table:

Find the hidden mailbox rule:



Delete the hidden mailbox rule:

That pretty much solved the problem that I had.  Of course I don't know yet if there are other problems that this caused.  I don't believe so.  If you're Exchange Admin I am sure you are fully aware to be afraid of doing some of this stuff.  Of course I also remember that at the last job this same problem cropped up and no one knew how to solve it.

Thursday, March 14, 2013

Exchange 2010 WinRM error

I upgraded my Exchange boxes from Exchange 2010 SP1 to Exchange 2010 SP3 last night.  On the last mailbox server in the DAG I was unable to open the EMC or EMS.  I don't have the exact error but this is mostly what the EMS was throwing:  

"The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid."

The EMC was giving a kerberos error.  I couldn't copy and paste it into anything and could only click retry, so I don't have the error because I didn't write it down.

After reinstalling the WinRM IIS thing about 10 times and running quickconfig (qc, not all typed out) and going through every single setting in IIS and comparing the broken DAG machine with the working DAG machine I was stuck.  (And I'm sure you know exactly what I'm talking about because it's always the same answers to the same thing that never works but you try it 3 or 4 times because you don't know what else to do but you have to do something so...)

This blog pointed me in the right direction:

http://trikks.wordpress.com/2012/08/13/exchange-error-of-death-the-winrm-client-cannot-process-the-request-it-cannot-determine-the-content-type-of-the-http-response-from-the-destination-computer/

As it turns out the following file had errors in it:

E:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\PowerShell\web.config

On the server that was broken there were multiple entries with a path like the following:

<codebase href="file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Configuration.RedirectionModule.dll" version="14.0.0.0">


On my good server it had the same line as:

<codebase href="file:///E:\Program Files\Microsoft\Exchange Server\V14\bin\Microsoft.Exchange.Configuration.RedirectionModule.dll" version="14.0.0.0">


I renamed the broken web.config file and copied over the working one from my other box and everything just worked from that point.


This tool was no help to me on this error (probably because Exchange powershell was broken?):

http://blogs.technet.com/b/exchange/archive/2010/12/07/3411644.aspx

I hope this helps someone.

{edit - corrected typo in a sentence}