Thursday, April 21, 2016

mrsproxy.svc authentication scheme 'Negotiate' (403) Forbidden

The call to 'https://server.contoso.com/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client a
uthentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden..
    + CategoryInfo          : NotSpecified: (:) [New-MoveRequest], RemoteTransientException
    + FullyQualifiedErrorId : [Server=BL2PR02MB2129,RequestId=cd8a2403-0b3c-4fee-ad2c-a27305435b36,TimeStamp=4/20/2016
    8:51:30 PM] [FailureCategory=Cmdlet-RemoteTransientException] 8E22672D,Microsoft.Exchange.Management.Migration.Ma
  ilboxReplication.MoveRequest.NewMoveRequest
    + PSComputerName        : ps.outlook.com


Run:
Get-WebServicesVirtualDirectory |FL

Compare the URL listed above with the URLs in ExternalUrl and InternalUrl and adjust accordingly.  Create a new migration endpoint in Office 365 that uses the new name.

It's been a very long day.  I'll try to give a proper update once I have had some downtime.

[Edit below]
So in the planning for going from Exchange 2010 to Exchange 2013 I did a lot of research on load balancing and namespaces.  I decided to go for the each service has its own namespace model so that if for some reason the address book was down that owa would still show as alive.  Well I got a lesson in things you still don't know about Exchange so you're gonna learn today.

On 2010 everything was accessed via one namespace like mail.contoso.com.  So in our hybrid environment with Office 365 the migration endpoint mapping pointed to mail.contoso.com.  When I cut over the namespaces to 2013 I left mail.contoso.com for the friendly owa name and broke out all the other names.  I started getting errors in my scripts that I made to migrate mailboxes to Office 365.  The error listed above.  Of course it wasn't much help to determine what the actual problem was.

After lots of digging and hair pulling I finally stumbled upon the suggestion to use Get-WebServicesVirtualDirectory |FL and look for something else that was useless.  However I did note that the URL was not mail.contoso.com that was in there.  Not trusting that observation I changed the authentication and got the same error.  After that I removed and remade the migration endpoint mapping.  It tried to autodiscover the setting and failed.  I then put in ews.contoso.com and it was happy and my scripts were happy.

Wednesday, December 9, 2015

Edge transport doesn't talk to 2013 but works just fine with 2010

I could not get my Exchange 2010 Edge Transport to talk to my new Exchange 2013 server.  The Edge Transport continued to function just fine with Exchange 2010.  All the mail from my new 2013 box that was supposed to be going through the Edge Transport was just sitting in the mail queue.  I could see MSExchange EdgeSync 1032 errors in the logs.  I was also getting MSexchangeTransport 12023 errors in the event logs.  These two issues are related but I didn't know it last week when I found out that I was queuing mail.  I had to make a workaround to get mail flowing.  It wasn't until today that I fixed one and that lead me to a useful error of MSExchange EdgeSync 1033 that gave a certificate error.

I had to run
Enable-ExchangeCertificate -Thumbprint <thumbprint> -Services SMTP
to clear MSexchangeTransport 12023 error.  Using the ECP to set the "already set" SMTP flag did not work and no error or information was shown.  Running that command from EMS asked if I wanted to replace the default SMTP certificate. I answered yes and it told me there was an Edge Transport problem and to resubscribe.  Below is a partial screenshot.






I did the resubscription and all is good now.  Now I just have to tear down my workaround. 

Tuesday, December 1, 2015

Site to Zone Assignment list and IEHarden

I am going to apologize for knowingly posting something incomplete on here.  Was at work late working on this issue and I'm in zombie mode today.

If your Site to Zone Assignment list via GPO seems not to be working, and especially if it is working for some users and not other users it could be a registry setting.  If a user profile is made before changing the IE Enhanced Security Mode it will inherit the registry setting.  Then you change the setting for users via Server Manger -> "Configure IE ESC" new profiles are created without the registry setting.

I connected via remote registry and under the user section for my user sid I went to
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap
and found the value IEHarden set to 1.  If I set it to 0 and restarted IE it would do what I needed it to and expected it to.

I used this to solve my problems with the error of "an add-on for this website failed to run" and wouldn't start Java for a particular set of allowed sites.

Tuesday, October 20, 2015

Upgrading Dirsync to Azure Active Directory Connect (AADC) a cautionary tale

I usually try to be pretty thorough about things when I post.  I wasn't planning on posting a here's how you upgrade post so I have no screen shots or errors. Basically the upgrade is next next next.  The one problem that I had was that I needed to change the schedule for the sync process.  This is now a scheduled task that is automagically setup in AADC.

When you change the schedule DO NOT JUST CHANGE THE PASSWORD ON THE CREATED SERVICE ACCOUNT.  Make a new service account, or use another account to edit the scheduled task and run it.  If you want to change the password on the created service account it will mess up the encryption keys in AADC and you will have to jump through some hoops.  It is not a huge deal, but it takes way way less time to just make or use another account to run the task.  Just save yourself some time.

Tuesday, October 13, 2015

Forced reboots from WSUS

So lets say that you are oh I don't know upgrading Dirsync to the new hotness.  You do your research and everyone says it's just next next next and then happiness.  You don't even have to install any prereqs it will do that all for you.  (I'm looking at you https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnect-dirsync-upgrade-get-started/)  Well you have to update powershell and .NET.  You do that and reboot a couple times and then start the new Azure Active Directory Connect install.  It just all works.

Until you're in the middle of a full sync process and WSUS says hey you have new software that needs patches!  Reboot now!  I don't care what database operations you are doing that I am going to mess up!  BOW TO MY EVERY WHIM!

So I Google blocking reboots due to an update and every single suggestion I find says change this and reboot and you're golden.  What part of blocking a reboot do you not understand here people?

So I rush to download Process Explorer onto my machine.  I do the process find over the WSUS I'M GOING TO DESTROY YOU popup (I think it might actually say something like your machine is going to reboot in X time would you like to reboot now).  I right click that and go to the Threads tab and Suspend each thread.  That appears to have saved my hour of database operations from being interrupted.  I'll go in and unsuspend all that after my full sync is done and then let the server reboot for updates ON MY SCHEDULE.

Thursday, October 8, 2015

Installing Exchange 2013 (Something Went Wrong)

I'm not going to go into grand detail today.  Just a few quick notes on installing Exchange 2013 for the first time.  I have 2010 already in my org.

1) You can just install from the latest CU rollup.  I installed directly from 2013 CU9.

2) Install your first server as enterprise admin even if you have previously done the AD prep steps.  The first install of a 2013 server likes to create a user in root domain of your forest.  My Exchange servers live in a child domain and I got access denied when doing some user actions for the SystemMailbox.

3) When you go to access the ECP page you pretty much have to add ?ExchClientVer=15 at the end of the URL to get to see the ECP (https://mail.contoso.com/ecp/?ExchClientVer=15).  If not it just redirects you to a something went wrong page.  This appears to be because of the bug in the installer below.

4) Apparently there is a bug in some of the later CU installation processes.  After finalizing the install of 2013 and setting up the virtual directories I migrated a mailbox.  That mailbox was completely inaccessible. Outlook couldn't open it saying a folder was unavailable and OWA just gave the something went wrong error.  I was not able to even access https://server.contoso.com/autodiscover/autodiscover.xml.  It gave an error about
Parser Error Message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
Line 41:         <add assembly="Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, publicKeyToken=31bf3856ad364e35" />

 The application logs were filled with ASP.NET 4.0.30319.0 Event code: 3008.

From what I found on a couple other blogs is that MS split a file into two and the installer doesn't properly create the second file in all directories as it should in later CU updates.

Copy the sharedwebconfig.config file from:
E:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy
To:
E:\Program Files\Microsoft\Exchange Server\V15\ClientAccess

I'll now post the full error messages for completeness.

From IE:

Server Error in '/Autodiscover' Application.

Configuration Error

Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately. 

Parser Error Message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Source Error: 


Line 39:         <!--add assembly="Microsoft.Exchange.Common, Version=8.0.00207.0, Culture=neutral, publicKeyToken=31bf3856ad364e35"/-->
Line 40:         <!--add assembly="Microsoft.Exchange.Data.Common, Version=8.0.00207.0, Culture=neutral, publicKeyToken=31bf3856ad364e35"/-->
Line 41:         <add assembly="Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, publicKeyToken=31bf3856ad364e35" />
Line 42:       </assemblies>
Line 43:     </compilation>

Source File: E:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Autodiscover\web.config    Line: 41 

Assembly Load Trace: The following information can be helpful to determine why the assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' could not be loaded.


WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].


Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.34248



From Application Event Log:

 Event code: 3008
Event message: A configuration error has occurred.
Event time: 10/7/2015 4:12:01 PM
Event time (UTC): 10/7/2015 10:12:01 PM
Event ID: b89e6ff51088402180f450f7ad847cd1
Event sequence: 1
Event occurrence: 1
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/2/ROOT/Rpc-7-130887295216281384
    Trust level: Full
    Application Virtual Path: /Rpc
    Application Path: C:\Windows\System32\RpcProxy\
    Machine name: HOSTNAME

Process information:
    Process ID: 10544
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM

Exception information:
    Exception type: ConfigurationErrorsException
    Exception message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. (C:\Windows\System32\RpcProxy\web.config line 37)
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
   at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
   at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
   at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
   at System.Web.Compilation.BuildManager.ExecutePreAppStart()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)

Could not load file or assembly 'Microsoft.Exchange.Security, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.Load(String assemblyString)
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)



Request information:
    Request URL: https://FQDN:444/rpc/rpcproxy.dll?
FQDN:6001
    Request path: /rpc/rpcproxy.dll
    User host address: 
    User:
    Is authenticated: False
    Authentication Type:
    Thread account name: NT AUTHORITY\SYSTEM

Thread information:
    Thread ID: 28
    Thread account name: NT AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:    at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
   at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
   at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
   at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
   at System.Web.Compilation.BuildManager.ExecutePreAppStart()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)


Custom event details:

 

Wednesday, July 15, 2015

Exchange 2010 SP3 /mRecoverServer (the short version)

We "have" a HA Exchange 2010 setup.  The 2 x CAS/HT servers are Hyper-V and WLB.  The 2 x MBX/HT servers are clustered physical boxes.  At one point Outlook client connections just started dropping from Exchange seemingly at random.  After trying tons of things I found that after I rebooted one of the hosts it would calm down for a while.  Eventually I turned off the box and everything has been great since.  I have no clue why it was doing that.  I believe that something got borked inside Exchange on that box.

After having that box off for a while it was decided that I start testing Exchange 2013 on it.  I went to uninstall Exchange (once I was out of a critical work period) and it gave me all kinds of errors and would not uninstall gracefully.  I ran a setup uninstall command and that got me further but eventually failed.  That box was certainly in a failed state.

I knew that I couldn't just format c: and put 2013 on it and go on with my life as a higher up suggested to me.  I had to gracefully uninstall Exchange off that box.  (Well I didn't have to, but I *DID NOT* want to launch ADSIedit at all!)  I started doing research and found the recoverserver switch.  I really didn't want to format c: either and spend a day installing Windows and getting patches and doing all that fun stuff.  I figured that I could just say maybe delete a Microsoft directory under Program Files and recover from there.  Here is my trials and tribulations of a failed Exchange 2010 SP3 uninstall and the attempts to use recoverserver to get it back installed again:

GUI install certainly didn't work.  We all knew that would be the case.  So after a bit of searching I found the recoverserver command (which failed):

I also need to point out that I had problems running the install because I had all the services set to Disabled.  The installer needs to be able to start services or it will fail out.  

Error #1:

Setup /M:RecoverServer /TargetDir "E:\Program Files\Microsoft\Exchange Server\V14" /donotstartTransport

Error text:
The parameters specified are either missing additional required parameters or are not valid together.
To list the available command-line parameters, type Setup /?



I couldn't spot the subtle error.  Finally figured out the switch TargetDir needs a : between it and the target.

Working command:
Setup /m:RecoverServer /donotstartTransport /TargetDir:"E:\Program Files\Microsoft\Exchange Server\V14"

Error #2:

Error text:
The previous installation path could not be found in the registry. Only disaster recovery mode is available.
The previously installed version could not be determined from the registry. Only disaster recovery mode is available.

Exchange Server setup encountered an error.



Fix:
looked at registry path HKLM\software\Microsoft\Exchange server\V14
- deleted Key "Midfilecopy" and "Setup" under V14
DID NOT REBOOT THE SERVER

Source: https://social.technet.microsoft.com/Forums/exchange/en-US/209604ce-2f74-41b1-b0eb-c5ad19ece305/failed-to-upgrad-exchange-2010-sp1-the-previous-installation-path-could-not-be-found-in-the?forum=exchange2010



Error #3:




Error text:
Setup previously failed while performing the action "BuildToBuildUpgrade". You can't resume setup by performing the action "DisasterRecovery".

Exchange Server setup encountered an error.

Fix:
Under the registry Key that corresponds to the "installed" roles:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\<"installed" roles>

Delete Value "Action" with Data of "Install"
(If you want to skip past Error #4 then do the following while you're here:
Delete Value "Watermark" with Data of "Powershellsomethingsomethingsomething")

Source: http://baudlabs.com/setup-previously-failed-while-performing-the-action-install-error-during-exchange-2010-sp2-upgrade/

Error #4:

Error text:
A Setup failure previously occurred while installing the HubTransport role. Either run Setup again for just this role, or remove the role using Control Panel.

Fix:
You know that place you were just in?
Under the registry Key that corresponds to the "installed" roles:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\<"installed" roles>
Delete Value "Watermark" with Data of "Powershellsomethingsomethingsomething"

Source: https://myrefspot.wordpress.com/2013/01/10/exchange-2010-server-recovery-failed-on-hub-transport-role/

Error #5:

Error text:
Installing product C:\Temp\Exchange SP3 Install\exchangeserver.msi failed.
Fatal error during installation. Error code is 1603.

Fix:
I copied the install files over from a central location.  In the Updates folder under the extracted SP3 setup files at one point someone (probably me) placed the CU2 update in that directory.  It took me forever to figure this one out.  I deleted the file and the 1603 error went away.  It's always a good idea to just redownload and reextract the SP3 file to the machine you are working on in case some idiot messes up your source files directory.

Error #6:

Error text:
[ERROR] The virtual directory 'PowerShell-Proxy' already exists under 'FQDN.Server.Name/Default Web Site'.

Fix:
Went into IIS and deleted the PowerShell-Proxy virtual directory.

Had to remove the setup Keys and Values from under the registry as above. 
Had to reboot after this step since it got far enough into the install. 

Error #7:

Error text:
The following error was generated when "$error.Clear();
$wevtutil= join-path (join-path $env:SystemRoot system32) wevtutil.exe;
$manifestPath = [System.IO.Path]::Combine($RoleInstallPath, "Scripts\TSCrimsonManifest.man");
Start-SetupProcess -Name:"$wevtutil" -Args:"im `"$manifestPath`" "
        " was run: "Process execution failed with exit code 87.".
 
Fix:
Delete the registry Key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\Microsoft-Exchange-Troubleshooters/Operational

Had to remove the PowerShell-Proxy virtual directory.
Had to remove the setup Keys and Values from under the registry as above. 
Had to reboot after this step since it got far enough into the install. 

Source: https://social.technet.microsoft.com/Forums/exchange/en-US/17619f0e-bd05-4e40-9f60-49fc1bbb1602/problems-installing-mailbox-role-on-exchange-server-2010-sp3?forum=exchange2010

Summary:

In the registry:
\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v14\
Delete Keys: "Midfilecopy" and "Setup"
Under the 'server roles' Keys delete any Values that are "Action" and "Watermark"

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\
Delete Key: Microsoft-Exchange-Troubleshooters/Operational

In IIS:
Delete the PowerShell-Proxy virtual directory

Make sure your install files are good and do not have any extra stuff included in the folders.

Run an administrative command prompt then start setup from within the extracted SP3 folder making note of the subtleties of the command syntax:
Setup /m:RecoverServer /donotstartTransport /TargetDir:"E:\Program Files\Microsoft\Exchange Server\V14"