Brian's Virtually Useful Blog

Ok, I think I finally have a work around to the BL460C ASR reboots and ILO error 57's

April 27, 2009 · 1 comment
Reading a couple articles, it appears with the latest HP Proliant Support Pack (8.2), they have screwed the pooch and broken ILO to a point that it will cause Random Reboots, ASR's and other things. This seems to only effect 64 bit Windows Systems.

Here is what i've found...

HP ProLiant Integrated Lights-Out Management Interface Driver for Windows Server 2003/2008 x64 Editions

Latest = 1.14.0.0
Stable = 1.13.0.0

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&swItem=MTX-b0749333be7a4336a9957e40eb&prodNameId=3288156&swEnvOID=1113&swLang=8&taskId=135&mode=5

and

HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 x64 Editions

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&swItem=MTX-b016a4092d95486b88c4ebe86d&prodNameId=3288156&swEnvOID=1113&swLang=8&taskId=135&mode=5

Latest = 1.11.0.0
Stable = 1.8.0.0

Here is where I found it:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1240871128552+28353475&threadId=1323879
and
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1332008

and the fun keeps going

April 27, 2009
Well my previous post didn't solve the issue, looks like the issue is with the
"HP iLO Management Channel Interface Driver"

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1842750&swItem=MTX-b016a4092d95486b88c4ebe86d&prodNameId=3288156&swEnvOID=1113&swLang=8&taskId=135&mode=5

and i'm not the only with with an issue

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1240868410367+28353475&threadId=1323879

I am going to try and go back on the version to 1.13 and see if that stops the random reboots and the error ID 57's.

HP Blade Server rebooting for no apparent reason

April 24, 2009 · 5 comments
Just a bit less than a week after applying an HP Proliant Support pack to our BL460c G5 blades one of them running Windows 2003 begain rebooting randomly. The only real errors I could find were in the System log. These errors are about Event Source: hpqilo2 with an Event ID: 57. They had to do with Timeouts causing a ASR (reboot)

Description: The system has rebooted from a Automatic Server Recovery (ASR) event.
ProbableCause: 111 0x6f (Timeout)
ProbableCauseDescription: "ASR Reboot Occurred"

I did a full hardware swap since everything I found on google pointed to hardware. After that didn't resolve anything, I found that there is a new driver for HP ProLiant iLO 2 Management Controller Driver available. One of the known fixes is
"Resolved a problem where system could spontaneously reboot (ASR) if all CPU's were under continuous 100% load, and iLO 2 was reset (e.g. due to firmware update, changes to network settings, etc.)."

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3808910&swItem=MTX-b016a4092d95486b88c4ebe86d&prodNameId=3808911&swEnvOID=1113&swLang=13&taskId=135&mode=4&idx=2

This looks like it could be our issue, only time will tell, but if history is my guide then I should see within 48 hours if this machine will be stable or keep blowing up.

MAC ARP Poisoning, The case of the missing response packets ISA

April 16, 2009
Yesterday my network was subject to one of the most difficult to trace network problems i've seen. Basically at 12:45pm, the "internet died", and I rebooted our ISA server, it worked again. About 20 minutes later, it died again, this happened about 2-3 times more, I knew we had a major issue. Microsoft ISA Server is one of microsoft's best products, and especially ISA 2006 SP1 is very reliable. I put a packet trace outside my ISA firewall, it showed packets leaving my network AND returning, however, my ISA server reported that the packets were leaving, and NEVER returning. VERY ODD, ISA doesn't lie. We did the normal replace hardware, we even swapped out a router, switches and the ISA server hardware to no avail. This was very perplexing. Finally after 14 hours of wanting to tear my hair out, we found something in a packet trace. I had captured a ping of google.com when stuff worked, and when it wasn't working. This was just outside our ISA server(which is a back firewall, not a front one)but on the other side of our router (but this router doesn't do any packet filtering, so I ignored this *bad idea*). We noticed that packet responses LOOKED the same, but when we dug deeper we saw the MAC address on the response packets were different, but with the same correct IP. One Mac was accurate when it worked, and different when it wasn't. Someone had created a machine with the same IP Address of our router outside our ISA firewall. This was causing return packets from the internet to be misdirected to a server, and not to our router. So ISA wasn't lying, and doing some Mac Table lookups showed me which switches and ports to chase until I found the culprit rogue machine. After powering it down(and disconnecting the ethernet cable) all the problems stopped and I could finally go home at 3AM.
God I love IT.

User Can't open another users mailbox, "The name cannot be resolved."

February 24, 2009
So user1 is trying to open user2's mailbox, pure exchange 2007 environment, all patched, SP1, etc.. User2's mailbox says that user1 has full access.

So the exact error from outlook was "The name cannot be resolved. The name cannot be matched to a name in the address list." I assumed this was an outlook issue, deleted profiles, etc.. Nothing I did seemed to help, this was where I got frustrated and needed a new direction.

For troubleshooting I opened OWA logged in as user1, tried to open user2's mailbox, it says that it could not be found. I could open that mailbox logged in as myself through OWA and Outlook, and so could my two test user accounts. Very weird, user2's mailbox is a legacy account from Exchange 2000, 2003, but it doesn't make sense. After quite a bit of troubleshooting, moving the mailbox to other information stores, even detaching the mailbox and creating a new mailbox, no luck. I went into ADSIedit, checked all the properties, it was NOT hidden from the GAL.

Finally, I just deleted the user2 AD user2&mailbox. Created a new windows user2, reattached the original mailbox to that user, and after about 20 minutes of sync time, it worked and user1 was able to see user2's mailbox again.

So the issue was with User2's AD account, not the mailbox. Hopefully if it happens again I can nail it down a little more.

Removing Legacy Permissions from Pure Exchange 2007 Environment

February 12, 2009 · 5 comments
First let me explain the context of the issue. We had an exchange admin who over the years upgraded our exchange from 5.5 to 2000 to 2003. This person also had to occasionally add himself permissions to a mailbox in order to check on this or that and assist with issues, normal exchange admin stuff. This person decided to move onto another role inside the company, but now no longer should be an exchange admin. These permissions were removed sucessfully, and we then upgraded fully to Exchange 2007.
Later doing a full audit of mailbox permissions, we noticed that this person still had their "manually assingned mailbox rights" to about 50 users mailboxes. A simple powershell remove-mailboxpermission command and boom MOST of them were gone.
However some of them gave us a disturbing error:
========================================
Remove-MailboxPermission : Cannot remove ACE on object "CN=Mailbox.Name,OU=Resources,OU=Exchange,OU=Accounts,DC=Domain,DC=local
" for account "Domain\old.admin" because it is not present.
At line:1 char:25
========================================
After a VERY LONG call to Microsoft to just confirm we had an issue and i'm not a moron, they were completely out of ideas. I tried some vbscripts to "output" our permissions. One from this blog http://gsexdev.blogspot.com/2005/04/reverse-msexchmailboxsecuritydescripto.html was useful, but didn't quite get me there, Also modifying Greg's Script, I was able to remove permissions to msExchMailboxSecurityDescriptor (with this help http://msdn.microsoft.com/en-us/library/aa705958(VS.85).aspx), but unfortunatley this permission still existed inside of powershell and in reality. ADSI Edit had nothing that could help me, there simply wasn't a permission anywhere you can modify it. Microsoft suggested I add the permission back to the user, but I got the error the permission was already there. They also suggested I give the old IT user Domain admin permissions, shockingly that did nothing. They also said I should detach both the old IT admin's mailbox, and one of the mailboxes with the permissions, this also did nothing, once they were re-attached the permissions were still there.

Ultimately, I tried the easist thing in the world, which of course worked. I got on my old XP desktop with Exchange 2003 admin tools installed. This worked like a champ, went into the properties of the users who needed to be modified, went to "Exchange Advanced" then "Mailbox Rights" and removed the old IT admin from the permissions.

I personally consider this a bug with Exchange 2007 admin tools/powershell. but we'll see how Microsoft sees it.

Adding NT AUTHORITY\SELF back for permissions to your mailboxes

February 5, 2009 · 4 comments
Some of you that have upgraded exchange for years like myself may notice that in EXCHANGE 2007, users don't get rights to their own mailboxes, they user "NT AUTHORITY\SELF" to gain access.

Some older users were granted rights directly to their own mailboxes. Before you remove those (and lock them out of their email) run this Powershell script to grant that Authority account back in.

Get-mailbox | Add-Mailboxpermission -user "NT AUTHORITY\SELF" -Accessrights FullAccess