omreport : failed to load external entity "/opt/dell/srvadmin/var/lib/openmanage/xslroot//oma/cli/about.xsl"

If you're having the following error when executing omreport :

 I/O warning : failed to load external entity "/opt/dell/srvadmin/var/lib/openmanage/xslroot//oma/cli/about.xsl"  
 error  
 xsltParseStylesheetFile : cannot parse /opt/dell/srvadmin/var/lib/openmanage/xslroot//oma/cli/about.xsl  
 Error! XML Transformation failed  

Then install srvadmin-omcommon package :

 # yum install srvadmin-omcommon  

Read More

DRAC Firmware update failed : Error: 30001 Method httpCgiErrorPage()

Have tried to update an old DRAC4 Firmware from firmware 1.5 to 1.75 via Linux binary and came to an unplaisant surprise :

 Dell Remote Access Controller 4/P  
 The version of this Update Package is newer than the currently installed version.  
 Software application name: Dell Remote Access Controller 4/P Firmware  
 Package version: 1.75  
 Installed version: 1.50  
 Continue? Y/N:Y  
 Executing update...  
 WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE IS IN PROGRESS.  
 THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!  
 ......................................................................................
 /tmp/duptmp.xml:6: parser error : Extra content at the end of the document  
 <SVMExecution lang = "en">  
 ^  
 /tmp/.dellSP-XmlResult12908-32487.M19124:6: parser error : Extra content at the end of the document  
 <SVMExecution lang = "en">  
 ^  
 unable to parse /tmp/.dellSP-XmlResult12908-32487.M19124  
 /tmp/.dellSP-XmlResult12908-32487.M19124:6: parser error : Extra content at the end of the document  
 <SVMExecution lang = "en">  
 ^  
 unable to parse /tmp/.dellSP-XmlResult12908-32487.M19124  

Doesn't look good and of course if I try to access the DRAC via HTTPs, I've got a nice CGI error :

 Error: 30001 Method httpCgiErrorPage()  

Looked on the web and somebody (who contacted Dell Support) advises to shutdown the server, unplug the DRAC card for a while and plug it in back... Well explain to your CTO that you need to shutdown a production server, unrack it, unplug a card just because a DRAC update failed o_O
Reference: http://lists.us.dell.com/pipermail/linux-poweredge/2008-January/034556.html

The solution that worked for me was to install the racadm Dell tool on my bastion and reset the firmware remotely.

• First install racadm :

 # yum install srvadmin-racadm4.x86_64  

Note : This is for DRAC4, didn't had the issue with newer DRAC.
Note 2 : You need to have the Dell OSMA repository installed on your server:
http://www.openfusion.net/linux/dell_omsa

•  Then run the following command :

 # racadm -rDRAC_IP -i racreset  

Note : Change DRAC_IP with your DRAC IP.
Note 2 : This operation will NOT erase your DRAC configuration.

•  Wait a while, pray, and if you're lucky as me you should be back on line (with the original firmware version of course).

Final word, I stopped being lazy and updated the firmware via the Web GUI which is a long and annoying process. Of course I used Internet Explorer as I felt like Murphy's law was around this day ^^

Hope that helps !

Read More

Yum stuck/hangs at "Running Transaction Test"

If yum is stuck at the "Running Transaction Test" step, double check that you don't have a stalled network mount (NFS,SMB,etc) somewhere.

Umount it and retry your yum/rpm command.

More info on how to umount a stalled NFS share :
http://sysnet-adventures.blogspot.fr/2013/05/umount-stalledfrozen-nfs-mount-point.html

Read More

Umount a stalled/frozen NFS mount point

NFS is known to be a little nasty when it comes to umount stalled shares.

Most of the time a simple umount doesn't work, which is a bit frustrating specially when it comes to production servers; The process just hangs and there is no way to interrupt...

Below two procedures to umount stalled  NFS shares. You should try method one before method two as it requires some network "hacks".

Method 1 :

Use a forced lazy umount, this method works 90% of the time :

 # umount -f -l /mnt/nfs_share
Note : Don't use bash auto-completion !!!  

Method 2:

This method is to be used only if method one failed.

The trick is to temporarily steal the NFS server IP adress on the NFS client (the one with stalled mount) so this client thinks that the NFS server is still alive.

Warning : Use method 1 above if your NFS server is still reachable from the NFS client. Otherwise you'll have an IP conflit and trust me you really don't want that to happen.

Let's assume the NFS server IP is 192.168.0.1

1. Double check that the NFS server is down with ping or nmap.
2. If your NFS client has very restrictive IPTables rules shut them down temporarily
3. On the NFS client, set the NFS server IP as secondary address

 # ifconfig eth0:0 192.168.0.1  
 Note : Adjust interface to your own needs  

4. Umount the NFS with a forced lzay umount

 # umount -f -l /mnt/nfs_share  
 Note : Don't use bash auto-completion !!! 

5.  Check that the NFS mount is gone
6. Remove secondary interface
   

    # ifconfig eth0:0 down  
    Note : Adjust interface to your own needs  

7. Restart IPTables if needed
8. Be happy
9. Go to sleep, it's been a long day (or night)

 If you have multiple NFS clients stalled, you can set the secondary IP on one client only.

• Client 1 : Step 1 to 5

• Client 2 to n : Step 4 and 5

• Client 1 : Step 6 to 9

This will only work if your NFS client can communicate between each others (watch for IPTables or any other filtering softwares/devices).

Hope that helps ! (that helped me a lot :)

Read More

Remove absolute path from MD5 file

The following command will remove absolute paths from a md5 file :

  sed -i -r "s/ .*\/(.+)/  \1/g" file.md5  

This is quite useful when you download backup files which have MD5 containing absolut paths.

For example :

 $ cat file.md5  
 8ee6e966f2cb7a84620ed137604e00c5 /data/prod/file  

If you want to check this file on another server, you won't be able to do it unless you put the data file in the exact same directory (/data/prod/).

After running the above sed command, your md5 file will look like :

 $ cat file.md5  
 8ee6e966f2cb7a84620ed137604e00c5 file  

You can then check your md5 print with "md5sum -c file.md5". You just need to have the MD5 file in the same directory as the data file.

Read More

Whitelist files with Clamav

Exclude files from scan :

Sometimes, you will need to whitelist files from a scan, clamscan offers the --exclude option but its usage is not really user-friendly...

Imagine that you need to exclude two files /data/rep/file1 and /data/rep2/file2, the command line would be :

 # clamscan -r -i --exclude=/data/rep/file1 --exclude=/data/rep2/file2  

This is fine if you have few files to whitelist but it quickly becomes unreadable when you have dozen files and directories.

The solution is to input a file to clamscan with xargs. Create a text file containing all files/directories you need to whiltelist (one file/directory per line) :

 # cat /var/lib/clamav/whitelist-files.txt  
 /data/rep/file1  
 /data/rep2/file2  

You can also add regexp like *.mp3 (be aware that this is quite dangerous)

Run clamscan with the following command :

 # sed -e 's/^/--exclude=/' /var/lib/clamav/whitelist-files.txt | xargs clamscan -r -i /directory_to_scan/  

Don't forget to put double quotes or escape when you exclude paths with special characters (especially spaces).

Last but not least, always double check that the files you're whitelisting are completely safe. You can check that out with a meta AV engine like Jotti :
http://virusscan.jotti.org/en

Whitelist a virus signature :

To whitelist a virus a signature, you need to get the ClamAV signature definition, this is the code you have on the right side of the infected file line. For example :

 /data/file.flv: CVE_2012_0773-2 FOUND  

In this case the signature definition is CVE_2012_0773-2, add it to /var/lib/clamav/whitelist-signatures.ign2

That's all ! Be very cautious when whitelisting Virus signatures.
Hope that helps !

Read More

Multiple cluster per site, Site Recovery Manager (SRM) with svMotion capability

Here is a design VMware told me was 'unique' LOL. Sounded perfectly normal to me.

My plan was (obvious to me):
vSphere/SRM 5.0
2 locations
2 vCenters, of course required for SRM
2 clusters in each vCenter, one cluster at each location per vCenter.
Equallogic SRAs which probably doesnt matter who you have for a SAN SRA vendor.

This would give us sort of an 'X' configuration and allow us to svMotion/vMotion from location to location during business hours by simply mounting a common volume as well as have SRM capability across datacenters if bad things happened. I realize there is no longer a need for the common volume for svMotion in vSphere 5.1. Also keep in mind we have a lot of bandwidth between our sites. Your results may vary.

So the design looked valid, if interesting, on paper. All the svMotion worked fine. I just could not make the Array Managers 'see' or work with one pair of clusters within SRM. How this was visible, was that in my Array pair under site recovery for one of the vCenters, the Devices tab in the Array Pair would not show a 'Remote Device' and a 'Datastore' both for all the datastores I wanted to use. Only the ones associated with one pair. This meant I could SRM across one pair of clusters but not both

The resolution was to create a second Array Manager pair reversing the storage listings in both sites. So if in site A you had storage1 to storage2 and site B you had storage2 to storage1, that is enough to enable a single cluster per site. I had to go in to site A for example and add another pair for storage2 to storage1 and vise versa in site B giving me 2 pairs per site.

I hope this helps and I will elaborate if you want.

Charlie

Read More

Enable LDAP SSL/TLS user authentification in Zabbix

By default Zabbix Web interface doesn't offer the SSL/TLS encryption option for the LDAP connector however the feature is available in the PHP code.

If you need to enable the startTLS feature, you will have to edit the PHP file manually :

Edit /usr/share/zabbix/include/classes/class.cldap.php on Zabbix Web server :

 vi /usr/share/zabbix/include/classes/class.cldap.php  

Search for the 'starttls' definition (line 44) and set the value to 'true' :

 'starttls' => true,  

Save the file and you're good to go !

Hope that helps !

Read More

Dell 11-12th gen 710-720 firmware update observations

I have recently been playing around with a new install of the Dell Management vCenter plugin version 1.6. Very nice tool. Should save TONS of time. I also had to learn a bit about the server's firmware upgrade process now that the LifeCycle controller is involved. Here are a few things. Again this is either r710 or r720. I have been using both a lot lately.
_________________________________________________________
This error after a PERC firmware update:
PR1 PERC replaced part detected
This message is safe to ignore either way.
Enter lifecycle controller from the boot menu, and run an inventory to clear the message.
_________________________________________________________


I forget which update did this, probably the NIC firmware. The machine would not boot. The message was:
"Plug and play Configuration Error:
Option ROM Shadow RAM Allocation error."

The fix is to go into the NIC BIOS and disable all option ROMS for every NIC. Too many loading can cause this as there is not enough Shadow RAM to hold them all if you have many NICs. They apparently all get enabled with the firmware update.
_________________________________________________________


I will post more as I find them.
Hope this helps someone

Charlie

Read More

Updated*** Dell OpenManage 7.2 on ESXi5 fails with error 15

I was using VMware Update Manager but the method you use to install probably doesn't matter.

This appears to be the package is not signed correctly. Like the rest of you, I am busy so I didn't spend much time finding a fix. I went back to the 7.1 version of the agents and it installed no problem.

Initially I thought maybe the older 6.5 version might have been the problem and uninstalled it. The command to remove the old VIB, which was 6.5 for me, was esxcli software vib remove OpenManageThat didn't solve the problem however so I got smart, putty'd in and looked at the update log located at /var/log/esxupdate.log. I found this:
esxupdate: esxupdate: ERROR: InstallationError: ('Dell_bootbank_OpenManage_7.2-0000', "('Dell_bootbank_OpenManage_7.20000', 'Could not find a trusted signer.')")

I changed Update Manage to push out 7.1 and it installed fine.

If anyone has more info please share!

Hope this helps someone.

Charlie

Read More

Windows Server 2008R2 missing gateway IP address

OK so this one was fun.  Guess we can't test every scenario in the real world right?

We had several Windows 2008R2 servers that would not have an IPV4 gateway IP address after a reboot. If I entered one and saved it, then went back and looked, it would be gone again immediately. These machines were ESXi5 VMs but that should not have mattered, you'll see why.

I saw lots of articles about going into the registry and adding the gateway IP address when it disappears and cannot be added again normally. Nope. That should not be a fix anyway, more of a hack.
Other articles about deleting the hidden NICs, etc. Still no.

The issue for me was Symantec Enterprise Protection (SEP). Some of the servers here had a version 11 package with firewall, network protection, whatever, enabled. In my defense this was before my time. I do not typically let SEP handle the firewall on server OS's. We upgraded to version 12 package that did not have the firewall enabled. The upgrade from the management console does not call for a reboot because we're all about uptime these days right? Symantec is under the same pressure I'm sure. Well a week or 2 later when our scheduled maintenance patching came around, the machine booted. Bang, no gateway.

The 'no reboot required' install for SEP upgrades works fine normally except in the case when you go from firewall to no firewall. Evidently you need to remove the firewall while still in version 11, or remove SEP version 11 entirely, and reboot, then upgrade.

If you are already stuck with no gateway, like I was, here is the order of things to get back online.

1. Get a version 11 package with the firewall enabled onto a CD or ISO file that you can mount to the Windows OS (because you are unable to get on the network)
2. Uninstall SEP12, you can leave LiveUpdate on.
3. Reboot
4. Reinstall the SEP11 package with firewall
5. Reboot (you should get your gateway back here)
6. Uninstall SEP11 with firewall
7. Reboot
8. Reinstall 12, done.

According to Symantec support, this is a 'known issue' and will be fixed in the next version. How that happens when it is more a problem with the old version, I do not know. Also there have been several releases of version 12 already. If they were going to prompt to uninstall you would think it would have already been instituted.

Anyway hope this helps someone!

Charlie

Read More

Hello!

First post. 

I am brand new to blogging. No kidding right? The reason it came to this is I find a lot of interesting methods in administrating all things IT that I would like to share with others. Or remember for myself :) Or should be documented somewhere but I had to work to get so I'll share.

A little background. I have been in IT since 1998 or so. Before that I turned wrenches. Yeah long story. You can find me in LinkedIn also. I am an MCSE many times over since NT4, a 3 time VCP and have worked in some amazing environments.

I hope my posts help someone


Charlie

Read More