Microsoft Silverlight update repeatedly fails to install with error 0x80070643

The Problem
My wife made me aware that she always had one update to install while shutting down her computer. Being an IT guy by day, it took me a little before I got around to troubleshooting the issue. Anyways a quick glance in the "System" Event Logs revealed that a security update for Microsoft Silverlight seemed to be the issue.

Now my wife doesn't use Silverlight very much and I may have been able to get away with simply uninstalling it. I wanted to figure out the issue though and with that in mind continued on in my troubleshooting.

 It was update KB2890788 and was failing with error 0x80070643. This problem had started on September 8th with the update KB2847559 transitioning to the other update on the 10th.

Up to this point my wife had done the install on update option, so I thought I would try manually installing the update via the Windows Update control panel applet. Selecting only it and pressing install led to a "Preparing to install.." status for quite some time. In the "Application" event logs I was getting informational events from "Windows Error Reporting" basically telling me the update wasn't installing.

 

Never got an error on the Windows Update section, so I closed  it.

The Solution
I then manually downloaded the latest version of Silverlight from Microsoft's site. I then installed it and it upon completion it told me it was successful. You can see in the picture below the install log in the background as well as the successful completion message.

Upon completing the install I had it check for updates again and the update it had been failing on no longer appeared in the list.

So it appears that the install of Silverlight became corrupt and the update was unable to apply. A fresh install corrected the issue.

Read More

Easy way to find your public IP in scripts or CLI

There is a lot of online tools that gives your public IP but most of them are either not accepting cli User-Agent or they requires nasty parsing with AWK,grep,sed,etc

Best way to get your public IP address is to use ifconfig.me which only returns the IP simple and efficient.

 $ curl http://ifconfig.me  

The only problem is the responsiveness (few seconds).

Another cool way is to query the opendns servers :

 $ dig +short myip.opendns.com @resolver1.opendns.com

This method is pretty fast but requires the dig command (part of the bind-utils package).

Read More

Dell Firmware update fails with "mktemp: too many templates"

If you have the following error while updating a Dell server firmware (BIOS, RAID, etc) via Linux binary (*.BIN) :

 mktemp: too many templates  

Then check the binary's filename for specials characters, in my case Chrome added a "(1)" at the end of the filename. Remove it, restart the update process and you're good to go !

Read More

Linux server sends SYNACK packet only after receiving 8 SYN

Got a really weird issue recently, in some rare case (mostly Mac and Mobile phone clients), the connection to a linux server was really really slow (about 12s).

The issue was not only impacting Apache but all TCP services like SSH, hence it was not a particular service issue/misconfiguration.

The Chrome console on a MacBook Pro showed that the initial connection took about 10s, on the other hand a Win7 client in the same LAN had no problem at all.

After some digging on the client and server side, I found out that the client needs to send 8 SYN packets before the server replies with a SYNACK which explain why the connexion is so slow. Once the SYNACK is send back to the client, the communication speed is back to normal.

One hour headache later, it turn out that I enabled some Sysctl TCP tunning values that somehow introduced the issue.

I disabled the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse features and everything went back to normal.

I think the problem comes from the net.ipv4.tcp_tw_reuse option, but as the issue impacted a production service (and is really hard to reproduce) I didn't try to re-enable tcp_tw_recycle.

Some posts advice to disable window scaling, I strongly disencourage this as it would result in poor network performances.

Hope that helps !

Below the tcpdump output that shows the 8 client's SYN packets before the SYNACK is sent back. Test was performed on SSH service as you can see, the TCP handshake took 10 secondes.

 # SYN 1  
 15:57:26.303076 IP (tos 0x0, ttl 53, id 9488, offset 0, flags [DF], proto TCP (6), length 64)  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xdf5f (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835124724 ecr 0,sackOK,eol], length 0  
 # SYN 2  
 15:57:27.306416 IP (tos 0x0, ttl 53, id 37141, offset 0, flags [DF], proto TCP (6), length 64)  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xdb71 (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835125730 ecr 0,sackOK,eol], length 0  
 15:57:28.315804 IP (tos 0x0, ttl 53, id 2415, offset 0, flags [DF], proto TCP (6), length 64)  
 # SYN 3  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xd785 (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835126734 ecr 0,sackOK,eol], length 0  
 15:57:29.330233 IP (tos 0x0, ttl 53, id 62758, offset 0, flags [DF], proto TCP (6), length 64)  
 # SYN 4  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xd398 (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835127739 ecr 0,sackOK,eol], length 0  
 15:57:30.335779 IP (tos 0x0, ttl 53, id 29003, offset 0, flags [DF], proto TCP (6), length 64)  
 # SYN 5  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xcfa9 (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835128746 ecr 0,sackOK,eol], length 0  
 15:57:31.345254 IP (tos 0x0, ttl 53, id 5246, offset 0, flags [DF], proto TCP (6), length 64)  
 # SYN 6  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xcbba (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835129753 ecr 0,sackOK,eol], length 0  
 15:57:33.382242 IP (tos 0x0, ttl 53, id 5958, offset 0, flags [DF], proto TCP (6), length 64)  
 # SYN 7  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0xc3dc (correct), seq 2356956535, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 835131767 ecr 0,sackOK,eol], length 0  
 15:57:37.881881 IP (tos 0x0, ttl 53, id 21274, offset 0, flags [DF], proto TCP (6), length 48)  
 # SYN 8  
   client_ip.49316 > server_ip.ssh: Flags [S], cksum 0x5c3d (correct), seq 2356956535, win 65535, options [mss 1460,sackOK,eol], length 0  
 15:57:37.881907 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)  
 # SYNACK (at last !!!)  
   server_ip.ssh > client_ip.49316: Flags [S.], cksum 0x7a12 (correct), seq 3228952474, ack 2356956536, win 14600, options [mss 1460,nop,nop,sackOK], length 0  
 15:57:37.885362 IP (tos 0x0, ttl 53, id 62772, offset 0, flags [DF], proto TCP (6), length 40)  
 # ACK  
   client_ip.49316 > server_ip.ssh: Flags [.], cksum 0xdfde (correct), seq 1, ack 1, win 65535, length 0  

Read More

Juniper JunOS transfer on commit fails

I had quite a surprise when I discover that my transfer on commit stopped working on my SRX firewall.

The error in the logfile was :

 ACCT_XFER_FAILED: Error transferring /var/transfer/config/*  

Not really explicit...

Turn out that /cf/var/ was full, it needs at least some free MB to work properly, that sound weird as the config file only requires several KB.

The configuration is indeed copied to /cf/var/transfer/config/ before being transferred over the network. If /cf/var/ is full then the configuration cannot be copied and the transfer process finds nothing to send hence the error message above.

If you have the same error, cleanup some old logfiles and maybe decrease the amount of data your logging.

For any other errors, I recommend this post which explains all the other issues you may have with transfer on commit.

http://www.net-gyver.com/?p=655

Hope that helps !

Read More

Squirrel Mail crash with "PHP Fatal error: Call to undefined function sqimap_run_literal_command()"

I have encountered a bug on Squirrel Mail (version 1.4.8-21), the client's credentials are accepted but the page is then stucked on "webmail/src/redirect.php"

On the Apache side, I noticed the following PHP error :

  PHP Fatal error: Call to undefined function sqimap_run_literal_command() in /usr/share/squirrelmail/functions/imap_general.php on line 518  

After some debugging, I found out that Squirrel Mail does NOT properly handle specials characters; In this case I had an accent in the user password.
I reseted the user's password and everything went back to normal.

Hope that helps !

Read More

Bash : Wait for a command with timeout

Here is a very useful little command that wait for a process to finish and kill it if doesn't exit after a pre defined timeout.

The command is called timeout and is part of the coreutils package witch is embedded by default in CentOS 6.

Usage is very simple, you just need to specify the timeout, the process to start and optionally the signal to send in case of failure.

Timeout returns the exit value of the process and if the timeout has been reached, the value 124 is returned.

Here are some examples on how to use timeout :

Successful process : 

 $ timeout 10s ls /tmp  
 ....  
 ....  
 $ echo $?  
 0  

Unsuccessful process :

 $ timeout 10s ls /blabla  
 ls: cannot access /blabla: No such file or directory  
 $ echo $?  
 2  

Timed out process :

 $ timeout 5s sleep 10  
 $ echo $?  
 124  

More details in the man page.

Hope that helps !

Read More

CIP 007 R2 Ports and Services - Part 1 (Basic Port Enumeration)

Disclaimer: The information I provide is that of my own and does not reflect on the organizations I work for. The information I share should not be the only thing you rely upon for compliance and is provided as-is.

One of the areas of great concern for individuals involved with NERC CIP is CIP standard 007. It is the standard under which most of the work for compliance is done to secure assets. CIP 007 is also the most violated standard. A big offender is standard R2, Ports and Services. It is also a difficult one to comply with since there a large amount of services and ports per asset.

This topic will be broken into multiple posts due to the amount of information that can be shared. Ports and services will also be broken into different posts. I will begin by discussing some areas of overlap.

CIP 007 R2 states, "An entity will disable all unused ports and services not required for normal or emergency operation." The requirement for this standard is interesting. Let's discuss what this means in relation to ports first.

This relates to logical network ports like TCPport 25, which is most often used by SMTP. Ports, as related to compliance and security, considers only listening devices. For those new to network ports this means the network ports that are open and waiting for a connection from other devices or potentially from another process on that computer. NERC CIP does not currently have standards for physical ports on an asset. This does not mean physical ports should not be tracked. As part of its overall security standpoint a company should track and control the physical ports on their devices. Additionally this approach may serve the side benefit of ensuring information is collected when these ports do fall under a compliance standard of one type or another. A person or entity may decide that they want to define what a port and listening port is. As of the time I wrote this post, there was not yet a definition by NERC of either of these, but be sure to check that as new information surfaces daily.

There are multiple ways to collect information about a devices ports. Many devices are capable of listing the ports via their management face through a program or internal function. On a windows and linux based box, a person can run the netstat via the command line. Below is just a small sample of the output that may be seen from running "netstat -abno" windows based box.

 

This method is going to usually be the most accurate way to collect ports from a device that supports it.

Another way to collect ports from a device is to use a port scanning tool such as NMAP, Angry IP Scanner, etc. This software works by manually checking ports on the device to see if they are open. Usually the tool is run from a remote device. Remember to verify the full TCP and UDP port ranges 0-65535. Scanning only TCP is not sufficient.

Port scanners are usually the only method to collect ports from devices that do not have an internal method to collect ports information. An example of such a device may be a PLC. The output of an NMAP scan using the Zenmap GUI is shown below:

The benefits of using a port scanning tool is that a person can enumerate ports from devices that are incapable of enumerating ports natively. Additionally, all data is usually stored in one location. Some device vendors, like Allen Bradley, will provide software to scan their devices for open ports. In most cases a person or entity should use what their vendor requires.

There are some negatives to using a Port scanning tool.

• Port Scanning isn't always consistent, so a port may be missed. Try multiple scans to catch all listening ports on a device.
• If there are devices such as a firewall, in between the device conducting the scans and the device being scanned, reliable port data may not be retrieved.
• Some devices do not handle being scanned well. They may lock up, slow down, or function in methods that are not acceptable.
• This may mean that a person or entity would want to have an identical device in a non production network that can be re-configured in an identical manner. This device than can be scanned to collect port information.
• Possibly scanning the devices while not in production also may be necessary.
  

With all of that in mind I would recommend that when companies purchase new devices they ensure that they are capable of natively enumerating port data. There are many network switches, firewalls, control devices that can do this.

There will always be devices that cannot do this so make sure that the person or entity can reliably enumerate the ports of these devices via a Port Scan. Additionally some malicious software can hide listening ports from native commands such as netstat, so an occasional port scan may be useful for comparison purposes.

In the next NERC CIP related post I will share additional port enumeration information as well as methods to "disable" the ports and start to cover services.

Read More

Cobbler reposync failed

I run daily cobbler reposync crons and it appears sometimes the process fails with the following error :

 Exception occured: <class 'cobbler.cexceptions.CX'>  
 Exception value: 'cobbler reposync failed'  
 Exception Info:  
  File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 126, in die  
   raise CX(msg)  
   
 Exception occured: <class 'cobbler.cexceptions.CX'>  
 Exception value: 'cobbler reposync failed'  
 Exception Info:  
  File "/usr/lib/python2.6/site-packages/cobbler/action_reposync.py", line 125, in run  
   self.sync(repo)  
   File "/usr/lib/python2.6/site-packages/cobbler/action_reposync.py", line 169, in sync  
   return self.yum_sync(repo)  
   File "/usr/lib/python2.6/site-packages/cobbler/action_reposync.py", line 402, in yum_sync  
   utils.die(self.logger,"cobbler reposync failed")  
   File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 134, in die  
   raise CX(msg)  
...
!!! TASK FAILED !!!

I don't really have an explanation for this but it seems that the reposync process doesn't really fail as if I run the reposync command manually the process goes fine.

The createrepo command line is shown when you execute "cobbler reposync", for example :

 hello, reposync  
 run, reposync, run!  
 creating: /var/www/cobbler/repo_mirror/Dell-CentOS5/.origin/Dell-CentOS5.repo  
 running: /usr/bin/reposync -l -m -d --config=/var/www/cobbler/repo_mirror/Dell-CentOS5/.origin/Dell-CentOS5.repo --repoid=Dell-CentOS5 --download_path=/var/www/cobbler/repo_mirror  

In this case the command line is :

reposync -l -m -d --config=/var/www/cobbler/repo_mirror/Dell-CentOS5/.origin/Dell-CentOS5.repo --repoid=Dell-CentOS5 --download_path=/var/www/cobbler/repo_mirror  

The process should go fine, however the return value $? is '1' which can explain why the cobbler commands fails.

Read More

Emulate bad or WAN network performances from a particular IP on a Gigabit LAN network

If you're developing Web or mobile applications, you'll certainly be confronted to poor network conditions.

The problem now is "how can I test my application under bad network conditions". Well you could rent a forein internet connection or use tools that reports performance from various remote countries however this is not a good debugging environment.

The solution is to use TC and NetEM on your front development server (typically Web or reverse proxy server), then use filters so only one client station (the debugging station) is impacted.
Don't forget to use filter otherwise all your clients will be impacted.

Below an example on how to emulate a network with :

• 1Mbps bandwidth
• 400ms delay
• 5% packet loss
• 1% Corrupted packet
• 1% Duplicate packet

The debugging client IP is 192.168.0.42  (i.e the IP impacted by the bad network performance);
The following commands need to be executed on the front developement server, please set the appropriate NIC for you environment (eth0 used below) :

 # Clean up rules  
   
 tc qdisc del dev eth0 root  
   
 # root htb init 1:  
   
 tc qdisc add dev eth0 handle 1: root htb  
   
 # Create class 1:42 with 1Mbps bandwidth  
   
 tc class add dev eth0 parent 1:1 classid 1:42 htb rate 1Mbps  
   
 # Set network degradations on class 1:42  
   
 tc qdisc add dev eth0 parent 1:42 handle 30: netem loss 5% delay 400ms duplicate 1% corrupt 1%  
   
 # Filter class 1:42 to 192.168.0.42 only (match destination IP)  
   
 tc filter add dev eth0 protocol ip prio 1 u32 match ip dst 192.168.0.42 flowid 1:42  
   
 # Filter class 1:42 to 192.168.0.42 only (match source IP)  
   
 tc filter add dev eth0 protocol ip prio 1 u32 match ip src 192.168.0.42 flowid 1:42  

To check that the rules are properly set use the following commands :

 tc qdisc show dev eth0  
 tc class show dev eth0  
 tc filter show dev eth0  

Once you're done with the testing, cleanup the rules with the command :

 tc qdisc del dev eth0 root   

There is many other options you can use (correlation, distribution, packet reordering, etc), please check the documentation available at :

http://www.linuxfoundation.org/collaborate/workgroups/networking/netem

If this setup fits your requirements, I advice you to create a shell script so you can start/stop the rules with custom values. Be aware that you can also make filters based on source/destination ports, etc.

If you have more complex requirements, you can try WANem, which is a live Linux Distribution with a graphical interface on top of NetEM. Please be aware that this requires route modifications on your client and server (or any other routing tricks).

http://wanem.sourceforge.net/
http://sourceforge.net/projects/wanem/files/Documents/WANemv11-Setup-Guide.pdf

I didn't had the opportunity to try it, please let me know if you have any feedback.

Read More

Dell DRAC Console/KVM with Chrome or Firefox

Here is a really simple trick to access to your DRAC remote console (i.e virtual KVM) with Chrome or firefox.
This trick has been tested with DRAC 5,6 and 7 only.

Requirement : You need to have a working JRE

• Log in to your DRAC Web interface, go to "System -> Console Media"
• Clic on "Launch Virtual Console"
• The browser will ask you to open or save a file, save it on your Hard Drive
• The downloaded file has the form "viewer.jnlp(x.x.x.x@x@idrac-xxxxxxx,+xxxxxxxxx,+User-xxxxx@xxxxxxxxx)"
• Rename the file "viewer.jnlp" (i.e remove the garbage data after the extension)
• Double clic on the file and you're done.

Really easy but so handy !

Hope that helps

Read More

WebDAV on IIS7.5 with Windows 7 client

Install a cert (trusted) on the website, enable basic auth disable anon. Done. No need for app pool identity, etc.

Read More

Into to NERC CIP into my Blog

NERC CIP two acronyms  when placed together that strike fear into the heart of any that have been given the responsibility to abide by its requirements. NERC, North American Energy Reliability Corporation, is the entity that is in charge of developing and enforcing the standards used to ensure reliability of the North American bulk power systems. 

There are many standards that are under NERC's umbrella but if you are here reading this it is because you desire to know more about CIP. CIP, Critical Infrastructure Protection,  is the standards that relate specifically to the protection of the electrical control systems. CIP is the combination of standards that are also a bit of a touchy topic for many due to the feel by some that NERC is not doing enough to protect the electrical grid. To any who have had to abide by the full set of NERC CIP standards this is hard to believe since it requires a lot of work. I will not be debating the standards, although I feel that they are forcing many entities to make steps towards security, which is a good thing.

NERC CIP has been my life for the last year.  As with the rest of my blog the intent of this new period will be to share things I have learned to hopefully help all everyone else trying to accomplish their work. Before we get started  there are two tips I want to share.

1st - Compliance is not security and some things it requires will not make sense, but if it is required you must do it.

2nd - If you don't have evidence of something you did, you didn't do it. This is important to remember. A small portion of what you will do will be work, the other large portion of it will be collecting evidence and documenting your work.

This is the first post of many. One area I will be focusing on is the infamous CIP-007. I will still be doing my posts of troubleshooting and such, but NERC CIP is a big part of  my life now. So on to it.

 

For some good reading:

http://www.nerc.com/pa/CI/Comp/Pages/default.aspx

Read More

omreport : failed to load external entity "/opt/dell/srvadmin/var/lib/openmanage/xslroot//oma/cli/about.xsl"

If you're having the following error when executing omreport :

 I/O warning : failed to load external entity "/opt/dell/srvadmin/var/lib/openmanage/xslroot//oma/cli/about.xsl"  
 error  
 xsltParseStylesheetFile : cannot parse /opt/dell/srvadmin/var/lib/openmanage/xslroot//oma/cli/about.xsl  
 Error! XML Transformation failed  

Then install srvadmin-omcommon package :

 # yum install srvadmin-omcommon  

Read More

DRAC Firmware update failed : Error: 30001 Method httpCgiErrorPage()

Have tried to update an old DRAC4 Firmware from firmware 1.5 to 1.75 via Linux binary and came to an unplaisant surprise :

 Dell Remote Access Controller 4/P  
 The version of this Update Package is newer than the currently installed version.  
 Software application name: Dell Remote Access Controller 4/P Firmware  
 Package version: 1.75  
 Installed version: 1.50  
 Continue? Y/N:Y  
 Executing update...  
 WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE IS IN PROGRESS.  
 THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!  
 ......................................................................................
 /tmp/duptmp.xml:6: parser error : Extra content at the end of the document  
 <SVMExecution lang = "en">  
 ^  
 /tmp/.dellSP-XmlResult12908-32487.M19124:6: parser error : Extra content at the end of the document  
 <SVMExecution lang = "en">  
 ^  
 unable to parse /tmp/.dellSP-XmlResult12908-32487.M19124  
 /tmp/.dellSP-XmlResult12908-32487.M19124:6: parser error : Extra content at the end of the document  
 <SVMExecution lang = "en">  
 ^  
 unable to parse /tmp/.dellSP-XmlResult12908-32487.M19124  

Doesn't look good and of course if I try to access the DRAC via HTTPs, I've got a nice CGI error :

 Error: 30001 Method httpCgiErrorPage()  

Looked on the web and somebody (who contacted Dell Support) advises to shutdown the server, unplug the DRAC card for a while and plug it in back... Well explain to your CTO that you need to shutdown a production server, unrack it, unplug a card just because a DRAC update failed o_O
Reference: http://lists.us.dell.com/pipermail/linux-poweredge/2008-January/034556.html

The solution that worked for me was to install the racadm Dell tool on my bastion and reset the firmware remotely.

• First install racadm :

 # yum install srvadmin-racadm4.x86_64  

Note : This is for DRAC4, didn't had the issue with newer DRAC.
Note 2 : You need to have the Dell OSMA repository installed on your server:
http://www.openfusion.net/linux/dell_omsa

•  Then run the following command :

 # racadm -rDRAC_IP -i racreset  

Note : Change DRAC_IP with your DRAC IP.
Note 2 : This operation will NOT erase your DRAC configuration.

•  Wait a while, pray, and if you're lucky as me you should be back on line (with the original firmware version of course).

Final word, I stopped being lazy and updated the firmware via the Web GUI which is a long and annoying process. Of course I used Internet Explorer as I felt like Murphy's law was around this day ^^

Hope that helps !

Read More

Yum stuck/hangs at "Running Transaction Test"

If yum is stuck at the "Running Transaction Test" step, double check that you don't have a stalled network mount (NFS,SMB,etc) somewhere.

Umount it and retry your yum/rpm command.

More info on how to umount a stalled NFS share :
http://sysnet-adventures.blogspot.fr/2013/05/umount-stalledfrozen-nfs-mount-point.html

Read More

Umount a stalled/frozen NFS mount point

NFS is known to be a little nasty when it comes to umount stalled shares.

Most of the time a simple umount doesn't work, which is a bit frustrating specially when it comes to production servers; The process just hangs and there is no way to interrupt...

Below two procedures to umount stalled  NFS shares. You should try method one before method two as it requires some network "hacks".

Method 1 :

Use a forced lazy umount, this method works 90% of the time :

 # umount -f -l /mnt/nfs_share
Note : Don't use bash auto-completion !!!  

Method 2:

This method is to be used only if method one failed.

The trick is to temporarily steal the NFS server IP adress on the NFS client (the one with stalled mount) so this client thinks that the NFS server is still alive.

Warning : Use method 1 above if your NFS server is still reachable from the NFS client. Otherwise you'll have an IP conflit and trust me you really don't want that to happen.

Let's assume the NFS server IP is 192.168.0.1

1. Double check that the NFS server is down with ping or nmap.
2. If your NFS client has very restrictive IPTables rules shut them down temporarily
3. On the NFS client, set the NFS server IP as secondary address

 # ifconfig eth0:0 192.168.0.1  
 Note : Adjust interface to your own needs  

4. Umount the NFS with a forced lzay umount

 # umount -f -l /mnt/nfs_share  
 Note : Don't use bash auto-completion !!! 

5.  Check that the NFS mount is gone
6. Remove secondary interface
   

    # ifconfig eth0:0 down  
    Note : Adjust interface to your own needs  

7. Restart IPTables if needed
8. Be happy
9. Go to sleep, it's been a long day (or night)

 If you have multiple NFS clients stalled, you can set the secondary IP on one client only.

• Client 1 : Step 1 to 5

• Client 2 to n : Step 4 and 5

• Client 1 : Step 6 to 9

This will only work if your NFS client can communicate between each others (watch for IPTables or any other filtering softwares/devices).

Hope that helps ! (that helped me a lot :)

Read More

Remove absolute path from MD5 file

The following command will remove absolute paths from a md5 file :

  sed -i -r "s/ .*\/(.+)/  \1/g" file.md5  

This is quite useful when you download backup files which have MD5 containing absolut paths.

For example :

 $ cat file.md5  
 8ee6e966f2cb7a84620ed137604e00c5 /data/prod/file  

If you want to check this file on another server, you won't be able to do it unless you put the data file in the exact same directory (/data/prod/).

After running the above sed command, your md5 file will look like :

 $ cat file.md5  
 8ee6e966f2cb7a84620ed137604e00c5 file  

You can then check your md5 print with "md5sum -c file.md5". You just need to have the MD5 file in the same directory as the data file.

Read More

Whitelist files with Clamav

Exclude files from scan :

Sometimes, you will need to whitelist files from a scan, clamscan offers the --exclude option but its usage is not really user-friendly...

Imagine that you need to exclude two files /data/rep/file1 and /data/rep2/file2, the command line would be :

 # clamscan -r -i --exclude=/data/rep/file1 --exclude=/data/rep2/file2  

This is fine if you have few files to whitelist but it quickly becomes unreadable when you have dozen files and directories.

The solution is to input a file to clamscan with xargs. Create a text file containing all files/directories you need to whiltelist (one file/directory per line) :

 # cat /var/lib/clamav/whitelist-files.txt  
 /data/rep/file1  
 /data/rep2/file2  

You can also add regexp like *.mp3 (be aware that this is quite dangerous)

Run clamscan with the following command :

 # sed -e 's/^/--exclude=/' /var/lib/clamav/whitelist-files.txt | xargs clamscan -r -i /directory_to_scan/  

Don't forget to put double quotes or escape when you exclude paths with special characters (especially spaces).

Last but not least, always double check that the files you're whitelisting are completely safe. You can check that out with a meta AV engine like Jotti :
http://virusscan.jotti.org/en

Whitelist a virus signature :

To whitelist a virus a signature, you need to get the ClamAV signature definition, this is the code you have on the right side of the infected file line. For example :

 /data/file.flv: CVE_2012_0773-2 FOUND  

In this case the signature definition is CVE_2012_0773-2, add it to /var/lib/clamav/whitelist-signatures.ign2

That's all ! Be very cautious when whitelisting Virus signatures.
Hope that helps !

Read More

Multiple cluster per site, Site Recovery Manager (SRM) with svMotion capability

Here is a design VMware told me was 'unique' LOL. Sounded perfectly normal to me.

My plan was (obvious to me):
vSphere/SRM 5.0
2 locations
2 vCenters, of course required for SRM
2 clusters in each vCenter, one cluster at each location per vCenter.
Equallogic SRAs which probably doesnt matter who you have for a SAN SRA vendor.

This would give us sort of an 'X' configuration and allow us to svMotion/vMotion from location to location during business hours by simply mounting a common volume as well as have SRM capability across datacenters if bad things happened. I realize there is no longer a need for the common volume for svMotion in vSphere 5.1. Also keep in mind we have a lot of bandwidth between our sites. Your results may vary.

So the design looked valid, if interesting, on paper. All the svMotion worked fine. I just could not make the Array Managers 'see' or work with one pair of clusters within SRM. How this was visible, was that in my Array pair under site recovery for one of the vCenters, the Devices tab in the Array Pair would not show a 'Remote Device' and a 'Datastore' both for all the datastores I wanted to use. Only the ones associated with one pair. This meant I could SRM across one pair of clusters but not both

The resolution was to create a second Array Manager pair reversing the storage listings in both sites. So if in site A you had storage1 to storage2 and site B you had storage2 to storage1, that is enough to enable a single cluster per site. I had to go in to site A for example and add another pair for storage2 to storage1 and vise versa in site B giving me 2 pairs per site.

I hope this helps and I will elaborate if you want.

Charlie

Read More

Enable LDAP SSL/TLS user authentification in Zabbix

By default Zabbix Web interface doesn't offer the SSL/TLS encryption option for the LDAP connector however the feature is available in the PHP code.

If you need to enable the startTLS feature, you will have to edit the PHP file manually :

Edit /usr/share/zabbix/include/classes/class.cldap.php on Zabbix Web server :

 vi /usr/share/zabbix/include/classes/class.cldap.php  

Search for the 'starttls' definition (line 44) and set the value to 'true' :

 'starttls' => true,  

Save the file and you're good to go !

Hope that helps !

Read More

Dell 11-12th gen 710-720 firmware update observations

I have recently been playing around with a new install of the Dell Management vCenter plugin version 1.6. Very nice tool. Should save TONS of time. I also had to learn a bit about the server's firmware upgrade process now that the LifeCycle controller is involved. Here are a few things. Again this is either r710 or r720. I have been using both a lot lately.
_________________________________________________________
This error after a PERC firmware update:
PR1 PERC replaced part detected
This message is safe to ignore either way.
Enter lifecycle controller from the boot menu, and run an inventory to clear the message.
_________________________________________________________


I forget which update did this, probably the NIC firmware. The machine would not boot. The message was:
"Plug and play Configuration Error:
Option ROM Shadow RAM Allocation error."

The fix is to go into the NIC BIOS and disable all option ROMS for every NIC. Too many loading can cause this as there is not enough Shadow RAM to hold them all if you have many NICs. They apparently all get enabled with the firmware update.
_________________________________________________________


I will post more as I find them.
Hope this helps someone

Charlie

Read More

Updated*** Dell OpenManage 7.2 on ESXi5 fails with error 15

I was using VMware Update Manager but the method you use to install probably doesn't matter.

This appears to be the package is not signed correctly. Like the rest of you, I am busy so I didn't spend much time finding a fix. I went back to the 7.1 version of the agents and it installed no problem.

Initially I thought maybe the older 6.5 version might have been the problem and uninstalled it. The command to remove the old VIB, which was 6.5 for me, was esxcli software vib remove OpenManageThat didn't solve the problem however so I got smart, putty'd in and looked at the update log located at /var/log/esxupdate.log. I found this:
esxupdate: esxupdate: ERROR: InstallationError: ('Dell_bootbank_OpenManage_7.2-0000', "('Dell_bootbank_OpenManage_7.20000', 'Could not find a trusted signer.')")

I changed Update Manage to push out 7.1 and it installed fine.

If anyone has more info please share!

Hope this helps someone.

Charlie

Read More

Windows Server 2008R2 missing gateway IP address

OK so this one was fun.  Guess we can't test every scenario in the real world right?

We had several Windows 2008R2 servers that would not have an IPV4 gateway IP address after a reboot. If I entered one and saved it, then went back and looked, it would be gone again immediately. These machines were ESXi5 VMs but that should not have mattered, you'll see why.

I saw lots of articles about going into the registry and adding the gateway IP address when it disappears and cannot be added again normally. Nope. That should not be a fix anyway, more of a hack.
Other articles about deleting the hidden NICs, etc. Still no.

The issue for me was Symantec Enterprise Protection (SEP). Some of the servers here had a version 11 package with firewall, network protection, whatever, enabled. In my defense this was before my time. I do not typically let SEP handle the firewall on server OS's. We upgraded to version 12 package that did not have the firewall enabled. The upgrade from the management console does not call for a reboot because we're all about uptime these days right? Symantec is under the same pressure I'm sure. Well a week or 2 later when our scheduled maintenance patching came around, the machine booted. Bang, no gateway.

The 'no reboot required' install for SEP upgrades works fine normally except in the case when you go from firewall to no firewall. Evidently you need to remove the firewall while still in version 11, or remove SEP version 11 entirely, and reboot, then upgrade.

If you are already stuck with no gateway, like I was, here is the order of things to get back online.

1. Get a version 11 package with the firewall enabled onto a CD or ISO file that you can mount to the Windows OS (because you are unable to get on the network)
2. Uninstall SEP12, you can leave LiveUpdate on.
3. Reboot
4. Reinstall the SEP11 package with firewall
5. Reboot (you should get your gateway back here)
6. Uninstall SEP11 with firewall
7. Reboot
8. Reinstall 12, done.

According to Symantec support, this is a 'known issue' and will be fixed in the next version. How that happens when it is more a problem with the old version, I do not know. Also there have been several releases of version 12 already. If they were going to prompt to uninstall you would think it would have already been instituted.

Anyway hope this helps someone!

Charlie

Read More

Hello!

First post. 

I am brand new to blogging. No kidding right? The reason it came to this is I find a lot of interesting methods in administrating all things IT that I would like to share with others. Or remember for myself :) Or should be documented somewhere but I had to work to get so I'll share.

A little background. I have been in IT since 1998 or so. Before that I turned wrenches. Yeah long story. You can find me in LinkedIn also. I am an MCSE many times over since NT4, a 3 time VCP and have worked in some amazing environments.

I hope my posts help someone


Charlie

Read More

Shrew VPN Client + Juniper SRX : "session terminated by gateway" (Autodisconnect)

If like me, you're trying to connect to a Juniper dynamic VPN with Shrew VPN Client, be aware that this not yet possible.

The connection works but the tunnel is constantly disconnected after 60 seconds.

I asked the core developer "Matthiew Grooms" about this and after few debug, it seems like a fix is needed in Shrew's code:

"It's pretty clear whats going on but it won't be possible to fix without 
a rewrite of the modecfg code on the Shrew Soft VPN client, which is probably 
needed anyway."

Full technical details are available at :
https://lists.shrew.net/pipermail/vpn-help/2012-December/014091.html

If anybody found out an alternative solution please share !

Read More

Finding less used wireless channel on Linux

Want to find the less used wireless channel around you ? This little one liner will give a summary of all wireless channels with the number of SSID associated.

 echo "Nb SSID - Channel" ; iwlist scan 2>/dev/null | grep "Channel:" | cut -d':' -f2 | sort -n | uniq -c   
 Nb SSID - Channel  
       4 1  
       3 2  
       3 3  
       1 5  
       5 6  
       4 7  
       3 9  
       4 11  
       3 12  

Right column is the Channel number and left column is the number of SSID found on this channel.

 Of course, this is just a basic overview, for real deployments you should use a more sophisticated tool like inSSIDer.

Also an interesting post on how to choose the right channel :
http://www.dslreports.com/faq/14250

Read More

Set scale in (non interactive) scripts with the bc command

If you need to do some kind of math operation in shell scripts you might want to write :

 $ echo "9/2" | bc  
 4  

By default bc truncate the result which is bit annonying...

Using bc -l option gives you a far too precise result :

 $ echo "9/2" | bc -l  
 4.50000000000000000000  

Fortunately, bc comes with the "scale" option to set the scale to whatever presision you wish.

To do it non interactively, you need to specify the scale in the echo before the operation.
For example, to set a precision of 2 digits :

 $ echo -e "scale=2 \n 9/2" | bc  
 4.50  

So the script code would look like :

 val=$(echo -e "scale=2 \n 9/2" | bc)  

Read More

Replace end of lines (\n) by any character

Just found out a classic UNIX command I never used before.

The command "paste" allows you to replace end of lines (i.e \n) by any character of your choice.

I normally used sed for this purpose but the syntax is quite...dirty. For example to replace all end of lines by a space with sed, the command is :

 $ sed ':a;N;$!ba;s/\n/ /g' /path/to/file  

With paste the syntax is much more human readable :

 $ paste -s /path/to/file  

By default "paste" replaces end of lines with tabs, to specify a delimiter use the -d option.

Replace end of lines with spaces :

 $ paste -s -d' ' /path/to/file  

Replace end of lines with commas :

 $ paste -s -d',' /path/to/file  

 Use '-' for reading from stdin :

 echo -e "a\nb\nc\nd" | paste -s -d',' -  
 a,b,c,d  

More info in the man as always.

This tips will not change your life but i found it quite useful !

Hope that helps !

Read More

Choosing RAID Level / Stripe Size

Below interesting articles on how to choose your RAID level / stripe size.

Good litterature on RAID :
http://www.fccps.cz/download/adv/frr/hdd/hdd.html

RAID Level Explained :
http://www.techrepublic.com/blog/datacenter/choose-a-raid-level-that-works-for-you/3237

RAID Stripe Explained :
http://www.anandtech.com/show/788/5

RAID Benchmarks :
https://raid.wiki.kernel.org/index.php/Performance

RAID Calculator :
http://www.z-a-recovery.com/art-raid-estimator.htm

In any case, always plan your workload type (Read/Writes, Sequential/Random, Large/Small File, Number of concurrent access).

Read More

Create large partitions on Linux / Bypass the 2TB partition Limit

The default partition schema (MBR based) limits partition to 2.2TB. With new hardrives this limit is easily reached.

In order to create partition bigger than 2.2TB you need to switch from MBR to GUID (GPT) partition table.
This can be done with the "parted" utility on Linux.

For exemple if you want to create a single big partition on /dev/sdb :

 # parted /dev/sdb  
 (parted) mklabel GPT  
 (parted) mkpart partition_name fstype 1 -1  
 (parted) print  
 Model: DELL PERC H700 (scsi)  
 Disk /dev/sdb: 4000GB  
 Sector size (logical/physical): 512B/512B  
 Partition Table: gpt  
 Number Start  End   Size  File system Name Flags  
  1   1049kB 4000GB 4000GB        data  

Note : I found out that partition name and fstype are quite useless.

You can then format the partition with the filesystem of your choice or create a LVM PV.

More info on GUID / MBR Limits :
http://en.wikipedia.org/wiki/GUID_Partition_Table

Parted official website :
http://www.gnu.org/software/parted/

More parted exemples :
http://www.thegeekstuff.com/2011/09/parted-command-examples/

Hope that helps ! 

Read More

Omreport doesn't update disk rebuild progress

Had to replace a hard drive on a Dell Server and omreport rebuild progress got stuck at 1%.

The solution is to restart the srvadmin service :

 # srvadmin-services.sh restart  

This is quite dirty but it's the only solution I found. This also happened when I changed a PERC H700 battery.

Another way to check the rebuild process is to export log with omconfig :

 # omconfig storage controller action=exportlog controller=0  

This creates a /var/log/lsi_MMDD.log file, with the rebuild progress :

 03/09/13 22:07:51: EVT#13296-03/09/13 22:07:51: 99=Rebuild complete on VD 01/1  
 03/09/13 22:07:51: EVT#13297-03/09/13 22:07:51: 100=Rebuild complete on PD 05(e0x20/s5)  
 03/09/13 22:07:51: EVT#13298-03/09/13 22:07:51: 114=State change on PD 05(e0x20/s5) from REBUILD(14) to ONLINE(18)  
 03/09/13 22:07:51: EVT#13299-03/09/13 22:07:51: 81=State change on VD 01/1 from DEGRADED(2) to OPTIMAL(3)  
 03/09/13 22:07:51: EVT#13300-03/09/13 22:07:51: 249=VD 01/1 is now OPTIMAL  

Same thing for the battery learn cycle.

Hope that helps !

Read More

Dell Openmanage/Omreport failed after updating to CentOS 6.4

After updating a testing machine from CentOS 6.3 to 6.4, the Dell OpenManage tools stopped working AT ALL.
It seems that with the lastest CentOS kernel (2.6.32-358.2.1.el6.x86_64), they moved away some IPMI drivers from kernel modules to "built-in"

The result is :

 # omreport chassis  
 Health   
 # srvadmin-services.sh start  
 Starting Systems Management Device Drivers:  
 Starting dell_rbu:                     [ OK ]  
 Starting ipmi driver:                   [FAILED]  
 Starting Systems Management Device Drivers:  
 Starting dell_rbu: Already started             [ OK ]  
 Starting ipmi driver:                   [FAILED]  
 Starting DSM SA Shared Services:              [ OK ]  
 /var/log/messages reports :   
 instsvcdrv: /etc/rc.d/init.d//dsm_sa_ipmi start command failed with status 1  

Solution : 

 # yum install OpenIPMI  

Note : There is no need to start or chkconfig the service.

You can check that the IPMI components are seen with the following command :

 # service ipmi status  
 ipmi_msghandler module in kernel.  
 ipmi_si module in kernel.  
 ipmi_devintf module loaded.  
 /dev/ipmi0 exists.  

Then start Openmanager services :

 # srvadmin-services.sh start   

Read More

Problems sending Messages via Voice with Windows Phone 8

Problem

So I started having issues after moving to Windows Phone 8 sending texts via voice. I would press and hold down the Windows key to get it to accept my command.

After doing this I would state my command of "Text (Person's Name)" and if I said it without food in my mouth it would recognize the command and show the next dialog box.

It would then bring up the next screen giving me the option to "say your message"

After stating my message whatever it was, in this case it was "Purple people eater" then it would show me this screen.

Now if I continue to do everything by voice as well as sending then everything works, but their are instances where I manually want to add some additional text that it is having a hard time recognizing. To do this pressing on the pencil in the middle, aka "Edit", allows you to do just that and brings up the next screen. I can then add what I want and then press the "Send" button.

 The problem is that I then get an error shortly after sending it.

If you have a sharp eye then you noticed in the last two pictures it is actually showing two recipients. Now I never added the second recipient. When you press "Edit" it for some reason adds just the last name of your recipient. This last name does not have any number associated so it errors out although for some reason it thinks it has an email address of the last name as shown below:

 

This can be re-created as many times as you want. I am using Windows Phone 8 running on a Nokia Lumia 920 running the portico the latest update through ATT. Now enough with the problem.

 

Solution



So I started thinking it was odd that it was just the last name. I remember there was a way to change how contacts were displayed in the "people" hub. So if you go under "Settings" and then "Applications." Clicking on "people" brings up the people settings.

Now changing the "Sort list by" either to "First name" or "Last name" results in the same behavior. So I decided to change the "Display names by" setting from "Last, First" to "First Last."

 

Upon testing the same procedure show above, clicking on "Edit" shows the following message screen.

As you can see above there is only one name now. Sending this message does not result in any error being displayed later.

 

So if you are facing the follow behavior you can either change your "Display names by" to "First Last" or you can delete the last name of your contact out of the "To:" field each time. Either way I hope this gets fixed in a future patch.

 

Sorry for the large number of images, just wanted to properly document this bug.

 



 

 

Read More

Communication Error when Printing PDFs from Windows 8 Reader

The Problem

I recently installed Windows 8 and did an upgrade from Windows 8. This worked great and I have been enjoying it for the last couple of days. Today I ran into an issue though. I was trying to print out a PDF using Microsoft's Reader.

With Windows 8 you do this by bringing up the charm bar on the right of the screen in the app you want to print and clicking on devices.

To bring up the charm bar for a device with keyboard/mouse press "Windows Key + C" or take your mouse to the very upper-right or lower-right of the screen and it will appear

After clicking on Devices it should display your printer as well as some other devices. You then click on your printer. After doing this you should get a Print Preview as well as some options that allow you to print out the document.

In my case though instead of getting the preview as you see to the left I got an error, which I wish I had print screened but sadly didn't. It stated something along the lines of not being able to communicate with the printer. This was odd for me and I guessed it had some issue with the driver. The odd thing though was that when I was in the "Desktop" mode than the printer printed fine. I guessed then that the driver from Windows 7 was not compatible with the Metro section of Windows 8.

This was odd to me since the print drivers for Windows 8 are awesome in that they are supposed to provide basic print functionality for pretty much every printer. Meaning that unlike what many people did when they upgraded to Vista or 7 you don't need to purchase a new printer. You can read more about this here: MSDN Blog Article: http://blogs.msdn.com/b/b8/archive/2012/07/25/simplifying-printing-in-windows-8.aspx

The Solution

I went into the printer properties for my printer and it showed the right driver. I then deleted my printer. I then went to re-added my printer. I noticed one of the other ways didn't work so the way that worked for me to find my printer was to open "Devices and Printers" and click on "Add a Printer"

My printer was then discovered and I added it. It then used one of the class drivers which provide compatibility with printers that are not compatible with Windows 8, called "Brother Laser Leg Type2 Class Driver". This was for a Brother HL2070N.

I thought that the fact that Windows 8 had picked this driver I was probably in a better place. Sure enough opening my PDF, bringing up the charm bar then devices and clicking on my printer brought up the print preview and allowed me to successfully print out my PDF.

So my recommendation for people having printer problems would be to delete them and then re-add them. Windows 8 will select the best driver and you should be good to go.

Hope this helps everyone using the Microsoft Reader, athough I am guessing this would have been a problem with Adobe Reader too if it is metro compatible.

 

Read More