Solution to Cloudera SCM Express libpython2.4.so.1.0 problem

Ran into a strange issue installing Cloudera’s Hadoop Distribution using SCM Express on a cluster of 64-bit CentOS 5.7 machines recently.  

Install process would get to Hue common/proxy packages, fail, then proceed to roll back. I saw a number of similar reports in various message groups but no solution, so we are documenting our experience here:

Installing : hue-common 2/2/usr/share/hue/build/env/bin/python2.4: 
error while loading shared libraries: libpython2.4.so.1.0: cannot open 
shared object file: No such file or directory 

Target machines had Python (64-bit library) installed:

[root@machine1 /]# locate libpython2.4.so.1.0 
/usr/lib64/libpython2.4.so.1.0 

… but it appeared as though the installation scriplet was expecting the 32-bit version.  Logs revealed that the script was trying to install both 32-bit and 64-bit packages that would then stomp on each other: 

Installing Hue common package… 

BEGIN yum list installed hue-common 
Loaded plugins: fastestmirror 
Error: No matching Packages to list 
END (1) 
BEGIN yum list available hue-common 
Loaded plugins: fastestmirror 
Loading mirror speeds from cached hostfile 
* base: mirror.net.cen.ct.gov 
* extras: mirror.umd.edu 
* updates: mirror.rackspace.com 
Available Packages 
hue-common.i386 1.2.0.0+114.4-2 cloudera-cdh3 
hue-common.x86_64 1.2.0.0+114.4-2 cloudera-cdh3 
END (0) 
BEGIN yum -y install hue-common 
Loaded plugins: fastestmirror 
Loading mirror speeds from cached hostfile 
* base: mirror.net.cen.ct.gov 
* extras: mirror.umd.edu 
* updates: mirror.rackspace.com 
Setting up Install Process 
Resolving Dependencies 
–> Running transaction check 
—> Package hue-common.i386 0:1.2.0.0+114.4-2 set to be updated 
—> Package hue-common.x86_64 0:1.2.0.0+114.4-2 set to be updated 
–> Finished Dependency Resolution 

According to Cloudera’s support team, there was a bug in the initial set of CDH3u2 packages that caused both the 32-bit and 64-bit packages to be installed, but this was fixed a few days after the initial packages were put out there.  For some unknow reason (oudated cache somewhere?) our installation process was running into the same bug.

Solution/workaround suggested by the support team – to add to the [main] section of your /etc/yum.conf the following line:  

exclude=*.i386 *.i586 *.i686

Note: this needs to be done before running any SCM instal scripts.  As a result, SCM only pulls 64-bit versions of the packages – and you avoid Python library conflicts.

Also – Cloudera tech support team is very responsive.

Other known CDH3 bugs and workarounds can be found here.

UPDATE: someone else also struggled with this issue, and by doing a remove of the i386  package and reinstalling the x86_64, they got Hue running without having  to reinstall SCM completely. 


December 19 2011 update: based on our Google Analytics reports, many companies are struggling with similar installation issues.  iTrend will be happy to assist you with your Big Data initiative – we have all necessary engineering resources in-house.  Email your requirements to info@itrend.tv and mention this blog.

 

Technologist, parallel entrepreneur. Interests: travel, photography, big data, analytics, predictive modeling.

Tagged with: , , , , , ,
Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: