Apache Hadoop 0.23 is here, with significant scalability enhancements

This version offers several interesting improvements over the previous stable release, affecting both HDFS and Map Reduce – and the framework’s ability to scale:

  • HDFS Federation

    Lets you use multiple independed Namenodes/namespaces and scale the name service horizontally.  Namenodes are independent and don’t require coordination with each other. Datanodeas are used by all Namenodes as common storage. 

    Key benefits, according to the release document on apache.org:

     

    • Namespace Scalability – HDFS cluster storage scales horizontally but the namespace does not. Large deployments or deployments using lot of small files benefit from scaling the namespace by adding more Namenodes to the cluster
    • Performance – File system operation throughput is currently limited by a single Namenode. Adding more Namenodes to the cluster scales the file system read/write operations throughput.
    • Isolation – A single Namenode offers no isolation in multi user environment. An experimental application can overload the Namenode and slow down production critical applications. With multiple Namenodes, different categories of applications and users can be isolated to different namespaces.

    Screen_shot_2011-11-16_at_10

  • MapReduce NextGen aka YARN aka MRv2

    The new architecture splits JobTracker functionality into separate components: resource management (assignment of computer resources to applications) and job life-cycle management (application scheduling and coordination).

    Screen_shot_2011-11-16_at_10

These should let you do more with the same infrastructure, Hadoop needed this.

Also – check out the following presentation by Hortonworks founder, Arun C. Murthy.

 

 

Technologist, parallel entrepreneur. Interests: travel, photography, big data, analytics, predictive modeling.

Tagged with: , , , ,
Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: