Karmasphere Study Shows Hadoop Projects Start As Skunkworks; Quickly Move To Active Development And Production

Also Projects 50-60% Increase in Hadoop Developers

Menlo Park, CA – September 29, 2010 – Karmasphere™, the leader in Big Data Intelligence software, today announced the results of a proprietary survey of Hadoop developers and users that reveals insights into the speed of Hadoop adoption and usage trends, growth in Hadoop developers, specific reasons for selecting Hadoop, challenges faced by developers and shift in language usage. The study shows that nearly 68% of Hadoop developers said their Hadoop projects were started as skunkworks, and within a year 86% of projects have moved into either active development or production. Additionally, organizations are finding that the longer Hadoop is used, the more useful it is found to be; 65% of organizations using Hadoop for a year or more indicated more than three reasons for using Hadoop, as compared to 36% for new users. The study also projects a 50-60% increase in the number of Hadoop developers within organizations already using Hadoop.
“This data confirms what many of us in the industry had long suspected; the adoption of Hadoop by enterprises is happening rapidly, and the more exposure developers and analysts have to Hadoop, the more reasons they find to use it,” commented Martin Hall, chief executive of Karmasphere.

When asked to name the top reasons for using or evaluating Hadoop, to “mine data for improved business intelligence” was cited as the main reason by 19% of Hadoop developers, “reducing cost of data analysis” was cited by 15%, and “log analysis” was cited by 13%.

Challenges to Overcome

According to the study, the top four general challenges in using Hadoop are: “steep learning curve” (44% cited), “hiring qualified people” (34%), “availability of appropriate products and tools” (33%), and “having enough information on how to get started” (32%). Seventy (70%) percent of Hadoop developers believe the challenges they are experiencing will impact their Hadoop implementations.
When it comes to specific Hadoop programming challenges, 63% cited “debugging Hadoop jobs” and 47% cited “monitoring Hadoop jobs” as the primary challenges they face.

“As with any new technology, there are challenges to overcome. When we wipe away those challenges, with tools such as ours, we should see growth accelerate further,” commented Hall.

Growth in Hadoop Developers; Shift in Language Usage

The study also indicated a dramatic growth in the number of developers using Hadoop over the next year, growing between 50-60%. When it comes to Hadoop language and analytics use, Java will remain the primary language, 85% Hadoop developers cited it as the primary language today with 74% expecting it to be the primary language next year. While growth is expected in the usage of Hive/SQL, moving from 44% today to 52% next year, the usage of Pig and Streaming is expected to remain constant. Usage of Mahout is expected to jump from 14% usage today to 24% usage in a year.

About the Study

The study gathered information from 102 Hadoop developers and users, during the week of August 30, 2010. (Error margin: ±9.7%)
For survey results, please click here.

About Karmasphere

Karmasphere is a big data intelligence software company bringing Apache Hadoop power to developers and analysts. Karmasphere enables companies to unlock the competitive advantages within their large datasets by providing an easy-to-use class of client-side software. Karmasphere’s products, built around the Karmasphere Application Framework™, feature independence across any Hadoop environment, easy one-click deployment across any cloud/cluster, and a rich and friendly user-interface to maximize productivity, discovery and insight. For more information please see www.karmasphere.com.

Note: Karmasphere, Karmasphere Application Framework and Karmasphere Studio are trademarks of Karmasphere. All other trademarks are property of their respective owners.