January 20th, 2010 by martinh
It doesn’t surprise us that three of InfoWorld’s 2010 Technology of the Year Awards involved MapReduce, Hadoop and NoSQL technologies. We want to congratulate our partners at Amazon Web Services for extending “to just about every dimension of cloud computing currently known.” And congratulations to the Apache Foundation for its Apache Hadoop, an open source solution for managing very large, parallel processed datasets. As InfoWorld puts it, “Though the concept is simple, the result is powerful.”
November 9th, 2009 by martinh
Our lead engineer, Shevek, gave a 30 minute presentation at the October 21st Bay Area HUG meeting on Yahoo!’s campus. It’s a pretty good introduction to the studio and its ability to help you prototype, deploy and debug Hadoop jobs and to monitor and view the job, cluster and associated file systems on a regular Hadoop distribution or Amazon’s Elastic MapReduce.
November 6th, 2009 by martinh
We came across the stealth mode guys at MapR Technologies recently. They describe what they’re doing as “engineering game-changing Map/Reduce related technologies”. They’re conducting a survey with some interesting questions for Hadoop users. Taking the survey buys you coffee for a few days.
November 2nd, 2009 by martinh
Thanks to lots of use and feedback over the past few weeks, we have made further improvements to the Studio tool. In addition to various bug fixes and performance improvements, there are two new pieces of functionality to draw attention to:
- Job and Task Counters
- File System Status
As usual, we’re interested in feedback either on the mailing list or in private email. For new users, this is available as the current installable version. For existing users, you should get it automatically, depending on your update check interval.
Hadoop Job and Task Counters
We’ve had a lot of requests for the display and visualization of task and job counters. So new to this release is the ability to view and graphically chart counters in Hadoop jobs. You can also save and re-use custom chart templates, which enables you to build your own general, job and/or task-specific counter dashboards.
File System Status
We’ve also added graphical viewing of cluster file systems so you can see their status at a glance.
October 26th, 2009 by martinh
The Bay Area Hadoop User Group is a thriving community that we mingled with last week. Thanks to Dekel for the invitation and squeezing Shevek’s presentation in at the last minute. We know we really ought to get a screencast done but, in the meantime, here are the slides that Shevek presented.
October 14th, 2009 by martinh
We’ve been bowled over by the response to the announcement of Karmasphere Studio’s product beta at Hadoop World. A ton of people have downloaded and started using it.
The most common question we’ve received is “why NetBeans?”. To our knowledge, it hasn’t been a roadblock to anyone using it but the simple answer is “it was a convenient vehicle to launch our first product”.
More features and functionality are just around the corner. If you haven’t already, please give us feedback. New stuff is based on what we’re being asked for.
October 14th, 2009 by dave
We have published a set of tutorials to help users get started and oriented with Karmasphere Studio’s capabilities.
Job Development and Debugging on the Desktop with the Job Developer
Job Deployment, Monitoring and Debugging on a Hadoop Cluster with the Hadoop Manager
Job Deployment, Monitoring and Debugging on Amazon Elastic MapReduce and S3 with the Hadoop Manager
October 1st, 2009 by martinh
Today we’re publishing the first beta version of Karmasphere Studio for Hadoop. It’s available for free as a plugin to NetBeans.
From your Mac, Linux or Windows desktop, you can:
- Graphically develop and debug MapReduce applications for Apache Hadoop.
- Deploy, monitor and debug your jobs in real-time
- Work with clusters running every major version of Hadoop (0.18, 0.19 and 0.20)
- Deploy, debug and monitor your jobs and manage clusters in the cloud using Amazon Elastic MapReduce.
- Use HDFS and Amazon S3 file systems.
- Monitor jobs on clusters and browse HDFS file systems via SSH tunnel or SOCKS proxy.
In the Job Developer, you prototype your MapReduce job without initially needing a cluster, since Hadoop emulation is included. Your job is visualized as a workflow, which can show the results of individual phases when you provide a local sample data set. Edit your code and see the results immediately.
The Hadoop Manager enables you to work with jobs, clusters and file systems. From here you can deploy jobs to one or more clusters, use local, HDFS and Amazon S3 file systems, see a graphical real-time profile of your cluster capacity and job usage, and dive deeper into the status of your job while it’s running.
Cluster integration with Amazon Elastic MapReduce provides insight into all your jobs and temporary clusters, including the ability to browse the cluster’s HDFS and access job logs.
The product also includes through-the-firewall support so that you can monitor jobs on clusters and browse HDFS file systems via SSH tunnel or SOCKS proxy.
We hope you’ll try it and let us know what you like, what you’d like to see changed or added and, since it’s a beta, tell us any problems you have.
October 1st, 2009 by martinh
For those wanting to run jobs in the cloud without setting up their own Hadoop distribution, Amazon Elastic MapReduce gives you the opportunity to spin up a cluster, small or large, for a job when you need it. And to pay for only as much as you use.
This seemed like a natural fit with Karmasphere Studio. So, in the Beta product, we’ve incorporated easy deployment to Elastic MapReduce clusters, support for the Amazon S3 file system and deep insight into your clusters and jobs running on Amazon Elastic MapReduce.
- See the status and progress of all your jobs,.
- Profile your cluster capacity and job usage graphically.
- Examine job steps and logs in real time.
- Browse the cluster’s HDFS file system.
- Access files on Amazon S3
You can use the Amazon Elastic MapReduce functionality with jobs developed inside or outside Karmasphere Studio, You can use it as a desktop console for jobs you’ve deployed on Amazon Elastic MapReduce from anywhere, including Karmasphere Studio, the Elastic MapReduce console or command line tools.
September 27th, 2009 by Karmasphere
We’ll be at Hadoop World with more news about Karmasphere Studio for Hadoop.
We’re especially interested in hearing about all the enterprise uses of Hadoop popping up and the needs of Hadoop job developers as we evolve the Studio product.
If you want to learn more from us face-to-face at the event, send us an email and we’ll try to coordinate a meeting.
