Earlier this month we announced that Dell would be open sourcing the Crowbar “barclamps” for Hadoop. Well today is the day and the code is now available at our github repo.
Whats a Crowbar barclamp?
If you haven’t heard of project Crowbar it’s a software framework developed at Dell that started out as an installation tool for OpenStack. As the project grew beyond installation to include monitoring capabilities, network discovery, performance data gathering etc., the developers behind it, Rob Hirschfeld and Greg Althaus, decided to rewrite it to allow modules to plug into the basic Crowbar functionality. These modules or “barclamps” allow the framework to be used by a variety of projects. Besides the OpenStack and Hadoop barclamps written by Dell, VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.
To help you get your bearings
As I mentioned in the opening paragraph, the code for the Hadoop barclamp is now available. To help you get started, below are a couple of videos that Rob put together. The first walks you through how to install Crowbar and the second one explains how to use Crowbar to deploy Hadoop.
I’m always interested in what’s happening at Canonical and with Ubuntu. Last week at Hadoop World I ran into a couple of folks from the company (coincidentally both named Mark but neither Mr. Shuttleworth). Mark Mims from the server team was willing to chat so I grabbed some time with him to learn about what he was doing at Hadoop World and what in the heck is this “charming” Juju?
Some of the ground Mark covers
Making the next version of Ubuntu server better for Hadoop and big data
(0:34) What are “charms” and what do they have to do with service orchestration
(2:05) Charm school and learning to write Juju charms
(2:54) Where does “Orchestra” fit in and how can it be used to spin up OpenStack
(3:40) What’s next for Juju
But wait, there’s more!
Stay tuned for more interviews from last week’s Hadoop world. On tap are:
One thing Hadoop isn’t great at right out of the box is data analytics, that’s where a company like Karmasphere comes in. Karmasphere provides business intelligence software that data analysts can use to use to mine the data that Hadoop sucks up.
Last week at Hadoop World I grabbed some time with Karamsphere’s Chairman and co-founder, Martin Hall to learn more about where he and his company play in the wild world of big data.
Some of the ground Martin covers
Where does Karmasphere play in the big data stack, how is it used and by whom
(0:38) Where did the idea for developing Karmasphere come from
(1:58) What is the Karmasphere “secret sauce”
(2:18) What are the main industries and use cases where their offerings are used
(3:40) What can we look forward to in future releases
But wait, there’s more!
Stay tuned for more interviews from last week’s Hadoop world. On tap are: Mark Mims of Canonical, Todd Papaioannou from Battery Ventures, John Gray of Facebook, Erik Swan of Splunk and Nosh Petigara of 10gen/MongoDB.
It wouldn’t be surprising if you were surprised to learn that Dell is developing software. To say that this is an area we haven’t been known for in the past would be an understatement. While we may not pose a direct threat to Microsoft any time soon, we have been coding in a few focused areas. One of those areas is cloud installation and management and is represented by our project Crowbar. While Crowbar began life simply as a way to install Openstack on Dell hardware, it has expanded from there.
Today’s news is that we have developed and will be opensourcing “barclamps” (modules that sit on top of crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop. All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters. But don’t take my word for it. Take a listen to Crowbar’s architect Rob Hirschfeld as he explains Crowbar and today’s announcement:
Who owns your data? Hopefully the answer is you and while that may be true it is often very difficult to get your data out of sites you have uploaded it to and move it elsewhere. Additionally, your data is scattered across a bunch of sites and locations across the web, wouldn’t it be amazing to have it all in one place and be able to mash it up and do things with it? Jeremie Miller observed these issues within his own family so, along with a few friends, he started the Data Locker project and Singly (Data Locker is an open source project and Singly is the commercial entity behind it).
I caught up with Jeremie right after the talk he delivered at OSCON. Here’s what he had to say:
Some of the ground Jeremie covers:
The concept behind the Data Locker project, why you should care
How the locker actually works
The role Singly will play as a host
Where they are, timeline-wise, on both the project and Singly
Dell has been a part of the OpenStack community since day one a little over a year ago and today’s news represents the first available cloud solution based on the OpenStack platform. This Infrastructure-as-a-service solution includes a reference architecture based on Dell PowerEdge C servers, OpenStack open source software, the Dell-developed Crowbar software and services from Dell and Rackspace Cloud Builders.
Crowbar, keeping things short and sweet
Bringing up a cloud can be no mean feat, as a result a couple of our guys began working on a software framework that could be used to quickly (typically before coffee break!) bring up a multi-node OpenStack cloud on bare metal. That framework became Crowbar. What Crowbar does is manage the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in a matter of hours (or even minutes) instead of days.
Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
Code to the Community
As mentioned above, today Dell has released Crowbar to the community as open source code (you can get access to it the project’s GitHub site). The idea is allow users to build functionality to address their specific system needs. Additionally we are working with the community to submit Crowbar as a core project in the OpenStack initiative.
Included in the Crowbar code contribution is the barclamp list, UI and remote API’s, automated testing scripts, build scripts, switch discovery, open source Chef server. We are currently working with our legal team to determine how to release the BIOS and RAID which leverage third party components. In the meantime since it is free (as in beer) software, although Dell cannot distribute it, users can directly go the vendors and download the components for free to get that functionality.
More Crowbar detail
For those who want some more detail, here are some bullets I’ve grabbed from Rob “Mr. Crowbar” Hirschfeld’s blog:
Important notes:
Crowbar uses Chef as it’s database and relies on cookbooks for node deployments
Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as “barclamps.”
Each barclamp has it’s own Chef configuration, UI subcomponent, deployment configuration, and documentation.
On the roadmap:
Hadoop support
Additional operating system support
Barclamp version repository
Network configuration
We’d like suggestions! Please comment on Rob’s blog!
The next in my series of videos from the Hadoop Summit features Cloudera‘s Vice President of product, Charles Zedlewski. If you’re not familiar with Cloudera you can think of them as the Red Hat of Hadoop world.
I sat down with Charles to learn more about Cloudera, what they do and where they came from.
Some of the ground Charles covers:
Cloudera’s founding, what its original goals and vision were and where its founders came from.
(1:35) What Cloudera does for customers 1) packages Hadoop and 2) helps them run it in production environments.
(3:27) What channels Cloudera leverages and where they play in the ecosystem
(4:11) Charles’ thoughts on the Yahoo spin-out Hortonworks and how it might affect Cloudera.
Tonight at the opening reception for Cloud Expo, I ran into Peder Ulander, CMO of Cloud.com. We found a quiet spot off the show floor and I got him to tell all about Cloud.com, where they’ve been and where they’re going.
Some of the ground Peder covers
What is cloud.com, where does it play in the cloud ecosystem and what does it help customers do?
[01:22] Who are some of Cloud.com’s customers (hint: Nokia, Zynga, Korean Telecom…) and in what industries are they in?
[03:25] Where did the idea for cloud.com come from and what experience did the founders leverage in creating it?
Last week at the OpenStack Design Summit I met up with Erica Brescia, CEO of BitRock who was accompanied by BitRock’s founder and CTO Daniel Lopez Ridruejo. Although BitRock’s recently announcedBitNami Cloud Hosting currently runs solely on Amazon’s EC2, Erica and Daniel were at the OpenStack event to see if OpenStack might make sense as a second platform.
I grabbed some time with Erica and we did a quick interview and demo:
Some of the ground Erica covers:
BitRock, facilitating the installation of software. Who are the big open source players that BitRock makes easy to deploy.
Bitnami.org providing end users with completely free native installers, virtual appliances or cloud templates
Who are the targets users for BitNami cloud hosting
The demo covers:
Creating a server on Amazon EC2 in minutes, with apps added
Earlier this week, I attended the first two days of the OpenStack Design Summit out in Santa Clara, CA. Before I took off I grabbed sometime with Jim Curry, GM of Rackspace Cloud Builders and the leader of Rackpace’s OpenStack efforts.
I got Jim’s thoughts on how the summit was going, how the project was going as well as some breaking news.
Some of the ground Jim covers:
Jim’s areas of responsibility
How this week’s Design summit is different from the first two? (how its grown and changed)
Some of the hot topics at this summit
Breaking news re. the next release, Diablo, milestones and a regular cadence
Note: I had a brain cramp and said Ubuntu follows the Mozilla schedule, I meant GNOME (go figure)
At the last OpenStack design summit, I sat down with Eucalyptus co-founder Graziano Obertelli and got his thoughts on the effort. This morning I bumped into a now clean-shaven Graziano and thought Id get his input on this week’s summit.
Some of the ground Graziano covers:
What Graziano’s goals are for this weeks OpenStack summit
Earlier today the OpenStack Design Summit kicked off here at the Hyatt in Santa Clara. This four day event is bringing together developers, users and business people to discuss OpenStack and design its future.
Among this morning’s attendees was James Staten Forrester Research’s cloud guru. I grabbed James at the first break and got his thoughts on the event’s kick-off and OpenStack in general.
Some of the ground James covers:
Why he chose to attend and what he’s looking for
What he thought of the opening presentations and how something like an OpenStack could alleviate some of the pain of outages like Amazon had last week.
What type of outcomes he would like to see from this weeks summit.
How important are programs like Rackspace’s cloud builder effort.
This morning at Facebook’s headquarters in Palo Alto the company announced their Open Compute project Partners and kindred spirits were there to tell the story behind Open Compute and explain what they think it means to the industry. One group of kindred spirits were the individuals from Rackspace. I got some time with Jim Curry who heads up OpenStack at Rackspace after the event officially ended.
Here is what Jim had to say:
Some of the topics Jim covers:
Driving efficiencies in data center design requires looking at the issue holistically.
Learning from Facebook’s successes and failures.
Looking forward to collaboration in an area that hasn’t historically had a lot of collaboration.
Engagement with Facebook engineers on how to run OpenStack on their hardware.
Today at its headquarters in Palo Alto, Facebook and a collection of partners such as Dell, Intel and AMD — as well as kindred spirits like RackSpace’s founder (the company behind OpenStack) and the CIO of the Department of Energy — are on hand to reveal the details behind Facebook’s first custom-built data center and to announce the Open Compute project.
Efficiency: saving energy and cost
The big message behind Facebook’s new data center, located in Prineville Oregon, is one of efficiency and openness. The facility will use servers and technology that deliver a 38 percent gain ìn energy efficiency. To bring the knowledge that the company and its partners have gained in constructing this hyper-efficient hyper-scale data center Facebook is announcing the Open Compute project.
Much the way open source software shares the code behind the software, the Open Compute project has been created to provide the specifications behind the hardware. As a result, Facebook will be publishing the specs for the technology used in their data center’s servers, power supplies, racks, battery backup systems and building design. By releasing these specs, Facebook is looking to promote the sharing of data center and server technology best practices across the industry.
How does Dell fit in?
Dell, which has a long relationship with Facebook, has been collaborating on the Open Compute project. Dell’s Data Center Solutions group has designed and built a data center solution using components from the Open Compute project and the server portion of that solution will be on display today at Facebook’s event. Additionally Forrest Norrod, Dell’s GM of server platforms will be a member of the panel at the event talking about the two companies’ common goal of designing the next generation of hyper efficient data centers.
A bit of history
Dell first started working with Facebook back in 2008 when they had a “mere” 62 million active users. At that time the three primary areas of focus in with regards to the Facebook IT infrastructure were:
Decreasing power usage
Creating purpose-built servers to match Facebook’s tiered infrastructure needs
Having tier 1 dedicated engineering resources to meet custom product and service needs
Over the last three-plus years, as Facebook has grown to over 500 million active users, Dell has spefically helped out to address these challenges by:
Building custom solutions to meet Facebook’s evolving needs, from custom-designed servers for their web cache, to memcache systems to systems supporting their database tiers.
Delivering these unique servers quickly and cost effectively via Dell’s global supply chain. Our motto is “arrive and live in five”, so within five hours of the racks of servers arriving at the dock doors, they’re live and helping to support Facebook’s 500 million users.
Achieving the greatest performance with the highest possible efficiency. Within one year, as the result of Dell’s turnkey rack integration and deployment services, we were able to save Facebook 84,000 pounds of corrugated cardboard and 39,000 pounds of polystyrene during that same year.
Congratulations Facebook! And thank you for focusing on both open sharing and on energy efficiency from the very beginning!
OpenStack, the open source cloud platform based on code donated by NASA and Rackspace, has gained considerable traction since it was launched eight months ago. The community has rapidly grown and there have been several releases. Now its time to get potential customers trying it out and kicking the tires.
With the idea of removing friction to adoption and make the testing out of the platform as easy as possible, Dell, Equinix and Rackspace are announcing today the availability of a free OpenStack cloud demonstration and test environment.
The idea of the demo environment is to allow organizations to easily evaluate OpenStack and assess application performance on the platform in a low risk environment for free. The next step after a successful demo would be a proof of concept.
Movin workloads around the country
This demo environment is initially available in three U.S. data centers and in Q2 of this year this offering will also be available in Equinix data centers in Europe and Asia. The initial data centers are:
Equinix Silicon Valley
Equinix Asburn, VA
Rackspace Chicago
By having geographically dispersed facilities customers are able to test out the moving of applications and workloads between them.
The various parts
The OpenStack demo environment is made up of the following components
Platform Equinix, a global delivery platform of 92 network neutral data centers in 35 metro markets
Widening the circle
The name of the game here is making the trying out of OpenStack as easy as possible. There are a lot of community members and open source aficionados who are already testing out OpenStack. The idea with OpenStack cloud demonstration environment is to provide a set up where a greater number of organizations feel comfortable evaluating the platform for themselves.
One of the trickiest things to get right in an open source project is the governance model. Who makes up the various boards and gets what authority is something struggled over and something that virtually no one gets right straight out of the gate. Its particularly interesting if you are a commercial entity sponsoring a project and want to maintain a certain amount of influence over the endeavor but also want it to grow and flourish.
Two weeks ago Jonathan Bryce, Rackspace cloud co-founder and one of the leads of the OpenStack project policy board, announced the changes that were being made to OpenStack’s governance.
I ran into Jonathan on Monday during South by Southwest and sat down with him to get some more insight into what the changes were and why they were being made.
Some of the ground Jonathan covers:
From Mosso to Rackspace cloud to OpenStack
How they’ve been surprised by the great uptake by the community and how this has led them to evolve the governance structure.
What the various boards are and what their make up will be
Last week and this, Austin’s downtown has been taken over by the South by Southwest festival (SXSW). What started out nearly 25 years ago as a music festival/conference has grown to include parallel film and interactive events as well. During the event every bar and venue downtown is occupied with bands, films or tech companies showing their stuff.
Check out the mini montage below that includes a quick interview I did with Dell solutions engineer Greg Althaus right after he finished the demo.
Don’t take our word for it
Dell is currently field testing crowbar and plans to donate the code to the OpenStack community after testing. If you are interested in testing crowbar yourself, email us at OpenStack@Dell.com.
Some of the ground the video covers
Intro montage: Welcome to the Kung Fu Saloon, setting up and a snippet of the demo
[0:40] Talking to Greg
What actually is crowbar and how does it work with OpenStack compute and storage?
How fast can you spin up a cloud using it?
Where does OpsCode’s Chef fit in?
Our plan to donate this code to the community after field testing.
One small step for OpenStack, one giant step for mankind! Ok, that may be taking it a bit far but today there is some exciting news that Dell and Rackspace have announced.
If OpenStack is new to you it’s the open source cloud platform based off of the Nebula cloud code base donated by NASA (hence all the rocket allusions) and the storage code base from Rackspace. The project is now about eight months old and its community is over 50 organizations strong including NASA, Citrix, AMD, Intel, NTT, cloud.com, RightScale, Dell, Opscode and Puppet labs.
To help the community and organizations get up and going building out their own OpenStack based clouds we are announcing the following:
Proof of Concepts: Rackspace and Dell are working together to deliver an OpenStack solution to market later this year. As the first step along the way we are currently seeking customers for OpenStack proofs of concept (POC). If you are interested in learning more about the POCs and possibly participating or getting engaged, email: OpenStack@Dell.com.
Code: Dell has developed an OpenStack installer that can be used to quickly install OpenStack on bare metal Dell PowerEdge C servers. This tool is being field tested, and Dell will contribute the code to the OpenStack community once testing is complete. To help users out further, once the OS is installed, the servers can be updated with all the latest packages and be automatically configured by Opscode’s Chef to provide their services in the OpenStack infrastructure.
Technical whitepaper: A technical whitepaper, “Bootstrapping OpenSource Clouds” which lays out the design of a hyperscale OpenStack cloud on Dell PowerEdge C servers.
But wait, there’s more, “Cloud Builders”…
To help support OpenStack deployments, Rackspace announced today that its launching a new division, “Cloud Builders.” Cloud Builders “will provide training, certification, deployment services and ongoing support and management to enterprises and service providers via the team of OpenStack experts and its partner network.”
Specifically the offerings will include (taken from the Rackspace blog):
Training & Certification – Rackspace Cloud Builders will provide training classes and certification testing for designing, developing and administering OpenStack Clouds. Rackspace also plans to empower a network of training and certification experts.
Deployment Services – Rackspace Cloud Builders, in conjunction with other experts from the OpenStack community, will help customers design and deploy OpenStack Clouds.
Support & Management – Customers will have ongoing access to remote support and escalation assistance from the OpenStack specialists, including proactive monitoring and fixes.
OpenStack is picking up momentum, stay tuned for more and in the meantime, set the controls for the heart of the sun! 🙂
Earlier this month when the Bexar release for OpenStack went live, a meet up was held in Santa Clara. As a part of the event, a series of lightening talks were given by various OpenStack community members. One of the speakers was Dell’s very own Rob Hirschfeld, a senior cloud solutions architect, who has been actively involved with the OpenStack project from the get-go.
Here is the short presentation that Rob gave where he talks about some of the key characteristics of a hyperscale environment and how it differs from a traditional enterprise data center.
Posted by Barton George 
Barton's Blog 