One of the interviews I did at Dell World was a video with IT in Canada. I did the video with Paul Cooper, Dell’s country manager for Canada.
In the first half of the video I talk about how Dell got into the cloud and where we play in the space. In the second half Paul talks about the roll the telcos will play in the delivery of cloud services in Canada as well as issues around privacy and data sovereignty.
Check it out.
From the article itself, here’s a great summary of our cloud participation and shows how we have built, bought and partnered along the way:
Dell’s excursion into cloud began with organic development of server and data centre capability in specialized systems to meet the needs of large cloud providers (Facebook, Microsoft Azure and Bing), progressed through modification of these systems for marketing to the “next 1,000”, and shifted to partnership with software makers such as Joyent to develop complete cloud solutions, and with companies such as VMware for the creation of a full service public cloud offering.
Supporting acquisitions along the way include companies with specific capabilities such as SecureWorks, which was purchased to address web security concerns that continue to dog broader cloud adoption, and BOOMI, a specialist in cloud integration, which enables Dell to better service customers who adopt a hybrid cloud approach to sourcing compute resources.
Yesterday I was out in the bay area to help moderate a Hackeratti shindig on Sand Hill road. One of my fellow moderators was Dave Rosenberg former CEO and founder of MuleSoft. Dave who is also an active contributor to cnet has recently started a new endeavor, Nodeable. This new venture, which Stephen O’Grady of Redmonk called “twitter for machines” also features former Canonical VP of corporate services, Neil Levine.
Before we began our moderating duties, I sat down with Dave to learn a little more about his cool new venture.
Some of the ground Dave covers:
Just what is Nodeable
(0:40) The rise of the cloud developer and the profile of the developer that Nodeable is targeted at.
(2:10) Kicking off their private beta and heading to a public beta in Q4
(2:48) What part of Nodeable is open source and which part is the secret sauce
(3:26) Nodeable’s architecture and the languages its written in
(4:00) Where does Dave hope to see Nodeable a year from now
Rob Hirschfeld, aka “Commander Crowbar,” recently posted a blog entry looking back at how Crowbar came to be, how its grown and where he hopes it will go from here.
What’s a Crowbar?
If you’re not familiar with Crowbar, its an open source software framework that began life as an installation tool to speed installation of OpenStack on Dell hardware. The project incorporates the Opscode Chef Server tool and was originally created here at Dell by Rob and Greg Althaus. Just four short months ago at OSCON 2011 the project took a big step forward when, along with the announcement of our OpenStack solution, we announced that we were opensourcing it.
DevOps-ilicous
As Rob points out in his blog, as we were delivering Crowbar as an installer a collective light bulb went off and we realized the role that Chef and tools like it play in a larger movement taking place in many Web shops today: the movement of DevOps.
The DevOps approach to deployment builds up systems in a layered model rather than using packaged images…Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.
On beyond installation and OpenStack
As the team began working more with Crowbar, it occurred to them that its use could be expanded in two ways: it could be used to do more than installation and it could be expanded to work with projects beyond OpenStack.
As for functionality, Crowbar now not only installs and configures but once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
The first project beyond OpenStack that we used Crowbar on was Hadoop. In order to expand Crowbar’s usage we created the concept of “barclamps” which are in essence modules that sit on top of the basic Crowbar functionality. After we created the Hadoop barclamp, others picked up the charge and VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.
It takes a community
Crowbar development has recently been moved out into the open. As Rob explains,
Last week we held Dell’s first Dell World event here in Austin, Texas. The two-day event was targeted at CxOs and IT professionals and featured an expo, panels and speakers such as CEOs Mark Benioff, Paul Otellini, Steve Ballmer and Paul Maritz as well as former CIO of the United States, Vivek Kundra. And of course, being Austin, it also featured a lot of great music and barbeque.
At the end of the first day Michael Cote grabbed sometime with me and we talked about the event.
Some of the ground I cover:
Dell World overview and our Modular Data Center
(3:35) Talking to press/analysts about our new Web|Tech vertical and our focus on developers
(6:00) The event’s attempt to up-level the conversation rather than diving into speeds, feeds and geeky demos.
The Dell Modular Data Center on the expo floor (photo: Yasushi Osonoi:@osonoi)
Big Data represents the next not-completely-understood got-to-have strategy. This first dawned on me about a year ago and has continued to become clearer as the phenomenon has gained momentum. Contributing to Big Data-mania is Hadoop, today’s weapon of choice in the taming and harnessing of mountains of unstructured data, a project that has its own immense gravitational pull of celebrity.
So what
But what is the value of slogging through these mountains of data? In a recent Forrester blog, Brian Hopkins lays it out very simply:
We estimate that firms effectively utilize less than 5% of available data. Why so little? The rest is simply too expensive to deal with. Big data is new because it lets firms affordably dip into that other 95%. If two companies use data with the same effectiveness but one can handle 15% of available data and one is stuck at 5%, who do you think will win?
The only problem is that while unstructured data (email, clickstream data, photos, web logs, etc.) makes up the vast majority of today’s data, the majority of the incumbent data solutions aren’t designed to handle it. So what do you do?
Deal with it
Hadoop, which I mentioned above, is your first line of offense when attacking big data. Hadoop is an open source highly scalable compute and storage platform. It can be used to collect, tidy up and store boatloads of structure and unstructured data. In the case of enterprises it can be combined with a data warehouse and then linked to analytics (in the case web companies they forgo the warehouse).
And speaking of web companies Hopkins explains
Google, Yahoo, and Facebook used big data to deal with web scale search, content relevance, and social connections, and we see what happened to those markets. If you are not thinking about how to leverage big data to get the value from the other 95%, your competition is.
So will Big Data truly displace Cloud as the current must-have buzz-tastic phenomenon in IT? I’m thinking in many circles it will. While less of a tectonic shift, Big Data’s more “modest” goals and concrete application make it easier to draw a direct line between effort and business return. This in turn will drive greater interest, tire kicking and then implementation. But I wouldn’t kick the tires for too long for as the web players have learned, Big Data is a mountain of straw just waiting to be spun into gold.
If you’re planning on attending the OpenStack Design summit and conference next week in Beantown you’ll have to check us out. I’m bummed that I will be missing the summit for the first time, I have a big internal presentation next week, but the rest of the Dell OpenStack crew will be there in force. Dell is a sponsor at the event and we will have a keynote, speaking sessions and demos.
What have we got in the works?
Besides checking out Crowbar and our Openstack solution which we launched back at OSCON we will have a whisper suite where we will be showing our latest and greatest stuff that is currently in the works. If you’d like to see what we have up our sleeve, email us at OpenStack@Dell.com and we can schedule a time slot for you to come and see for yourself.
Updated: For more details what we’ll be doing at the summit check out Rob’s blog
I have “long” thought that data is the currency of the Internet, it is what is monetized and when aggregated, parsed and made accessible, where the value lies for businesses and individuals. I was interested then to read in Rich Miller’s article about another analogy that had recently been drawn:
“Data is the new oil,” said Andreas Weigend, social data guru and former chief scientist at Amazon.com. “Oil needs to be refined before it can be useful. Big data startups are the new refineries.”
Another way of putting this might be to say that the internet runs on data.
Just a few thoughts to share over my much needed morning coffee. Talk amongst yourselves.
Dell has been working for the last four plus years outfitting the biggest of the big web superstars like Facebook and Microsoft Azure with infrastructure. More recently we have been layering software such as Hadoop, OpenStack and crowbar on top of that infrastructure. This has not gone unnoticed by web pub GigaOm:
Want to become the next Amazon Web Services or Facebook? Dell could have sold you the hardware all along, but now it has the software to make those servers and storage systems really hum.
They also made the following observation:
Because [Dell] doesn’t have a legacy [software] business to defend, it can blaze a completely new trail that has its trailhead where Oracle, IBM and HP leave off.
Letting customers focus on what matters most
Its a pretty exciting time to be at Dell as we continue to move up the stack outfitting web players big and small. The idea is to get these players established and growing in an agile and elastic way so they can concentrate on serving customers rather than building out their underpinning software and systems.
In my last entry I featured a video with the Bing Maps imagery team. In it they talked about why they went with Dell’s Modular Data Center (MDC) to help power and process all the image data they crunch. For a deeper dive and a look at one of these babies from the inside join Ty Schmitt and Mark Bailey in the following video as they walk you through the MDC and how it works.
Some of the ground Ty and Mark cover
The various modules that make up the MDC
The topology of the system
How the outside temperature dictates which of the three cooling methods is used
The racks inside the MDC and how they were able to pull the fans out of the individual servers
A little while ago I posted an entry talking about how Bing Maps was using Dell’s Modular Data Centers to power their new uber-efficient, uber-compact data center (or as Microsoft calls it “microsite”), located in Longmont, Colorado. But don’t take my word for it…
Below is a recent video of members of the Bing Maps’ imagery team, Tom Barclay, Brad Clark and Ryan Tracy, talking about what their needs were and why they chose Dell. (BTW, the written case study is also available now).
Some of the ground the team covers
Bing Maps leading the way and trying things out at Microsoft before the rest of the company.
Producing the imagery for Bing Maps including photographing all of the US and Western Europe and then stitching it all together with the help of tremendous processing power.
Their goal was to bring on additional capacity to support current and future site goals at the lowest cost, in the fastest amount of time with the least amount of down time.
A few weeks ago we announced that Dell, with a little help from Cloudera, was delivering a complete Apache Hadoop solution. Well as of last week its now officially available!
As a refresher:
The solution is comprised of Cloudera’s distribution of Hadoop, running on optimized Dell PowerEdge C2100 servers with Dell PowerConnect 6248 switch, delivered with joint service and support from both companies. You can buy it either pre-integrated and good-to-go or you can take the DIY route and set up yourself with the help of
A few weeks ago Michael Cote joined Dell from the excellent analyst firm, Redmonk which focuses on software and developers. Cote who spent five plus years with Redmonk has joined Dell in our corporate strategy group, focusing on software. I for one am very glad he’s here and feel that he’s joined at the right time in Dell’s trajectory to make a big impact.
I grabbed some time with him to get his initial thoughts and impressions. Here are his thoughts both witty and wise.
[Note: So there’s no confusion, this Michael Cote is Dell’s second Michael Cote. The first is the the former CEO of SecureWorks which Dell acquired.]
Some of the ground Cote covers:
Intro: Man is it hot, Cote’s background
(0:34) Why Cote made the move: going to the other side of the fence
(1:55) What is his new position and what will he be doing: his cloudy focus
(2:44) His first impressions: serious about solutions
In case you’re not familiar with Cloud Foundry, it’s an open source Platform as a Service project initiated at VMware. More specifically it provides a platform for building, deploying, and running cloud apps using Spring for Java developers, Rails and Sinatra for Ruby developers, Node.js and other JVM frameworks including Grails.
The project began two years ago when VMware’s CEO Paul Maritz recruited Derek Collison and Mark Lucovsky out of Google and set them to working on Cloud Foundry. Collison and Lucovsky, who built and maintained Google’s API services, were brought into leverage their experience of working with hugely scaled out architectures.
The Cloud Foundry project has only been public for a matter of months and one question that I’m sure has popped into your mind is what if I want to pilot Cloud Foundry in my own environment, won’t installation and configuration be a total pain?
Enter the Crowbar
Crowbar is an open source software framework developed at Dell to speed up the installation and configuration of open source cloud software onto bare metal systems. By automating the process, Crowbar can reduce the time needed for installation from days to hours.
The software is modular in design so while the basic functionality is in Crowbar itself, “barclamps” sit on top of it to allow it work with a variety of projects. The first use for crowbar was for OpenStack and the barclamp for that has been donated to the community. Next came The Dell | Cloudera solution for Apache Hadoopand, just recently, Dreamhostannounced that they currently working on a Ceph barclamp. And now…
Two great tastes that taste great together
Today’s big news is that VMware is working with Dell to release and maintain a Crowbar barclamp that, in conjunction with Crowbar, will install and configure Cloud Foundry. This capability, which will include multi-node configs over time, will allow organizations and service providers the ability to quickly and easily get pilots of Cloud Foundry up and running.
Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
Dell’s chief architect for big data, Aurelian Dumitru (aka. A.D.) presented a talk at OSCON the week before last with the heady title, “Hadoop – Enterprise Data Warehouse Data Flow Analysis and Optimization.” The session, which was well attended, explored the integration between Hadoop and the Enterprise Data Warehouse. AD posted a fairly detailed overview of his session on his blog but if you want a great high level summary, check this out:
Some of the ground AD covers
Mapping out the data life cycle: Generate -> Capture -> Store -> Analyze ->Present
Where does Hadoop play and where does the data warehouse? Where do they overlap?
Data continues to grow at an exponential rate and no place is this more obvious than in the Web space. Not only is the amount exploding but so is the form data’s taking whether that’s transactional, documents, IT/OT, images, audio, text, video etc. Additionally much of this new data is unstructured/ semi-structured which traditional relational databases were not built to deal with.
Enter Hadoop, an Apache open source project which, when combined with Map Reduceallows the analysis of entire data sets, rather than sample sizes, of structured and unstructured data types. Hadoop lets you chomp thru mountains of data faster and get to insights that drive business advantage quicker. It can provide near “real-time” data analytics for click-stream data, location data, logs, rich data, marketing analytics, image processing, social media association, text processing etc. More specifically, Hadoop is particularly suited for applications such as:
Search Quality — search attempts vs. structured data analysis; pattern recognition
Recommendation engine — batch processing; filtering and prediction (ie use information to predict what similar users like)
Ad-targeting – batch processing; linear scalability
Thread analysis for spam fighting and detecting click fraud — batch processing of huge datasets; pattern recognition
Data “sandbox” – “dump” all data in Hadoop; batch processing (ie analysis, filtering, aggregations etc); pattern recognition
The Dell | Cloudera solution
Although Hadoop is a very powerful tool, it can be a bit daunting to implement and use. This fact wasn’t lost on the founders of Cloudera who set up the company to make Hadoop easier to used by packaging it and offering support. Dell has joined with this Hadoop pioneer to provide the industry’s first complete Hadoop Solution (aptly named “the Dell | Cloudera solution for Apache Hadoop”).
The solution is comprised of Cloudera’s distribution of Hadoop, running on optimized Dell PowerEdge C2100 servers with Dell PowerConnect 6248 switch, delivered with joint service and support. Dell offers two flavors of this big data solution: Cloudera’s distribution with the free download of Hadoop software, and Cloudera’s enterprise version of Hadoop that comes with a charge.
It comes with its own “crowbar” and DIY option
The Dell | Cloudera solution for Apache Hadoop also comes with Crowbar, the recently open-sourced Dell-developed software, which provides the necessary tools and automation to manage the complete lifecycle of Hadoop environments. Crowbar manages the Hadoop deployment from the initial server boot to the configuration of the main Hadoop components allowing users to complete bare metal deployment of multi-node Hadoop environments in a matter of hours, as opposed to days. Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect a complete data analytics solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
The solution also comes with a reference architecture and deployment guide, so you can assemble it yourself, or Dell can build and deploy the solution for you, including rack and stack, delivery and implementation.
Late last week we announced that Dell’s Data Center Solutions group had outfitted Bing Maps’ uber-efficient, uber-compact data center (or as Microsoft calls it “microsite”), located in Longmont, Colorado. The facility is a dedicated imagery processing site to support Streetside, Bird’s Eye, aerial and satellite image types provided by Bing Maps. The site’s key components are Dell’s Modular Data Centers and Melanox Infiniband networking.
Brad Clark, Group Program Manager, Bing Maps Imagery Technologies described their goal for the project, “Our goal was to push technological boundaries, to build a cost effective and efficient microsite. We ended-up with a no-frills high-performance microsite to deliver complicated geospatial applications that can in effect ‘quilt’ different pieces of imagery into a cohesive mosaic that everyone can access.”
Keeping things cool
The challenge when building out the Longmont site was to design a modular outdoor solution that was optimized for power, space, network connectivity and workload performance.
The modules that Dell delivered use a unique blend of free-air with evaporative cooling technology, helping to deliver world-class efficiency and a Power Usage Effectiveness (PUE) as low as 1.03.
To watch the whole site being built in time-lapse check this out:
Who owns your data? Hopefully the answer is you and while that may be true it is often very difficult to get your data out of sites you have uploaded it to and move it elsewhere. Additionally, your data is scattered across a bunch of sites and locations across the web, wouldn’t it be amazing to have it all in one place and be able to mash it up and do things with it? Jeremie Miller observed these issues within his own family so, along with a few friends, he started the Data Locker project and Singly (Data Locker is an open source project and Singly is the commercial entity behind it).
I caught up with Jeremie right after the talk he delivered at OSCON. Here’s what he had to say:
Some of the ground Jeremie covers:
The concept behind the Data Locker project, why you should care
How the locker actually works
The role Singly will play as a host
Where they are, timeline-wise, on both the project and Singly
Last week at OSCON in Portland, I dragged Josh McKenty away from the OpenStack one-year anniversary (that’s what Josh is referring to at the very end of the interview) to do a quick video. Josh, who headed up NASA’s Nebula tech team and has been very involved with OpenStack from the very beginning has recently announced Piston, a startup that will productize OpenStack for enterprises.
Here is what the always entertaining Josh had to say:
Some of the ground Josh covers:
What, in a nutshell, will Piston be offering?
Josh’s work at NASA and how got involved in OpenStack
Timing around Piston’s general release and GA
The roles he plays on the OpenStack boards
What their offering will have right out of the shoot and their focus on big data going forward
I saw a great talk today here at OSCON Data up in Portland, Oregon. The talk was Practical Data Storage: MongoDB @ foursquare and was given by foursquare‘s head of server engineering, Harry Heymann. The talk was particularly impressive since, due to AV issues, Harry had to wing it and go slideless. (He did post his slides to twitter so folks with access could follow along).
After the talk I grabbed a few minutes with Harry and did the following interview:
Some of the ground Harry covers
What is foursquare and how it feeds your data back to you
Dell has been a part of the OpenStack community since day one a little over a year ago and today’s news represents the first available cloud solution based on the OpenStack platform. This Infrastructure-as-a-service solution includes a reference architecture based on Dell PowerEdge C servers, OpenStack open source software, the Dell-developed Crowbar software and services from Dell and Rackspace Cloud Builders.
Crowbar, keeping things short and sweet
Bringing up a cloud can be no mean feat, as a result a couple of our guys began working on a software framework that could be used to quickly (typically before coffee break!) bring up a multi-node OpenStack cloud on bare metal. That framework became Crowbar. What Crowbar does is manage the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in a matter of hours (or even minutes) instead of days.
Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
Code to the Community
As mentioned above, today Dell has released Crowbar to the community as open source code (you can get access to it the project’s GitHub site). The idea is allow users to build functionality to address their specific system needs. Additionally we are working with the community to submit Crowbar as a core project in the OpenStack initiative.
Included in the Crowbar code contribution is the barclamp list, UI and remote API’s, automated testing scripts, build scripts, switch discovery, open source Chef server. We are currently working with our legal team to determine how to release the BIOS and RAID which leverage third party components. In the meantime since it is free (as in beer) software, although Dell cannot distribute it, users can directly go the vendors and download the components for free to get that functionality.
More Crowbar detail
For those who want some more detail, here are some bullets I’ve grabbed from Rob “Mr. Crowbar” Hirschfeld’s blog:
Important notes:
Crowbar uses Chef as it’s database and relies on cookbooks for node deployments
Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as “barclamps.”
Each barclamp has it’s own Chef configuration, UI subcomponent, deployment configuration, and documentation.
On the roadmap:
Hadoop support
Additional operating system support
Barclamp version repository
Network configuration
We’d like suggestions! Please comment on Rob’s blog!
Posted by Barton George 
Barton's Blog 