Automation, devops drive open source deeper in the enterprise

Server provisioning and configuration management and automation are the latest examples of where the tech industry is being driven, largely by open source software. The leading open source server and IT infrastructure automation frameworks, Opscode Chef and Puppet Labs’ Puppet, sit on the leading edge of significant trends under way in enterprise IT — particularly disruption from cloud computing and devops, where application development and IT operations come together for faster, smoother delivery of software and services.

I’ve discussed the importance of open source software in cloud computing and in trends such as devops and polyglot programming. Consistently across all of these trends and the technologies that go with them, there are prominent roles for Chef and Puppet.

Read the full article at LinuxInsider.

CAOS Theory Podcast 2012.04.20

Topics for this podcast:

*OpenStack, Amazon, Eucalyptus and Citrix engage in open cloud warfare
*Microsoft spins off new company for openness
*Updates on automation players Puppet Labs and Opscode with Chef
*Percona turns attention to MySQL high availability
*Open APIs as the fifth pillar of modern IT openness

iTunes or direct download (28:42, 4.9MB)

Open source moving in mobile

We got another reminder of how disruptive open source software is to mobile computing this week, when Linux and Android merged back together. This appears to be good news for a number of parties, but Android and Linux developers and users seem particularly likely to benefit. The inclusion of Android code in the Linux kernel and the ability for Linux developers to more easily work on the Android environment and applications also ties into some of the key topics we’ll be covering in a Webcast March 21 titled ‘Open Source, A Tale of Two Cities in the Mobile Enterprise,’ presented by 451 Research and Black Duck Software.

This webcast, as the title implies, will focus on how open source can present both challenges and opportunities as enterprises adapt to market changes and mobile devices. This includes the fact that open source software frameworks, pieces and development are all enabling new applications to be quickly developed and deployed. However, this presents tremendous pressure on enterprise IT teams already dealing with disruption and change from cloud computing and the trend of ‘devops,’ which blends application development with IT operations and application deployment. The Webcast will cover how open source software is mixing with devops and other trends, such as the consumerization of IT and BYOD, to both disrupt and develop the mobile enterprise. We will also highlight some key open source software technologies in the mobile space and highlight some observed best practices for both vendors and customers.

CAOS Theory Podcast 2012.02.17

Topics for this podcast:

*NewSQL, new company in Akiban
*Discussion of APIs as the ‘new’ open source
*NoSQL leader 10gen grows, gets more agile
*Our coming report on Cloud Performance Management
*Zimory acquires sones NoSQL development team

iTunes or direct download (28:01, 4.8MB)

Open APIs are the new open source

We’ve seen the rise of open source software in the enterprise and also beyond the IT industry, but the real keys to openness and its advantages in today’s technology world — where efficient use of cloud computing and supporting services are paramount — exist in open application programming interfaces, or APIs.

Open source software continues to be a critical part of software development, systems administration, IT operations and more, but much of the action in leveraging modern cloud computing and services-based infrastructures centers on APIs. Open APIs are the new open source.

Read the full story at LinuxInsider.

Our view on the Changing Linux Landscape is out

We are pleased to present our latest CAOS special report, ‘The Changing Linux Landscape.’ This latest in our series of long-format reports takes a more in depth look at the Linux server market and how cloud computing, competition and the confluence of application development and IT operations known as devops are all affecting it.

Basically, we still see commercial vendors Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES) leading the market, but there are significant changes afoot, ushered in by cloud computing, wide use of other distributions such as Ubuntu, and continued use of unpaid community Linux such as CentOS and Debian. In addition, other distributions such Oracle Enterprise Linux continue to evolve and grow, as do the providers of Linux support, which now includes Microsoft. These additional competitors and choices, along with the new way of developing and deploying enterprise applications known as ‘devops,’ are all driving and disrupting the Linux server market.

This means challenges and opportunities – particularly in PaaS, which embodies devops practices – for both vendors and users. The report focuses on market dynamics with competitive analysis of leading Linux distributions, analysis of adoption drivers and hurdles, and customer case studies highlighting how Linux is put to work in today’s cloud computing environments.

PuppetConf and the state of devops

It’s been some time now that we’ve been talking about devops, the pushing together of application development and application deployment via IT operations, in the enterprise. To keep up to speed on the trend, 451 CAOS attended PuppetConf, a conference for the Puppet Labs community of IT administrators, developers and industry leaders around the open source Puppet server configuration and automation software. One thing that seems clear, given the talk about agile development and operations, cloud computing, business and culture, our definition of devops continues to be accurate.

Another consistent part of devops that also emerged at PuppetConf last week was the way it tends to introduce additional stakeholders beyond software developers and IT administrators. This might be the web or mobile folks, sales and CRM people, security professionals or others, but it is typically about applying business operations methodology to applications and IT, thus bringing in more of the business minds as well. The introduction of additional stakeholders was also a theme we heard from Puppet Labs CEO Luke Kanies in his keynote address. Kanies then discussed how the community was working to make Puppet the ‘language of operations,’ which it basically is along with competitors Chef from Opscode and CFEngine when it comes to devops implementations.

There was another interesting point on the PuppetConf stage from DTO Solutions co-founder and President Damon Edwards, who said devops should not be sold as a way to achieve cost savings, but rather as something that will bring return on investment (ROI). This is similar to the shift of open source software drivers we’ve seen in the enterprise, which are sometimes changing from cost savings and time to factors of performance, reliability and innovation.

Later in the conference during his keynote, Eucalyptus Systems CEO Marten Mickos also had some interesting observations concerning devops, which he described as managing the cloud from both sides. One of his points was that developers have the most to learn about operations. While I would agree to some extent, this statement is interesting when considered alongside my contention that most of the change in devops is happening on the IT administrator and operations side. Later in an interview, Mickos elaborated on his devops thinking, indicating the experts who orchestrate applications in cloud computing — both developers and admins — must understand the entire lifecycle and environment. Continuing our comparison of devops to open source, Mickos indicated the open source MySQL database that he helped usher into the enterprise was disrupting old technology, while devops is innovating new technology.

While it remains early days for devops in the case of many enterprise organizations, we continue to see and hear signs that devops practices, technologies, ideas and culture are making their way into more and more mainstream enterprise IT shops. While we expect devops practices to be implemented by many enterprises based on utility and need to leverage cloud computing, we see a higher level of awareness and engagement from leadership and executives than we did with open source software. This means we expect uptake of devops to happen more quickly and to generate more revenue and opportunity.

Time for your cloud gut check

It may be hard for Amazon, any of its users, critics or competitors to find a silver-lining in the recent cloud outage that took major sites offline for significant periods over the last week (ok, the critics and competitors are getting plenty), but I see a real upside for all: this has been our latest cloud computing gut check.

Just as we have seen in the case of open source software forks, dissents and competition, these challenges all represent a form of open source discipline that keeps code, communities and vendors ‘honest’ in the sense they must respond to developer and user demands and must also steer a successful path both organizationally and commercially. So while there is no doubt pain and loss from the Amazon outage, it is also a reminder that what does not kill your cloud computing deployment will only make it stronger.

It’s true, the outage illustrates that users and providers are still figuring out cloud computing, and that there is still much learning to be done. It was interesting to see some companies actually sending out press releases regarding how well they and their teams were able to keep their cloud-based environments going through the outage. Indeed, as highlighted recently by our own Tier 1 analysts Jason Verge and Doug Toombs, a number of heavy Amazon cloud users were able to largely sustain the blow of the outage and keep their clouds aloft, including Neftlix and Zynga. We can probably assume this kind of thing could happen with a private cloud, and if we don’t, we should. Still, the point is that the differentiation of technology and the team to effectively leverage it emerged as a critical differentiator during the Amazon cloud outage.

I believe the technology, tasks, procedures and preparedness that are represented in the winners versus the losers in this centers on ‘devops,’ a term we refer to often that involves the crossing of development, operations and other professionals in modern IT environments that both leverage and provide cloud computing services. Discussion of devops often centers on efficient use of cloud computing resources by both providers and users. Even when we consider ‘no-ops’ or more accurately ‘auto-ops,’ — whereby systems and operations are abstracted for developers and users — there is a definite need for knowledge, skill, experience and process when confronting cloud crashes, particularly on the operations side. Devops also represents a more holistic view of software in its environment(s), which is critical to crisis management and recovery for both Amazon and its users. Certainly Amazon and its partners are working hard to restore all of their cloud services to full functionality, but it is very interesting and encouraging to see customers and users adding in their know-how and talent to offset down servers and avoid downtime. It makes it clear why a large organization such as Facebook would benefit from opening its own datacenters and practices.

From Amazon’s and other providers’ perspectives – the cloud stubbed toe of this week also highlights how communication and reaction are perhaps as critical as the technical aspects of addressing what’s wrong and fixing it. Open source software also provides lessons here, indicating vendors and providers are best served by transparency and openness. What the message boards and Twitterverse are telling us now is that users will accept some degree of downtime and difficulty, but they want straight information on how long and how severely they will be down. Just as vendors face a challenge in fairly yet effectively pricing and charging for cloud computing, it may be difficult to provide guidance on recovery from an outage, but the same rules of PR crisis management apply: don’t over-promise and don’t under-deliver.

So just like a fork, leadership crisis or large, proprietary competitor is supposed to wreck an open source project or vendor, the latest cloud crash will finally stifle this cloud hype, bluster and momentum, right? Not quite. I would argue that just like a good fork, feud or megavendor foray into open source software is actually a strengthening, disciplinary measure, the latest cloud coughing will serve as a necessary gut check on cloud computing, thus helping us avoid a cloud bubble.

CAOS Theory Podcast 2011.04.15

Topics for this podcast:

*New CAOS/IM Special Report on database alternatives
*Future of Open Source, Future of Cloud Computing surveys
*Database heavyweights and the new challengers
*VC funding for open source in Q1 2011
*Cloudera and Apache team on Hadoop
*VMware’s Cloud Foundry PaaS, latest on devops

iTunes or direct download (30:14, 5.2MB)

DevOps and PaaS, yes, but now No-Ops?

We continue to closely watch the devops trend, with some new offerings and new nomenclature, but also validation of our contentions this would begin washing over more mainstream enterprise IT.

Some of the most recent discussion of devops is coming in context of VMware’s Cloud Foundry announcement and offering, an open source PaaS that gives developers another option for building, testing and deploying cloud applications and services. While I do believe Cloud Foundry and VMware’s decision to opt for an open path in PaaS is further evidence that cloud computing may be opening up.

Based on some of the initial Twitterverse reaction to Cloud Foundry, it is also further evidence that devops is contending with another term that has emerged in the discussion of deploying applications in and among today’s cloud computing resources and environments: ‘no-ops.’ The idea is that infrastructure – servers, storage and network — as well as its configuration and maintenance are so automated, there is really no need for the ‘ops’ or system administration part of devops. However, in the larger picture and in the long run, particularly at greater scale, there is undoubtedly need for system administrators. One of the bottom line findings of my research on devops is that the trend is very much about a dramatically changed purpose and role for system administrators, who are typically freed up of mundane OS maintenance and other tasks, but who must also embrace openness and transparency in their operations and scripts, which can be very foreign. While no-ops may be one way to respond to developers cries of ‘give us root,’ I believe that devops with the ops is required for a successful approach. That ops part may indeed be handed off to someone else, and the options and ability to do so have never been greater — again thanks mostly to readily-available cloud resources and infrastructure. Another perspective on devops is that it is bringing some of the agile and automated practices and procedures of software development into the datacenter and operations team, which have previously been focused on their own scripts and stability above all else.

So when I’m asked does devops mean devs doing more ops? Is it ops doing more dev? I say this: devops is the confluence of roles and duties among both software developers and IT operations professionals — many of whom are increasingly working in both jobs at various points or together in their careers. No-ops may emerge as a preferred option as organizations use and grow confidence in various PaaS offerings, as well as more openness in the clouds in general, perhaps. Still, I think that the ops folks still have a tremendous role to play, and I wonder about the PaaS innovation that will be possible when we see the same style of collaboration and communication in operations that we have had on the development side, in large part because of open source, an example being Facebook’s recent move to open up on its datacenters.

The future of cloud computing is the future for open source

I recently wrote a column about the lack of a cloud computing bubble, even though the hype and marketing levels around the cloud have risen along with innovative technologies and vendors. As we consider what’s next for cloud computing with a survey presented by 451 Group, North Bridge Venture Partners and GigaOm, we will also be able to get a good sense of what’s next for open source software, given the prominence and significance of open source in the clouds.

Given our most recent efforts to track open source software in the enterprise, it is relevant to note that we see a continued, symbiotic relationship between open source and cloud computing. In fact, in many ways, the future of open source depends on the future of cloud computing and vice-versa. One of the symbiotic relationships between open source software and cloud computing is also one of the main reasons I believe both will continue to be a big part of enterprise IT and a big opportunity for vendors and investors: customer enablement. The lessons, practices and community of today’s enterprise IT that have been ushered in by open source – more transparency on the plans for products and code, more flexibility in working with both legacy products and software as well as newer open components, add-ons and combinations, faster development and fewer dead ends via vendor death, acquisition or strategy shift — are being applied to cloud computing. We also see evidence of this customer enablement in the makeup of today’s communities, both open source and non, which include both developes and users/customers.

I continue to have some concern about how open will be open enough, and whether that will truly be open and collaborative enough for these new, customer-enabled cloud communities.

However, I remain convinced that cloud computing may be opening up and, just like open source, is much more than a catch-phrase or hyped-up marketing term. It is central to the continued success, growth and innovation of vendors and users in the key categories I cover, including open source and devops.

Red Hat-Makara means more open source in the clouds

It is with great interest that we watch Red Hat add in the cloud application management technology of its Makara acquisition to fill out its Cloud Foundatios PaaS offering. We believe Red Hat gains a much needed application managemeint piece for its cloud computing strategy and extension from its Red Hat Enterprise Linux (RHEL), JBoss middleware and other open source software for the enterprise. The Makara acquisition also arms Red Hat for the pending PaaS war in which it will be competing with much larger rivals, including Microsoft and VMware. Makara also represents Red Hat’s reach out to managed service providers, where we, along with other rivals, see ample opportunity for Linux. Finally, we believe Red Hat will increase the prominence of open source in cloud compouting and PaaS, though we see from our Seeding the Clouds report that open source software is a critical part of nearly all of the major cloud providers’ stacks.

Makara is also all about devops, which is having an impact on enterprise software development, IT operations, data management strategies, mobile application development and more. This is another great extension for Red Hat, which has a big role to play not only in the ‘ops’ part of devops, but also in the ‘dev’ part, as the company lays out with its Cloud Foundations PaaS technology and strategy.

Another interesting aspect of the Red Hat-Makara deal is that Makara had relatively recently adjusted its offerings in favor of public clouds, where the company reported there was more traction and revenue, as opposed to private clouds, which are nonetheless still growing in use. This might seem bad for open source, which has always seemed poised and positioned perfectly for private cloud building more than public cloud infrastructure. However, time has shown that there is just as much interest and demand for open source software in public clouds, such as the various Linux distributions popular on Amazon EC2 and other public clouds or MuleSoft’s Cloudcat Tomcat application server as a cloud service.

Actually, the fact that Makara was seeing more action and business in public clouds may mean that Red Hat will be able to more effectively monetize the use of Linux, JBoss and now Makara and its Red Hat Cloud Foundations open source software, given public cloud use typically comes with an expectation to pay, whereas private cloud building is often associated with DIY, support yourself and open source because it is free. This also highlights the shifting drivers for open source sotfware, which in the case of Makara and cloud application management have more to do with innovation and flexibility than cost.

Similar to OpenStack, the Red Hat-Makara deal is further evidence the market wants, needs and will support alternatives, particularly if they are open source. While part of Makara’s technology was already open source software, Red Hat intends, as we would expect, to open source all of the software. Thus, the deal will make even more of cloud computing, and particularly cloud application management and PaaS, open source.

Devops just keeps coming up

Devops — the pushing together of enterprise application development and the IT operations, mainly by cloud computing, collaboration and automation — continues to come up in conversations with large enterprise IT vendors and customers.

When we began covering devops, we saw it focused mainly on the software developers and IT administrators who were, themselves, devops, that is they worked in software development with previous or current experience in IT administration, or vice-versa, with IT admins that were software developers in previous lives. As our coverage of devops continued, it became clear that these groups were perhaps the main practitioners of devops, but certainly not the only ones that came into play, with business requirements, customer service, security and other stakeholders considered.

With our most recent and extensive research on the topic, our special report ‘Rise of the Devops,’ we considered these many stakeholders, as well as the drivers of devops, which include cloud computing, agile software development, automation and open source software. In addition, through our discussions with several devops end users that served as our case studies in the report, we came to the conclusion that devops was poised to move beyond Web 2.0, media and SaaS companies to more mainstream, enterprise IT, where in many ways, devops embodies effective, efficient use of cloud computing, whether for a cloud computing service provider or for a consumer of those services.

Since our report was published, the topic of devops continues to come up, often when talking with software development, ALM, cloud computing and other players. However, there are two areas that stand out: devops among database administrators and devops for mobile software development.

In terms of DBAs, they are clearly among the key stakeholders in a devops implementation, given the importance of data, its security and its place in today’s enterprise applications. We saw from another special report Seeding the Clouds, that the data layer was one place where we saw a heavy prevalence of open source software, including Apache Hadoop, NoSQL and other technologies. Data and the people who manage it have an equally significant role in devops.

As for mobile software and mobile application development, the main connection to devops seems to be the idea that organizations must get going right away on whatever projects, teams, products and objectives they can to begin the process of iterating, refining, repeating and succeeding. One of the key themes and best practices in the devops report is to begin the process of joining teams on software releases and updates understanding that the objective is not perfection, but rather, improvement. The better the start, the faster and greater the progress, but one of the key challenges of successfully implementing devops practices is finding a starting point. Once organizations begin to put people and teams together, the efficiencies can happen, but another big aspect of devops is that it is cyclical, going from dev to ops and back for a process that is improved at the same time it is automated. We hear and see a lot of the same type of thinking for mobile application development — getting it started and building from there.

We have provided some sense of devops implementations in some User Deployment Reports (subscribers only): Thomson Reuters and Fox Audience Network. Based on what we’ve seen so far and what we continue to hear from enterprise vendors and customers, we have no doubt that devops will continue to spread into more mainstream enterprise IT and into more enterprise IT conversations.

CAOS Theory Podcast 2010.10.15

Topics for this podcast:

*Open source deal analysis: Black Duck-Ohloh, Puppet-MCollective
*Oracle and open source brings future and forks
*VC for FOSS down in Q3
*Devops and its connection to open source

iTunes or direct download (27:44, 7.5MB)

The devops are coming

We are pleased to present our latest CAOS special report, The Rise of Devops, a collaboration with the 451 Group’s Infrastructure Computing in the Enterprise (ICE) service. The report came about as we continued to encounter evidence that enterprise application development and enterprise application deployment and IT operations were being pushed together. What we found is this trend is fueling some of the more innovative SaaS and cloud computing offerings and moves from companies where one might expect it — software providers, media, online investing, etc. Still, since devops, when done correctly, represents the realization of cloud computing benefits — increasing efficiency, infrastructure flexibility and organizational effectiveness — we also expect it to grow among more mainstream enterprise IT users.

Devops, the confluence of application development and IT operations, is being pushed by a number of factors, primarily: cloud computing, web and agile software development, open source software, automation and more. The trend, which is just in its infancy, is also facing significant hurdles, including technical, cultural and other issues, such as the need to keep dev and ops separate to simplify compliance and regulatory needs. As we’ve discussed previously, devops is also multi-layered, involving not just devs and IT system administrators/operations, but a number of other stakeholders, including business requirements, customer relations, sales, security and other personnel.

Some of our key findings include the fact that devops is happening and is poised for growth, that this growth is following a quiet but steady pattern we’ve seen previously with open source software and virtualization, and that faster
application release times, improved-quality software and reduced downtime are typically the biggest rewards.

The report is the product of our research, conversations and experience with devops, including participation in Devops Days. We also get a good sense of the devops experience in the report by looking at six devops case studies, which include organizations in Web 2.0, media, software and financial services.

One of the main points of the report is that devops is poised for significant growth. Since the proper and effective implementation of devops embodies the efficiency and elasticity of cloud computing, we have no doubt it will be felt across more and more of enterprise IT over time.