IT as a Service in a Market with your Stack

Could private OpenStack clouds revolutionise IT procurement?

The promise of private clouds with OpenStack is clear; vendor neutrality and cost savings through scale and commodity hardware. Unfortunately these goals seem elusive to businesses when matched with the choice between the operating complexity of raw OpenStack or expensive turn key, single vendor hardware clusters.

OpenStack could deliver "turn key" and cheap very simply if only the marketplaces existed using OpenStack programming interfaces and a crowd of open source suppliers.

Marketplaces

As the cloud provides the abstraction layer, there leaves three simple classes of product that could be sold through a market:

1. Hardware (pre-installed commodity distributions)
2. Private SaaS (anything that can run in a datacentre)
3. Support Services

The only thing missing seems to be a self service marketplace where these products can be realised by businesses directly.

If anyone is interested in such a market let me know? I feel a software project coming on...

Community Distributed Cloud

Is there a place for community run, distributed and open source cloud infrastructure - or does Amazon / Rack-space know best?

Features / Advantages

1. potential massive redundancy
2. free to use locally (cost of local hardware & bandwidth aside)
3. credits to use community resources by contributing:
   
   1. storage
   2. compute time
   3. bandwidth
   4. Development / support of the community cloud solution features
4. enable new class of hardware platform / service paid for by community

Initial applications / services

If your resources are always available, credits could be gained to use the community resource for:

1. raw storage grid
2. raw compute grid
3. off site backup
4. redundant application hosting

After saving up credits, the following applications could be used across many sites for a short period for:

1. Application multi node development and testing
2. render
3. scientific modelling
4. AI

 

Open source IT as a service!

Modern version control systems (e.g. GIT) and emerging cloud and platform standards (e.g. Openstack) provide an opportunity to share common open source service definitions for commodity hardware.

What?

A solution for sharing IT "service definition templates" for entire business services and the process for its maintenance.

Everything for operating a business' IT would be defined in a version control repository so it would be shareable.

Why?

The hope would be to raise the transparency (and ultimately the quality) of IT processes for businesses by getting IT staff to work collectively across industry to provide shared IT services and processes.

• Reduce IT support services repetition and increase productivity
through the commoditisation (i.e. templating) of sevices AND maintenance process.
• Increased security through peer review (complete transparency).
• Prevent vendor lock-in to cloud, or other software service companies.
• Customers of IT services to "in-house" service hosting whilst "out sourcing" the support for which ever level of the stack they choose.

How

A trusted "deployment appliance" (itself created from the source repository) could allow pre-tested open-source solutions to be maintained and deployed on compatible cloud infrastructure.The appliance for a solution could be deployed in two typical "modes", dev or live.

Dev

A dev appliance can be created open (can run on any compatible hardware or VM image).The following activity would be enabled in dev:

1. Developing changes
2. "Check in" changes
3. Deploying test changes, verifying (testing) and releasing proposed changes to the development instances of the service.

4. Share (pull request for repo to beta / live / 3rd party).

   Full best practice CI Devops processes recommended here and would be backed in to the appliance!

Test / Live

A beta or prod appliance would subscribe to levels of acceptable change from an "up stream" version of the repository.

Detail

Initial ramblings - need to start to pull together open design here:

Basically everything in source except binaries, they get downloaded direct from authoritative publishers and CRC checked.

Need a clone "create a dev appliance" process with "bootstrap commands" from an authoritative repository source e.g. wget and run "boot" script where a software stack appliance to facilitate collaborative DevOps is downloaded and configured:

1. Trusted base image downloaded, maybe patched and booted (PXE boot and / or cloud API).
2. "Inside" booted base image, pull authoritative Version Control System repository and install and configure the following:
   • Config Management e.g. Puppet / Chef
   • Orchestration e.g. Rundeck, mcollective
   • Issue tracking
   • Other workflow dashboard(s)
3. Other boot script actions could include:
   
   • Authorise current Repo(s)
   • Service release (deployment)
   • Generate site specific "config"
   • Start device instances via cloud API
   • Service test / verification

 

Deployment Automation

This post is partly a definition and partly describes ideals that have served me well – it is not meant to be prescriptive – please provide constructive feedback!

Useful deployment automation describes EVERY change required to install and configure a complete service.

A separation of concerns is required to describe the essentials here. I have used the word “scripts” but flat file input to any deployment automation tool could be used.

Ideals:

The following ideals have served me well:

• NO ENVIRONMENT specific information should be embedded in the scripts. This enables the same script to be tested thoroughly and, when appropriate, used from development, build, staging and in production see Environment Configuration.
• Scripts have to be stored in source control.
• Where possible, ALL script dependencies should be stored in the same repository and use RELATIVE paths.
• When dependencies are not stored (e.g. deployable artifacts) – they should be referenced unambiguously (CRC’s serve well here).
• Any script language or flat file input to a tool that is appropriate to the task should be used.
• Any automation should check every input and fail if ANY step is unsuccessful (problem solving is easy when detected at the point of failure).
• All automation should reuse the same building blocks by breaking down the entire system requirements into deployment

The separation of configuration from automation is the most important point here to enforce repeatable testable deployments. Doing this properly is at the heart of Release Management.

Tools:

The simplest tools with the least system dependencies are the best fit for some of the ideals I express above.

At its simplest, environment management can be easily implemented by even Batch and Shell scripts with a little help from run XSLT processors. XML configuration can easily provide all the environment specific input.

Other scripting languages that are equally applicable to system management and provide better support for XML configuration input include Perl, Ruby and on Windows; PowerShell.

More recently, tools which try to holistically solve deployment automation include RPath, Nolio and build servers like AnthillPro. These tools introduce very specific ways to working which need a high commitment to a single vendor without the ability to easily share the solutions.

Orchestration

In the spirit of separating concerns it seems best to separate the actual release that can take place on one host from the steps that coordinate the release across multiple computers. The same orchestration steps can then be carried out on a single node (e.g. development) or systems with varying number of nodes (e.g. test, staging or even separate production installations for separate customers).

The specific host names and the mapping of roles that they carry out should be stored separately from actual orchestration steps (which should be concerned with roles only). This information forms part of the Environment Configuration.

Open "Ecosystems"

How to make hardware and software that works together?

Build open "ecosystems" of hardware and software designed to work together at the point of sale.

How to build the ecosystem? By publishing the automation that configures hardware and software for use.

Softservice Distro

...What?

The term Distro, in relation to software, effectively boils down to a 'single package'. This is ultimately what allows a program (or set of many programs) to be shared, tested and ultimately improved in a repeatable way. A good package is one that makes this distribution process easy and accurate (read fully automated - think app stores).

I've coined the phrase 'Softservice' to describe all the software and configuration required to provision a high level user service on as many computing devices as required. The term 'Softservice Distro' would define how the softservice could be easily distributed and therefore shared and tested (just add hardware / virtual or otherwise).

In summary a single packaged distro of distros such that the entire configuration forms a single, testable, end user service or"Softservice Distro"!

...Why?

I think the cost, reliability and the effective security of computing services is held back as the complexity of distributed systems grows. I believe these issues could be largely mitigated by removing the duplication of effort that exists in maintaining software services at the level of a single vendors software package.

If the unit of a distro were to be extended to encompass an entire computing environment, required to provide services, then the costs and reliability could be effortlessly shared between providers or even end users.

...How?

DevOps are already solving these problems for their own clients. I'm just advocating that the solutions are shared so that a maintainable 'Softservice Distro' can be provided direct to end users for use and to other DevOps to improve. This isn't a cloud provided service as such - more a provisioning service from the cloud to be owned and run as end users require. This would effectively recycle the knowledge used by the DevOps community so that it becomes accessible to a broader base of users. See my previous post on the Deployment Automation Solution for a few more details.

The result could be a collaborative effort to provide incrementally improving computing services to benefit end users directly. The idea would be to avoid the disparately provided and maintained services we experience today - again, see my other post Solution Store not Hobby Kit.

Devops Discovered

I've just discovered that I do 'devops':

"I've been treading the line between development and operations my whole career. When in support roles I've striven to develop automation as it was always more fun. When working on large development projects I've been heavily involved in test driven development, agile methods and generally automating the development lifecycle."

The other day I googled deployment, automation or clouds or some such thing, as I often do, and out popped the devops movement!

A good definition of 'devops' is on the dev2ops blog as is a take on how this is relevant to deployment.