The Software Abstractions Blog

Cloud Computing: The Blurring Boundary between Applications and Infrastructure

 

 

IT and Co-location

In the distant past of software development - by which I mean, ten years ago - software companies made a clear distinction between Application development and Infrastructure management. Most companies had a dedicated IT team to configure, provision, install, manage, maintain and update the hardware infrastructure needed to run their software applications; separately, the software engineering team focused on building the software platform and applications. The hardware was typically hosted in a co-location facility. The IT team essentially “owned” this hardware, with responsibility for all the functions associated with it.

Over the past five years, the rise of the DevOps movement in startups and in large cutting-edge technology companies such as Google and Netflix, especially in Silicon Valley, created the first bridge between these two areas of software applications and infrastructure.

According to the DevOps philosophy, configuration and management of the underlying infrastructure services was treated more like software development, using higher-level languages like Ruby and Python, and standard software application methodologies and tools like agile development, source code control (e.g. git), versioning and branches, bug tracking (e.g. JIRA) and test automation. DevOps as a function came increasingly within the purview of the software engineering team rather than that of traditional IT.

Cloud Computing 

The rise of Cloud computing and “Infrastructure/Platform as a Service” offered by vendors like Amazon Web Services (AWS) has further accelerated this trend. For example, AWS has a built-in command-line tool called AWS CLI, as well as software API bindings for popular languages like Java, PHP and Python, that allow a computer program to perform actions for the entire Infrastructure life-cycle: from provisioning new hardware, to scaling it, managing and monitoring it, through the final tear-down when it is no longer needed.

Underneath the covers of course is an actual data center, with real servers, racks, power and networking equipment, security devices and policies, NOCs, and all the other “traditional IT” functions. But this layer is completely “virtualized” or hidden from the application developers, who can treat it as yet another software API or service.

The public cloud computing vendors, and private cloud data centers within large companies, still undertake all the classic IT projects and perform the same tasks as before. What is different however, is that from the perspective of a typical software startup or mid-sized SAAS application provider, a separate Infrastructure function has ceased to exist; instead it has been absorbed into the software engineering team, whose members are now expected to possess skills for managing cloud computing APIs, just as they are typically expected to understand how to interact with an operating system, or with a web application server or a handheld device API. This will not only affect the composition and operating practices of software engineering teams, but will also enable them to develop code at an increasingly faster pace in the future.

OpsWorks and Docker

A great example of this new approach of treating infrastructure provisioning, management and monitoring as software code is embodied by the AWS OpsWorks service, which uses an open source framework called Chef for application management. Chef programs, appropriately named “recipes”, can be used to configure servers, implement auto-scaling, apply security policies and monitor the resulting infrastructure stack.

The most recent step in this evolution is the increasing popularity of deployment containers, especially Docker (but also Rocket and LXC). In an explicit analogy to shipping containers which provide for all types of disparate goods to be handled in a uniform way by different modes of transportation, Docker enables application developers to interact with their infrastructure environment in a uniform way, regardless of whether the container is hosted on a laptop or on a data center server with high computing power. Both Amazon’s OpsWorks and their newly offered Container Service support container deployment.

Implications for Development Teams

This trend will have some interesting implications for the software development team in a typical small to medium-sized software-as-a-service (SAAS) company:

1. Team composition: Software developers in application development teams will need to understand both infrastructure (at the API level) and traditional application logic (algorithms + data structures). In larger development teams, this may be structured as separate sub-teams, each with a different individual focus.
2. Faster turnaround: Since the PAAS vendors allow computing resources to be added or removed with the click of a button or with a single API call, dev teams can now react much faster to the needs of customers or individual developers. This enables auto-scaling based on customer demand and individual development sandboxes for experimentation, which will change how developers work on their code.
3. Cost: Auto-scaling removes the need to provision servers up-front for peak demand. Along with economies of scale at the big data centers, this makes infrastructure resources significantly cheaper. At the same time, unless scaled-down appropriately, unused resources continue to incur periodic costs even when they are idle which can get very expensive. Due to these two competing forces, managing virtual infrastructure cost will be an important consideration for software architecture, code and development processes in the future.
4. Approach: Treating infrastructure configuration and management in a similar way to source code is a fundamentally different approach, lending itself well to the standard tools, processes and abstractions of software engineering as described previously. This will enable software development teams to take an agile, iterative approach to infrastructure development, analogous to standard software “scrum” practices.
5. Leaky abstractions: One challenge with the new approach is that software developers will be more aware of the details about the underlying virtual infrastructure. This could lead to “leaky” abstractions where they take advantage of local quirks within their particular infrastructure. Introducing containers like Docker will go a long way in mitigating this issue.

The Future: Cluster Management

As applications increasingly adopt a container-centric approach for deployment, two additional types of services and tools will become very important:

• Container management: This type of tool enables container deployment, orchestration, management and scheduling. A great example is the open source Kubernetes project from Google, others include Lattice and Marathon.
• Cluster management: This is the concept of providing a higher level of abstraction across the entire data center, i.e. an entire cluster of servers is treated as a single operating system kernel capable of running multiple projects simultaneously. Initially proposed by Google for their internal Borg project, this is now available to everyone through the open source Apache project Mesos. The new open source operating system CoreOS is designed from the ground up to run Docker containers across a cluster.

As these types of tools become mainstream in the future, developers will only need to focus on building out their application components inside standard containers and specifying the needs and requirements of those containers; it will be up to the underlying infrastructure services framework to optimize the required resources and cost to provide the application with everything it needs. 

 

US Health care: Poised for a Radical Transformation

 

 

 

The health care system in this country has been undergoing a major change over the past ten years and the pace of change is accelerating. We are now at an inflection point and it will never be the same again. This post examines the external forces that are driving the change and highlights the potential and likely effects.

 

External Forces

 

Legislative Changes

Over the past few years, the federal government has enacted a series of legislations and regulations related to Health IT that have created a profound long-term impact on the nation’s health care system. Some of the key laws are:

• HIPAA 1996: Although this law primarily focused on protecting health insurance coverage for workers and their families, it was the first shot aimed at increasing the efficiency of the nation’s health care system. It required the establishment of national standards for the use and dissemination of health care information, created a unique National Provider Identifier (NPI) enabling covered providers to participate in standard transactions, and laid the groundwork for strong security and privacy protections for patient health data.
• HITECH (as part of ARRA) 2009: This act proposes the meaningful use of interoperable electronic health records throughout the US health care delivery system as a critical national goal, with a financial incentive to providers and hospitals who can demonstrate that they have engaged in efforts to embrace this technology.
• ACA 2010: In addition to consumer protections, this act has a strong focus on improving the quality and lowering costs of health care services, including a Patient’s Bill of Rights, an emphasis on preventive services, and encouraging integrated health systems.

Taken together, this series of laws lays out a blueprint for health providers and organizations to follow. It strongly promotes the use of electronic records, interoperability, integrated care, patient empowerment and a focus on prevention, with the ultimate goal of driving down costs and improving the quality of care.

 

CMS Cost Reduction

As one of the largest payers in the industry, the Centers for Medicare and Medicaid Services (CMS) have a huge impact on the US health care system; even private insurers eventually tend to be consistent with CMS guidelines. In an effort to promote higher efficiency and reduce fraud within the health care system, CMS policies in recent years have been on a long-term path towards reduced reimbursements, increased documentation and stricter standards. As an example the recent introduction of the readmissions reduction program and associated penalties is forcing hospitals to put more focus on post-acute care and data interoperability for discharged patients.

 

Engaged Consumers (Patients)

At the end of the day, we’re all patients. Today’s consumers have unprecedented access to information (both accurate and misleading) about themselves and their conditions, thanks to Google, the world-wide web and a plethora of connected low-cost monitoring devices. There is an accelerating trend toward self-confidence, self-help and advocacy among patients in the US and all over the world. Tomorrow’s patients will be equal partners with their health providers in the planning and implementation of their treatment plans.

 

Technology

Software and hardware technology is making great strides in many areas of health care. Together these changes build on each other, creating an unprecedented climate of advancement in health technology.

Broadly, these innovations can be classified into a set of distinct groups:

• Cloud Computing: The increasing popularity of data center infrastructure and platforms as a service (IaaS/PaaS) has brought a new level of flexibility, scale and reduced cost to software applications that collect and analyze large-scale data sets. Advances in container and cluster management enable applications to run complex machine learning strategies on these massive data sets at a reasonable cost.
• Smart Devices: Smartphones are the true “personal computers” of our age; they are easily available, low-cost, always nearby, and almost ubiquitous even in less developed regions of the world. Any health technology for patient interaction, for example frequent mobile vision testing for patients with diabetes, can be quickly rolled out globally with ease, simply by turning it into an iPhone or Android app.
• Sensors: Powerful low-cost sensors - such as wearables, ingestibles and in-home devices - that can provide actionable, continuous, real-time data to health providers are becoming more common and will be deployed routinely for patient monitoring in the future.
• Biotech: Sensors, devices and algorithms in the fast-growing biotech industry continue to move forward. For example, an iPad-sized DNA sequencer can make this type of sequencing commonly available at low cost.
• Data: A massive focus on Big Data technologies and solutions has dominated the tech industry for the past few years, creating revolutionary techniques for handling data at scale. New open source software technologies for non-relational databases, data ingestion pipelines, map-reduce and other parallel processing algorithms, predictive analytics and unstructured data mining enable data collection and aggregation, analytics processing and visualization of results on large scale data streams in near real-time at a reasonable cost.
• Interoperability: As a key focus of the national health IT policy as articulated by the Office of the National Coordinator (ONC), the need for health data interchange among disparate health systems has created a slew of new technology solutions, including integration and interface engines, transport protocols such as the DIRECT project and data formats like HL7 CCD.
• Personalization: A key attribute of many of the new health technologies is personalized solutions and products; for example, applications that adjust to the user’s preferences, 3D-printing for custom manufacturing, medications that are specific to the patient’s DNA, and the potential uses of nanotechnology for a specific individual.
• Security and Privacy: In health care, more than in most other domains, the security and privacy of patient clinical data is extremely important, with enforcement and penalties encoded by the HIPAA Privacy Rule. An entire subset of the health technology industry is focused on products and services to help covered entities and business associates stay in compliance with these regulations.

In conjunction with the strong focus on improving health care in government and industry, in the past few years there has been a surge in venture capital investment in this domain. There is tremendous excitement in the tech community about health care opportunities, with lots of startups and increased innovation in this area. 

 

 

Likely Outcomes

 

Patient Health as a Service (PHaaS): The new Economic model

The continuous and glaring focus on reducing costs is driving the health care system towards a new economic model that generates revenue by focusing onkeeping the patient healthy and out of the system rather than by getting reimbursed for services rendered. Conceptually this is similar to providing regular maintenance for a car rather than services when it breaks down. The biggest advantage of this new model is that it aligns incentives much better between the health care system and the patient.

Some of the characteristics of this cost-conscious model are the following:

• A focus on prevention rather than episodic services
• Concierge health plans, such as a monthly fee for unlimited primary care
• Increased financial incentives for preventive services, screenings, improved nutrition and regular exercise
• A focus on early detection, increased follow-up, and monitoring of chronic conditions to reduce ER visits and hospital admissions
• Increased cost transparency and consumer choice for medical services
• Remote care: Telemedicine, remote dispensing, and other similar approaches
• Continuous monitoring: Sensor data, devices, smartphone apps
• Pushing services down the provider cost chain: PAs, NPs, increased pharmacy services, offshore support, reliance on automation

 

Automation and AI: The Robot will see you now

Hardware and software automation is still in its infancy within the health care system, but with the increased focus on reducing costs we are likely to see a sweeping change in the future similar to manufacturing automation using industrial robots in the 1980s and 1990s.

Improvements in health care automation technology are already under way in many areas:

• Robot-assisted surgery (especially microsurgery)
• Ingestible sensors, smart pills and bottles
• Cheap, rapid, easily-accessible lab results (like a blood test)
• Electronic, semantic patient data for demographic, clinical and fitness parameters
• AI-assisted diagnosis and measurement

 

The Big Data Revolution: Measure Everything!

New technology improvements have made it possible to collect, aggregate, analyze and visualize staggeringly large-scale data in near-real time at a reasonable cost. These new technologies will revolutionize health care both at a macro and a micro level.

At a macro level, it will enable public health systems and health organizations to manage the overall health of their entire patient population, looking for trends and automatically highlighting potential patient issues for proactive intervention.

At a micro level, it will be possible to continuously track an individual patient’s condition, treatment and progress remotely, and compare those parameters with nominal values against the backdrop of the entire population of similar patients.

The Big Data technologies available now for data storage and algorithmic analysis enable researchers to track and analyze exponentially larger data sets for treatment and research; so it is entirely possible to test a new theory against the entire data set, rather than extrapolate from a statistical sample of the population. This could change the way medical science researches diseases and their cures. Machine learning and predictive analytics based on this data could lead to unexpected discoveries.

Finally, automation and data analytics taken together enable physician scientists to make a systematic, objective evaluation of treatments and medications; AI assistance based on these evaluations can be used to enhance and support the traditional approach of physician experience and intuition, as venture capitalist Vinod Khosla argues in his influential paper: 20-percent doctor included.

 

Patient-centered Care System

Another likely outcome is a shift in focus towards a health care system that proactively reaches out and engages with patients on their terms: wherever they are and whenever they are available, as opposed to the current model where patients engage with health providers at specific locations at specified times.

A patient-centered care system will look very different from the current status quo:

• The health system will proactively reach out to the patient wherever they are, through a variety of devices and technologies
• The emphasis will be on localized services: remote care, home health, telemedicine, local pharmacies, ingestible sensors, monitoring devices
• An increased focus on compliance, post-acute care, and managing chronic conditions
• A holistic integrated approach to patient health: health maintenance, not pay per service, as mentioned earlier
• Continuous monitoring with automatic follow-up, similar to a home alarm system
• Personalized medication (genome- and symptom-specific) and customized delivery through a combination of in-person, mobile and automated services
• A focus on what is “normal” for this patient? For example, diabetes may turn out to be many different diseases, all of which have the same symptom of blood sugar issues

 

Integration: Interoperability and the Care Team

As the concepts of Meaningful Use and Health Information Exchange (HIE) take hold across disparate components of the health care system at all levels, widespread integration will make it possible to provide patient data visibility across the entire care team, including patient caregivers.

This broad-based integration will have a big impact not only on how care is provided, but also on the health organizations themselves - leading to the formation of vertically integrated health care systems or groups (ACOs) comprised of hospitals, emergency rooms, ambulatory clinics, labs, home health providers, pharmacies and even payers. These integrated organizations can not only track all aspects of the patient’s health but can combine that data with provider workflow and financials as well. As an example, coordinated care organizations in Oregon have been transforming the health care system in that state.

With the increased prevalence of deep cost consciousness, integrated care organizations and bundled payments, we are likely to see a broad consolidation of many small players into a few large entities that have the resources to deal with the new financial, technology, data and interoperability requirements.

 

Global Access to Healthcare

This to me is the most exciting development of all. The combination of algorithmic approaches, diagnostic support, telemedicine and low-cost tests means that high-quality health care can be made uniformly available in economically-challenged and isolated locations of the world.

We are already entering a phase where, in the near future, practically everyone will have access to a smartphone; these powerful computing devices are low-cost, easily accessible and nearly ubiquitous. At the same time, Cloud technologies with economies of scale are making computing and storage power progressively cheaper. Taken together, this will make it economically possible to provide prevention and monitoring health technologies to everyone, a true global democratization of health care.

 

The Future is Now

 

All of the effects listed above are interdependent and build on each other, which will lead to dramatic sweeping changes in health care within the next decade. The biggest hope is that it will lead simultaneously to better patient outcomes andreduced health care costs.

This type of industry-wide change is not new. The automation and optimization wave that swept through manufacturing in the late-80s and early-90s revolutionized how physical products are made; similarly ERP systems transformed how organizations collect, manage and interpret data from their business activities.

These types of changes not only impact and transform entire industries and economies, but they change the way we live. The current sweeping changes in health care have the potential to do just that on a global scale.

[Cover photo by Army Medicine / CC BY 2.0]

 

Links: Comparing Yahoo! S4 and Storm for Continuous Stream Processing (aka Real-time Big Data)

The next frontier in Big Data Analytics beyond Hadoop is Real-time processing. The two open source leaders in this space are: S4 from Yahoo! (now in Apache Incubator) and Storm from Nathan Marz (of BackType and Twitter). 

After some research on this topic, I found only a couple of good articles comparing the two, and a question on Quora. The links are given below.

S4 vs Storm

1. Awesome blog post from Gianmarco de Francisci Morales: http://gdfm.me/2013/01/02/distributed-stream-processing-showdown-s4-vs-storm/
2. Slideshow from Richard McCreadie of the University of Glasgow:  http://demeter.inf.ed.ac.uk/cross/docs/s4vStorm.pdf
3. Comprehensive answer by Divye Kapoor to a Quora question on this topic:  http://www.quora.com/What-would-you-choose-between-Flume-Yahoo-S4-and-Backtype-Twitter-Storm-and-why

 

In addition to S4 and Storm both of which are open-source, there are other commercial products in this space such as InfoChimps Cloud::streams .

Big Data and Graph Databases

One of the more interesting upcoming growth areas in Data Science is the use of Graph Databases and graph-based analytics on large, unstructured datasets.

This is a natural next step along the progression we’ve already been on: first raw MapReduce and Hadoop for large-scale data processing, then tools and frameworks (such as Streaming and Cascading), and then the addition of SQL-like layers on top such as Pig and Hive.

Why do we need Graph Databases?

Today applications and devices generate a flood of data. This high volume of data is typically incredibly dense and highly related; it does not fall neatly into pre-defined schemas. Facebook’s Open Graph and twitter’s interest graph are two obvious examples, but there are many other domains where this applies, such as Healthcare, E-commerce and sensor data.

This type of data of highly-connected entities is not easily modeled using traditional relational schemas; instead, using graph data structures makes it easy to represent connected data and to perform rapid analysis on these large datasets.

In a graph database, data is stored as nodes and relationships; both nodes and relationships have properties. Instead of capturing relationships between entities in a join table as in a Relational Database, a Graph Database captures the relationships themselves and their properties directly within the stored data.

To quote Derrick Harris from his GigaOm article:

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are.

Although graphs have recently become more popular because of their applicability in modeling social networks, graph analysis can be widely applied to analyze any kind of relationships.

Modeling the data as graphs allows data scientists to discover localized patterns; i.e. how are specific items related to other items. Even with large datasets, most analytics queries end up acting locally within a graph. All graphs share common patterns - simple examples include the diamond, butterfly and star patterns, however these simple patterns can be composed into arbitrarily complex patterns.

Graph Database: Neo4J

One of the exciting entries in this area is Neo4J, a Java-based open source graph database from Swedish company Neo Technology. Neo4J stores graph data directly and offers large-scale horizontal scalability using replication; in addition it offers ACID transactions and indexes similar to a traditional database. Neo4J also has a REST API and its own graph query language called Cypher.

A good starting point to learn about Neo4J is Robert Scoble’s video interview with Neo Technology’s CEO, Emil Eifrem. To me, the most interesting section is where Emil nails the value of graph databases in general (at the 6:30 mark in the video):

Sophisticated intelligence and reasoning is all about how are things related to one another. … Whenever the value is in the connection between things, that's when a graph database excels.

You can watch the video here:  http://vimeo.com/56040747

Interestingly, he says that the next step for Neo4J is to work on transparent partitioning, to enable the database to automatically co-locate highly-connected nodes. There is lots of good information about graph databases on the Neo4J site, here: http://www.neo4j.org/learn/neo4j .

As a student of data visualization, I also found the following infographic an interesting way to represent Graph databases:

[If the graphic does not display above, click here to view it on visual.ly .]

Other Technologies

Clearly, this is a hot area right now, with lots of different technologies and frameworks coming up. One offering comes from Intel: GraphBuilder is an open source Java library for constructing graphs out of large datasets for data analytics. YarcData, a Cray company, has a graph analytics hardware platform for real-time analysis: an in-memory appliance called Urika . And of course, there is an Apache project: Apache Giraph that addresses graph processing of big data.

This is still very much an emerging space, so it’s not clear which of these projects will evolve into popular successes. But given the explosion of data all around us and the prevalence of relationships among entities within that data, it’s certain that this area of computer science will receive a lot of attention in the near future.

 

Software Development using the SCRUM process

I just recently completed a three-day SCRUM training class, and I have to say that I quite enjoyed it. Normally, the thought of sitting in a chair listening to a corporate trainer for several days in a row fills me with dread, regardless of the quality of cookies and snacks provided; the little participatory exercises feel entirely contrived and artificial. Frankly, I went into the class somewhat skeptical of Scrum ...

But this class turned out to be very high quality and valuable. The presenter, Kenny Rubin of Innolution, a certified Scrum Trainer of the Scrum Alliance, was knowledgeable, informative and interesting; most important, he has enough experience with software development practices and teams to be able to give it a real-world flavor.

That is, the dogma of "the Scrum way" was tempered by the realities of professional software development. (Those of you working on a real-life software development project day to day, or managing software developers, know what I mean - my 20+ years of industry experience have taught me that most software projects are understaffed, have too many requirements, ultra-aggressive deadlines, and repeated requirements changes; it's a wonder we deliver anything at all!)

SDLC: New and Improved with Scrum!

SDLC - the Software Development LifeCycle - is not a new concept. I was first introduced to the concept during my graduate software engineering class at UCSB during the late 80s - by Prof. Richard Kemmerer, he of the red hair and temper to match. (The UCSB of the time was one of the many hotbeds of a radical new movement called the "Free Software Foundation", but that's another story!). The good Professor had lots of industry experience and tried to explain to us the myriad challenges of building and maintaining large-scale software in teams, so very different from the limited computer science coding assignments which we could simply turn in and forget. 

A key exercise in the class was to compare the then-prevalent Waterfall model of software development, with Barry Boehm's new Spiral model; the latter was clearly superior in terms of dealing with the uncertainty of changing requirements. This was in the late 80s; since then, I've seen countless new software processes and programming models come and go: Structured, Functional, Object-oriented, Booch, Rumbaugh, Jacobson, Coad, Yourdon, followed in the 90s by UML, Pair Programming, XP, and then in the 2000s, Lean, Agile, and so on. After seeing so many "great new thing" theories come and go, it's hard to take the flavor du jour too seriously.

After looking at the Scrum process though, I'm optimistic that it is likely to have greater staying power than some of the earlier short-lived fads. Scrum seems to recognize reality a bit more than some of the other recent processes that have generated buzz [e.g. putting two developers on creating a single piece of code when only one will do - is that model going to take over the world? Really?]; Scrum attempts to include this reality into the flow rather than trying to bend it to an artificial process.

Another advantage is that Scrum tries to take out-sourcing and off-shoring into account - relatively recent trends that are likely to continue to grow. Finally, Scrum borrows heavily from the models and processes that came before: not just Lean and Agile, but even earlier. The iterative sprints approach of Scrum is conceptually remarkably similar to Boehm's spiral model, isn't it?

At the same time, there are some glaring deficiencies in the Scrum model that will need to be addressed. 

Scrum Process: Key Points

Having been through the official Scrum training, I will highlight the key points of the Scrum process in this section.

The overall Scrum process definition is broken into Activities, Roles and Artifacts. (A canonical place for definitions is Mike Kohn's Mountain Goat Software web site: Introduction to Scrum. )

Sprint Activities

Scrum is an agile process that relies heavily on specific, prescribed activities. Since development is organized as a series of sprints, scrum specifies a set of activities that are repeated for each one. 

Sprint Planning Meeting: Figuring out which of the high-priority product backlog items can fit into the next sprint.

Daily Scrum Meeting: A quick daily meeting during which team members share their activities for the past day, planned activities for that day, and identify any impediments to progress.

Sprint Review: Conducted at the end of a sprint, this meeting provides an opportunity to demonstrate the functionality added during the sprint, displaying it to product owners, users and stakeholders.

Sprint Retrospective: The whole team participates in this meeting, which is an opportunity to reflect on a sprint that is ending and opportunities to improve for the next sprint.

Overview Image: (c) Mountain Goat software

Roles

Different members of the team are assigned specific roles in a Scrum process; these roles prescribe specific behaviors and tasks for the team members.

Scrum Master: The Scrum master is the team's coach, and is essentially a facilitator rather than a project manager or line manager. The main role of the scrum master is to remove project impediments that affect the team, and provide coaching about the process, enabling the individual sprints and meetings to go smoothly.

Product Owner: The product owner comes up with a vision for the product, and translates it into a prioritized set of features required in the product (the product backlog, in Scrum parlance). The product owner answers questions about the requirements and helps the team make trade-offs between different options.

The team: The team as a whole is the central player in the Scrum process. Each person contributes in whichever way they can to complete the work of each sprint. 

Artifacts

Scrum defines various artifacts that are used at different stages in the process.

Product Backlog: The product backlog is a prioritized list of "user stories" that, taken together, define the functionality that remains to be added to the product.

Sprint Backlog: The sprint backlog is much more limited in scope than the product backlog; it's a list of tasks the team needs to perform in order to deliver the functionality committed to for that sprint.

Release Burndown Chart: This chart shows the amount of work remaining in the release. Similarly, a sprint burndown chart shows the amount of work remaining in the sprint. These charts provide a visual indication of progress towards completion of the project.

Conclusion

One of the key takeaways for Scrum, is the focus on a unified, cross-functional, empowered Scrum Team. This team includes software developers, testers and web designers, as well as the product owner and scrum master; in short, everyone required to specify, build and test the product. And this team is both empowered to make decisions with regard to scope, schedule, features and resources, as well as held responsible for the results. An interesting concept!

And what of those shortcomings of Scrum that I alluded to earlier? Those will follow in a future post. Stay tuned ...

[Note: some of the material for the training class referenced above was licensed from Mike Cohn of Moutain Goat Software.]

Crowdsourcing Charity: Sustainable Innovations

Paul Buchheit (founder of Gmail and Friendfeed) wants to donate money to charity. He's trying an interesting experiment in collaborative charity, where he's planning to crowdsource the list of ideas he should consider funding. The ideas he gets are only recommendations, he says, and he will ultimately choose the best idea that appeals to him and that makes sense.

In that spirit, let me make a suggestion for Paul and others like him in Silicon Valley, one that takes a different perspective. If you're a successful tech founder or VC who likes philanthropy, consider this a free "business idea for giving back" that you can adopt as your own if it interests you. [Now I'm no VC, nor am I a rich, successful tech founder, so I have no idea if the approach below has any appeal to these folks; nevertheless it is an interesting thought exercise.]

Silicon Valley skills

In general, anyone with a bit of time or money and the inclination to help, can donate to charity, and many people do (bless 'em!). The most effective donations, however, leverage the unique skills and perspectives that one can bring as an individual, using one's own talents and gifts in addition to money, to help others.

So what could Silicon Valley entrepreneurs do? How about this: instead of donating money directly, why not use it as seed funding to build an organization that can keep on giving? After all that is what tech founders (and VCs) do best: they fund startups that take some initial amount of money and then leverage it massively to create a disproportionate impact on a target market - or die trying. Why not take the same approach to charity? Build an organization that leverages other people's time and money, and creates massive impact, much bigger than the pure monetary donation would create.

But what type of organization? There are already many, many charities, of many different types. How do you make a difference?

Third level of Abstraction

Give a man a fish, it is said, and you feed him for a day; teach him to catch fish, and you feed him for a lifetime. Extending the metaphor, what if you could create a construct that continuously spawns ideas, technologies and approaches for teaching everyone to catch fish?

What the world needs more than direct charity, is locally-sustainable ideas that allow populations in underdeveloped areas to take care of their own core needs - clean water, sustainable food production, cheap education and economic development - by themselves. As a philanthropist, what if you were to establish an organization to crowdsource innovations and ideas to bring about these very changes? You could create a mechanism to encourage such attempts and to reward successes.

The idea: So the idea I'm proposing, in a nutshell, is this: instead of making a direct monetary donation to charity, set up a recurring contest or start an organization that encourages and rewards innovations in technology, design, reuse, financial models, human factors or ecology conservation, for the social good. Most important, these innovations must lead to local, self-sustainable solutions that do not depend on ongoing external help to keep going.

Real-Life Examples: Business Models and Innovations

There are many examples of business models for encouraging innovation through crowd-sourcing:

• Competitions, such as the Netflix prize for improving their recommendation algorithm (which seems to be beaten) or NYC's design competition for post-disaster provisional housing
• Paul Graham's Y-Combinator micro-funding model for seeding startups
• Social incubators like socialedge

I'm sure there are many others that are not listed here, feel free to list them in the comments if you know of any.

What kind of innovations should such an organization look for? Ideally, these would be locally-managed, self-sustainable or reuse-based product design ideas, original approaches or creative financial models.

Here are some examples of innovations of this type:

• Clean water powered by playgrounds: Playpumps International
• SHRIMP Refugee Housing, which folds up so small that four of these can fit into a standard shipping container, and each one provides shelter and fresh water for a family of 4. [A slide show from Wired Magazine has a whole list of innovative shelter designs for providing emergency housing to disaster areas and refugees.]
• Buy a pair of chickens for AUS $10 and help a family get started in sustainable chicken farming. [via Darren Rowse of Problogger]
• Lighting up remote villages in India, using technology and innovative financial models: SELCO [via Sramana Mitra]

The world needs many more innovations of this type. Why not explicitly encourage them?

---

Why You Should Market Your Brand on twitter - NOW!

twitter has been an emerging phenomenon for a little while now. There were some very public performance problems and outages in the beginning, but the company seems to have moved past those now. twitter usage seems to be gaining critical mass, crossing over from the technology early-adopters to non-technology leaders, companies and celebrities (imagine Stephen Colbert and Jon Stewart of Comedy Central bantering about twitter on Election night).

There has been much discussion and debate recently about whether it is appropriate and worthwhile for businesses to spend time marketing their brands on twitter. In this article, we take an in-depth look at the advantages of doing so.

Given the ongoing dramatic growth of twitter usage, and if twitter adoption has indeed crossed over to the mainstream, then it seems inevitable that brand marketing will eventually catch up with it on a large-scale; adding twitter marketing to company budgets will become an increasingly easier decision once "everyone is doing it". However there are compelling reasons why you should consider starting your twitter marketing efforts right now rather than wait until there is a herd.

So if you need to convince your boss to let you spend time on this activity - promoting your brand on twitter - here is a list of reasons to get into twitter-marketing now:

Early mover Advantage

Years ago Seth Godin's book, Permission Marketing, launched a revolution in brand-customer interaction. Imagine if you had gotten into Email Marketing at that time, when it was first getting started - it would have been so much easier to get and hold consumer attention, make your brand stand out, and establish a solid position. In contrast, getting the same level of attention today using email is almost impossible, since practically every company practices email marketing; consumers face email fatigue and get deluged by spam, and are reluctant to give out their real email addresses to companies.

For twitter marketing it's not too late, however - it is still in its early stages, and this time you do have a chance to jump on it before most of your competitors (as @Southwest Airlines, @Whole Foods, @Baskin Robbins and @Zappos have already done), gaining momentum as it becomes mainstream.

Influencing the Influencers

A common argument against the twitter marketing strategy is that many of the users of your non-technical product may not yet be twitter users, and you should always market where your users are. This seems like common sense. However, it overlooks a fundamental truth: users often tune out the direct messaging coming from a company anyway, and instead use indirect sources to help them make decisions.

In Malcolm Gladwell's influential book, The Tipping Point, he describes how word-of-mouth influences brand popularity, trends and buying decisions. If you can get some of the super-connectors and influencers to adopt and evangelize your product, then interest in your product spreads like wildfire and demand soars.

How do you reach these influencers? In today's world, it's a safe bet that the influencers are heavy users of new media methods like blogs, social networks and twitter, even if your field of interest is not particularly technical. twitter is especially effective in encouraging and responding to these highly-connected users; the medium gives you direct, instant access to even the most popular of these folks at all levels. (For example, here are the twitter accounts for publishing guru Tim O'Reilly: @timoreilly, VC Fred Wilson: @fredwilson, VoIP pioneer Jeff Pulver: @jeffpulver, and even celebrities like Shaquille O'Neal: @the_Real_Shaq and Lance Armstrong: @lancearmstrong ). By connecting with them and interacting directly, you have a much better chance of getting mentioned on blogs, in forums and discussions, and being highlighted in their media outreach.

Customer Engagement

More than any other medium, twitter allows you to connect directly with your users and engage them in a direct, immediate, two-way information exchange. You can get timely feedback: users with negative comments can be heard and their concerns addressed; users with positive comments can be encouraged to become evangelists. You can even learn of alternative ways your product is being used, or discover new markets that can be targeted.

This strategy of connecting with individual users and building a "long tail" following nicely complements the previous strategy of targeting "influencers". Some studies contradict Gladwell's conclusions, showing instead that users trust recommendations from peers and friends far more than those of a few super-connected individuals; the safest bet is to court both types.

Low Cost

In general, twitter marketing is very cheap and easy, and shouldn't have any significant impact on spending for your other Marketing programs; important, since cost is a major concern in the current challenging economic climate. Success in connecting via twitter takes time and authenticity, not a large budget!

And if your company still doesn't buy it? Create a personal account on twitter (hey, it's free) and start connecting! You can begin engaging others in your field and thought leaders in other fields - by engaging at a personal level, you can start to build your own network of followers.

Oh, and if you'd like to connect with me, you can find me as @nitink on twitter - I always follow everyone back!

For further reading, here are some great articles on twitter Marketing:

Great Examples of Corporate twitter use - Jennifer Laycock
Using Twitter for Brands or Corporate Identities - Dawn Foster
How to use Twitter as a Twool - Guy Kawasaki
Looking for Mr. Goodtweet - How to pick up followers on twitter - Guy Kawasaki
Free (but welcome) advice on the corporate use of twitter - Elliot Ng
A User's Guide to Twittering - Wall Street Journal
Professional Branding Comes to Twitter - Svetlana Gladkova
Why I Love Twitter - Tim O'Reilly

---

Should you invest in New Media Marketing in a down economy?

Over the past couple of years, companies and brands have been increasingly engaging with New Media, which can be broadly defined as blogs, feeds, wikis, micro-blogging (twitter), social networks (such as LinkedIn and Facebook) and social bookmarks.

As the economy goes sour and budgets decrease, however, does it make sense to continue to invest Marketing dollars into these word-of-mouth, engagement programs?

Here are some of my initial thoughts.

As with everything else, it comes down to a cost-benefit analysis, comparing these types of programs with alternatives such as direct marketing, trade shows, magazine advertisements and so on.

Benefits

On the benefits side, some of the core value propositions of New Media Marketing are pretty strong:

Increased Brand Awareness
Research has shown that consumers most often buy brands that they are already familiar with; generating brand awareness and familiarity is thus as important as pure direct conversion rates.

Engaging with Customers and others in the Ecosystem
The first simple step to irrelevance for any brand is when a company stops listening to those who are most affected by its products. By engaging with customers, users, suppliers and others, a business can get direct, rapid feedback and potentially head off problems before they turn into major crises.

Getting Product Ideas
Discussions about your product and its use in social media are a fertile environment to get new product ideas, especially involving novel settings or uses of your existing products or discovering gaps that can be filled.

Lead Generation
In the same way, social media can be mined to harvest meaningful, highly-qualified leads, significantly reducing the cost of customer acquisition.

Find, Encourage, Enable and Reward Advocates
Who are the users who really like your products and are most vocal about it? How can you provide encouragement and rewards and remove barriers? By engaging directly with users, you can find the answers to these questions.

Participate in the Conversation
If your product is at all popular, you can be certain that customers and users are already talking about it somewhere, somehow - whether you know it or not. If you can participate directly in this conversation, you can at least react to what they are saying, even if you cannot and should not directly steer the conversation.

Monitor => Engage => Influence => Monitor
This is the virtuous cycle New Media enables. By its very nature, social media is interactive; it is a dialogue, rather than the outward-bound one-way Marketing messages of the past.

Costs

By its very nature, social media marketing is low-cost compared with traditional approaches. What it takes to be effective in this field is time (which, of course, also translates to cost) and authenticity.

The Marketing Mix

So is it worth investing in this type of user engagement? Obviously, the decision depends on your particular products and your environment, but increasingly in the future, the answer to that question is going to be another question:
Can you afford not to be a part of the conversation, especially if your competitors already are?

---

Terrorist Attacks in Mumbai, India

As I write this, the terror situation continues in Mumbai, India. The Times of India first reported the initial news sometime around 10 am this morning, which means it has already been going on for over twelve hours. Although Mumbai has suffered from other significant terrorist actions in the past few years, this is the most violent and long-running attack that I can remember. According to the latest reports, at least 101 people have been killed and hundreds more have been injured. The beautiful Taj hotel building has flames shooting through its top floors and terrorists are still holed up in the Oberoi Trident hotel.

One tragic outcome is that the Mumbai police have been hard hit by this attack, having lost three of their top people: ATS Chief Hemant Karkare, Encounter Specialist Vijay Salaskar and ACP Ashok Kamte, have been killed. Our thoughts and prayers are with their families.

Atypically for terrorist activities in this region, this particular attack seems to have specifically targeted foreigners. That, and a high level of coordination across multiple targets, points to a new level of sophistication for the local terrorists.

The best way to monitor the situation from afar, I've found, is via twitter. The catchall twitter tag #mumbai (thanks to @fredwilson for pointing this out) is currently getting multiple updates every minute. Although CNN is much more visual and includes more analysis, it's hard to beat twitter for real-time updates of breaking news. Gaurav Mishra provides some great commentary on the role of real time citizen journalism in covering these attacks (as does TechCrunch).

"The purpose of terrorism," Vladimir Lenin once said, "is to terrorize." Perhaps, but I think that these terrorists have underestimated the resilience of spirit of the people of Mumbai. In the same vein as New Yorkers, the Mumbaikars will survive this brutal act of terrorism and eventually emerge the stronger for it. Terrorism has become a scourge worldwide, but in the end, it cannot succeed; by simply going on with their daily lives, the regular folks - unsung heroes all - will eventually prevail.

---

"Doctor, I read it on Google"

As I observed earlier when writing about Emerging Trends in Healthcare, the relative knowledge levels are changing rapidly in the physician-patient relationship. In the past, the physician was clearly the expert and the patient simply accepted the doctor's judgement and counsel without question.
 
Today's web-savvy Healthcare consumers, however, are adept at finding comprehensive and detailed information about their own condition and disease. In many ways, in fact, they have an advantage over the doctor - a patient only has to learn about one particular condition (or a few); whereas a physician, even a specialist, must necessarily cover a far wider range of medical knowledge.

By and large, most doctors are uncomfortable with this sudden shift in patient knowledge level and the potential for second-guessing the doctor's opinion, that goes with it. New research from Microsoft suggests that the doctors may be right!

Microsoft Research has just published an article on this topic: Cyberchondria: Studies of the Escalation of Medical Concerns in Web Search. The study authors found that patients who perform Web Searches about their symptoms, tend to have an exaggerated sense of their illness. Says the article:

However, the Web has the potential to increase the anxieties of people who have little or no medical training, especially when Web search is employed as a diagnostic procedure.
...
We show that escalation is influenced by the amount and distribution of medical content viewed by users,

Overall, though, this finding is hardly surprising. As any layperson who has ever perused a medical encyclopedia knows, just reading about the specific symptoms for any random disease makes one start to start imagine those very same symptoms - it is part of the human condition! Obviously the same thing happens when patients read medical information online.

At the same time, there is a distinct upside for patient health in having all of this medical information easily available and accessible to all. Informed patients are in a better position to understand what is happening to them; they can handle it better and can take better care of themselves in addressing their own health problems.

In any case, this is a one-way street. The genie is out of the bottle now and patients will be increasingly better-informed in the future. Whether this is preferable or not, physicians had better gear up to deal with this change.

---