The Software Abstractions Blog

Beyond Machine Learning: The Future of AI

 

 

Professor Gary Marcus of NYU wrote in the New York Times last week that AI is stuck and still has a long long way to go. Even the advanced and trendy techniques of “deep learning” fall short of basic levels of intelligence.

The typical Artificial Intelligence setup today consists of sophisticated machine learning algorithms implementing analytics in the cloud on very large data sets. The implementation usually works as follows:

1. Collect data at scale - text or images
2. Run machine learning algorithms and deep learning technologies to build a model or network to solve the problem (classification, clustering, predictive analytics, etc.)
3. Data captured at the end point is sent to the model to answer a specific question

 

 

In particular, Google has made some breathtaking advances in AI through its relentlessly statistical approach (for a better understanding of this approach, watch this video by Peter Norvig); Facebook, Amazon, Uber and others are not far behind. The state of the art  for this approach will continue to improve as cheaper computing power, more powerful algorithms and ever larger datasets become available.

The next step of this evolution is to push this processing further out towards the end points, i.e. Edge computing, which refers to generating, collecting and analyzing data at the site where data generation occurs. As mobile devices offer increasingly more computing power and storage at lower cost, the nature of AI will change from a static algorithm running on a server in the background to an on-board agent at the end point that applies or even fine-tunes analytics models in real time right in the field.

Another possible next step is Evolution strategies, closely related to reinforcement learning, for rapidly training deep neural networks. For more information about this type of strategy, check out this post on the OpenAI blog.

 

Beyond Statistical Analysis

 

To get to the next level of AI, what is needed is not simply more computing power and better statistics; instead, it seems likely that that future discontinuous advances will come from combining entirely new approaches with the existing statistical and rules-based methods that have been the focus of Artificial Intelligence so far.

Intelligence and reasoning are difficult challenges; how do we humans do it so well? It’s simple - we cheat! We don’t usually look at a picture or tweet and try to extract every last ounce of information from it in isolation or based on a single homogeneous dataset; instead, we leverage many different types of outside information to get all the help we can. Even when we’re not actively seeking more information, we leverage our existing deep knowledge of the world and the relationships between objects; we look at the context of the data; we use past history as a guide; we can ask clarifying questions, collaborate with others and share experiences. Most important, we learn - both from those who have gone before us and from contemporaries who work in other areas.

This is why a five-year old with very limited real-world experience can easily identify the image of a cat or a dog most of the time, while Google’s algorithm, in a very significant AI achievement, has only recently learned to distinguish between cats and dogs after studying millions of images.

 

Alternative Approaches

 

 

What are some alternative approaches that could be used to mimic real intelligence? Here are some possibilities that could be combined with existing approaches to enhance the effectiveness of AI systems.

1. Knowledge representation: As a first step, building an ontology to define and relate core concepts, classification hierarchies, class properties, object relationships, and so on. This is already happening in a limited way within specific domains; but the overarching end goal is to build a rich model of the entire world and the classes and objects within it.
2. Multi-modal: Instead of analyzing a single data set in isolation, using multiple modalities to provide richer context and meaning; such as by combining data from images, text, audio files and macro environmental conditions. If the image displays an animal with four legs and its sound is identified as barking, then classifying it as a dog becomes a much simpler exercise.  
3. Interactive: Leveraging user interaction to refine the results of a passive search through existing data sets. For example, in a classification problem, the AI can ask additional questions to validate or refine its choice once it narrows down the solution space to a few items using standard algorithms. An extreme version of this approach is used by the 20Q toy that asks 20 questions to correctly classify an astonishingly disparate set of objects.
4. Fuzzy boundaries: Beyond solving the exact problem specified, an AI that offers a sufficient number of “good enough” solutions that could be obtained by relaxing one or more boundaries in the problem specification could also be extremely valuable.
5. Dynamic models: Rather than picking the one “best” trained model or network to use for solving problems or making predictions on new data, a meta-algorithm could dynamically switch between multiple models or modify existing models over time as external conditions change.
6. Temporal progression and causal effects: Using past information and experience to guide guesses for solving the current problem. This is already accounted for in today’s machine learning approaches, but those approaches are statistical rather than qualitative; reasoning about causality and time has the potential to enable the AI to handle discontinuous changes. For example, in stock trading it is hard for a statistical algorithm like MACD to predict the sudden impact on stock price when a company’s quarterly results announcement varies widely from the “market consensus”; but even novice traders can easily predict the direction of the resulting discontinuous change.
7. Collaboration: Running multiple algorithms in parallel can provide different results, which can then be accepted, rejected, combined or refined. This is not a new concept [e.g. Random forests are a kind of averaging mechanism], but the real value is in combining the results of multiple disparate approaches.
8. Strategy representation and Planning knowledge: Defining goals, comparing high-level strategies to achieve them and making plans based on the chosen strategy - this progression is currently a uniquely human trait. Computer scientists of the future will find better ways to represent strategy and planning knowledge and to reason inferences from it. As this knowledge gets better organized, AI algorithms could plan and dynamically switch strategies as needed to meet different goals.

 

Most of these ideas are already known, however they are not widely used in typical AI solutions at this point in time. The current focus on optimizing machine learning approaches is valuable in and of itself; however, in the long term the real value is to be found in the system that combines statistical machine learning with multiple modalities, user interaction, dynamic approaches, and planning strategies.

 

AI - an intelligent assistant

 

 

Further beyond that, truly great AI will be able to handle exceptions that break the rules, that don’t fit into the statistics or known data or logic. For example, even a child will confidently recognize the above image as a dog - even though it has only three legs and almost no tail, which could potentially confound an AI algorithm. (To paraphrase Groucho Marx: This is so simple a five-year-old could understand it; let’s ask a five-year-old.)

In fact Google’s AI would not be fooled by this particular image, but it can be fooled by optical illusions and pictures of noodles. A new approach called Generative Adversarial Networks can generate such images, which in turn helps to improve the image recognition algorithms.

More generally though, reliance on a single approach or algorithm leaves the AI vulnerable to random data or malicious attacks that target its blind spots; combining multiple disparate approaches makes the solution significantly more robust and less likely to fail even in the face of unexpected input data.

When AI gets to that point, it will become a truly able assistant, to help us humans reach ever higher levels of success.

 

Edits:

1. Of course, others have been thinking of these things before me. Here's a terrific post by Nikola Rohrseitz on this topic:  Beyond Machine Learning
2. Will Knight at MIT Technology Review brings up one of the biggest challenges related to deep learning:  The Dark Secret at the Heart of AI
3. One promising alternative approach for AI is:  Bayesian Program Learning

 

Blurring Boundaries: Apps or Infrastructure

 

 

IT and Co-location

In the distant past of software development - by which I mean, ten years ago - software companies made a clear distinction between Application development and Infrastructure management. Most companies had a dedicated IT team to configure, provision, install, manage, maintain and update the hardware infrastructure needed to run their software applications; separately, the software engineering team focused on building the software platform and applications. The hardware was typically hosted in a co-location facility. The IT team essentially “owned” this hardware, with responsibility for all the functions associated with it.

Over the past five years, the rise of the DevOps movement in startups and in large cutting-edge technology companies such as Google and Netflix, especially in Silicon Valley, created the first bridge between these two areas of software applications and infrastructure.

According to the DevOps philosophy, configuration and management of the underlying infrastructure services was treated more like software development, using higher-level languages like Ruby and Python, and standard software application methodologies and tools like agile development, source code control (e.g. git), versioning and branches, bug tracking (e.g. JIRA) and test automation. DevOps as a function came increasingly within the purview of the software engineering team rather than that of traditional IT.

Cloud Computing 

The rise of Cloud computing and “Infrastructure/Platform as a Service” offered by vendors like Amazon Web Services (AWS) has further accelerated this trend. For example, AWS has a built-in command-line tool called AWS CLI, as well as software API bindings for popular languages like Java, PHP and Python, that allow a computer program to perform actions for the entire Infrastructure life-cycle: from provisioning new hardware, to scaling it, managing and monitoring it, through the final tear-down when it is no longer needed.

Underneath the covers of course is an actual data center, with real servers, racks, power and networking equipment, security devices and policies, NOCs, and all the other “traditional IT” functions. But this layer is completely “virtualized” or hidden from the application developers, who can treat it as yet another software API or service.

The public cloud computing vendors, and private cloud data centers within large companies, still undertake all the classic IT projects and perform the same tasks as before. What is different however, is that from the perspective of a typical software startup or mid-sized SAAS application provider, a separate Infrastructure function has ceased to exist; instead it has been absorbed into the software engineering team, whose members are now expected to possess skills for managing cloud computing APIs, just as they are typically expected to understand how to interact with an operating system, or with a web application server or a handheld device API. This will not only affect the composition and operating practices of software engineering teams, but will also enable them to develop code at an increasingly faster pace in the future.

OpsWorks and Docker

A great example of this new approach of treating infrastructure provisioning, management and monitoring as software code is embodied by the AWS OpsWorks service, which uses an open source framework called Chef for application management. Chef programs, appropriately named “recipes”, can be used to configure servers, implement auto-scaling, apply security policies and monitor the resulting infrastructure stack.

The most recent step in this evolution is the increasing popularity of deployment containers, especially Docker (but also Rocket and LXC). In an explicit analogy to shipping containers which provide for all types of disparate goods to be handled in a uniform way by different modes of transportation, Docker enables application developers to interact with their infrastructure environment in a uniform way, regardless of whether the container is hosted on a laptop or on a data center server with high computing power. Both Amazon’s OpsWorks and their newly offered Container Service support container deployment.

Implications for Development Teams

This trend will have some interesting implications for the software development team in a typical small to medium-sized software-as-a-service (SAAS) company:

1. Team composition: Software developers in application development teams will need to understand both infrastructure (at the API level) and traditional application logic (algorithms + data structures). In larger development teams, this may be structured as separate sub-teams, each with a different individual focus.
2. Faster turnaround: Since the PAAS vendors allow computing resources to be added or removed with the click of a button or with a single API call, dev teams can now react much faster to the needs of customers or individual developers. This enables auto-scaling based on customer demand and individual development sandboxes for experimentation, which will change how developers work on their code.
3. Cost: Auto-scaling removes the need to provision servers up-front for peak demand. Along with economies of scale at the big data centers, this makes infrastructure resources significantly cheaper. At the same time, unless scaled-down appropriately, unused resources continue to incur periodic costs even when they are idle which can get very expensive. Due to these two competing forces, managing virtual infrastructure cost will be an important consideration for software architecture, code and development processes in the future.
4. Approach: Treating infrastructure configuration and management in a similar way to source code is a fundamentally different approach, lending itself well to the standard tools, processes and abstractions of software engineering as described previously. This will enable software development teams to take an agile, iterative approach to infrastructure development, analogous to standard software “scrum” practices.
5. Leaky abstractions: One challenge with the new approach is that software developers will be more aware of the details about the underlying virtual infrastructure. This could lead to “leaky” abstractions where they take advantage of local quirks within their particular infrastructure. Introducing containers like Docker will go a long way in mitigating this issue.

The Future: Cluster Management

As applications increasingly adopt a container-centric approach for deployment, two additional types of services and tools will become very important:

• Container management: This type of tool enables container deployment, orchestration, management and scheduling. A great example is the open source Kubernetes project from Google, others include Lattice and Marathon.
• Cluster management: This is the concept of providing a higher level of abstraction across the entire data center, i.e. an entire cluster of servers is treated as a single operating system kernel capable of running multiple projects simultaneously. Initially proposed by Google for their internal Borg project, this is now available to everyone through the open source Apache project Mesos. The new open source operating system CoreOS is designed from the ground up to run Docker containers across a cluster.

As these types of tools become mainstream in the future, developers will only need to focus on building out their application components inside standard containers and specifying the needs and requirements of those containers; it will be up to the underlying infrastructure services framework to optimize the required resources and cost to provide the application with everything it needs. 

 

US Health care: Radical Transformation

 

 

 

The health care system in this country has been undergoing a major change over the past ten years and the pace of change is accelerating. We are now at an inflection point and it will never be the same again. This post examines the external forces that are driving the change and highlights the potential and likely effects.

 

External Forces

 

Legislative Changes

Over the past few years, the federal government has enacted a series of legislations and regulations related to Health IT that have created a profound long-term impact on the nation’s health care system. Some of the key laws are:

• HIPAA 1996: Although this law primarily focused on protecting health insurance coverage for workers and their families, it was the first shot aimed at increasing the efficiency of the nation’s health care system. It required the establishment of national standards for the use and dissemination of health care information, created a unique National Provider Identifier (NPI) enabling covered providers to participate in standard transactions, and laid the groundwork for strong security and privacy protections for patient health data.
• HITECH (as part of ARRA) 2009: This act proposes the meaningful use of interoperable electronic health records throughout the US health care delivery system as a critical national goal, with a financial incentive to providers and hospitals who can demonstrate that they have engaged in efforts to embrace this technology.
• ACA 2010: In addition to consumer protections, this act has a strong focus on improving the quality and lowering costs of health care services, including a Patient’s Bill of Rights, an emphasis on preventive services, and encouraging integrated health systems.

Taken together, this series of laws lays out a blueprint for health providers and organizations to follow. It strongly promotes the use of electronic records, interoperability, integrated care, patient empowerment and a focus on prevention, with the ultimate goal of driving down costs and improving the quality of care.

 

CMS Cost Reduction

As one of the largest payers in the industry, the Centers for Medicare and Medicaid Services (CMS) have a huge impact on the US health care system; even private insurers eventually tend to be consistent with CMS guidelines. In an effort to promote higher efficiency and reduce fraud within the health care system, CMS policies in recent years have been on a long-term path towards reduced reimbursements, increased documentation and stricter standards. As an example the recent introduction of the readmissions reduction program and associated penalties is forcing hospitals to put more focus on post-acute care and data interoperability for discharged patients.

 

Engaged Consumers (Patients)

At the end of the day, we’re all patients. Today’s consumers have unprecedented access to information (both accurate and misleading) about themselves and their conditions, thanks to Google, the world-wide web and a plethora of connected low-cost monitoring devices. There is an accelerating trend toward self-confidence, self-help and advocacy among patients in the US and all over the world. Tomorrow’s patients will be equal partners with their health providers in the planning and implementation of their treatment plans.

 

Technology

Software and hardware technology is making great strides in many areas of health care. Together these changes build on each other, creating an unprecedented climate of advancement in health technology.

Broadly, these innovations can be classified into a set of distinct groups:

• Cloud Computing: The increasing popularity of data center infrastructure and platforms as a service (IaaS/PaaS) has brought a new level of flexibility, scale and reduced cost to software applications that collect and analyze large-scale data sets. Advances in container and cluster management enable applications to run complex machine learning strategies on these massive data sets at a reasonable cost.
• Smart Devices: Smartphones are the true “personal computers” of our age; they are easily available, low-cost, always nearby, and almost ubiquitous even in less developed regions of the world. Any health technology for patient interaction, for example frequent mobile vision testing for patients with diabetes, can be quickly rolled out globally with ease, simply by turning it into an iPhone or Android app.
• Sensors: Powerful low-cost sensors - such as wearables, ingestibles and in-home devices - that can provide actionable, continuous, real-time data to health providers are becoming more common and will be deployed routinely for patient monitoring in the future.
• Biotech: Sensors, devices and algorithms in the fast-growing biotech industry continue to move forward. For example, an iPad-sized DNA sequencer can make this type of sequencing commonly available at low cost.
• Data: A massive focus on Big Data technologies and solutions has dominated the tech industry for the past few years, creating revolutionary techniques for handling data at scale. New open source software technologies for non-relational databases, data ingestion pipelines, map-reduce and other parallel processing algorithms, predictive analytics and unstructured data mining enable data collection and aggregation, analytics processing and visualization of results on large scale data streams in near real-time at a reasonable cost.
• Interoperability: As a key focus of the national health IT policy as articulated by the Office of the National Coordinator (ONC), the need for health data interchange among disparate health systems has created a slew of new technology solutions, including integration and interface engines, transport protocols such as the DIRECT project and data formats like HL7 CCD.
• Personalization: A key attribute of many of the new health technologies is personalized solutions and products; for example, applications that adjust to the user’s preferences, 3D-printing for custom manufacturing, medications that are specific to the patient’s DNA, and the potential uses of nanotechnology for a specific individual.
• Security and Privacy: In health care, more than in most other domains, the security and privacy of patient clinical data is extremely important, with enforcement and penalties encoded by the HIPAA Privacy Rule. An entire subset of the health technology industry is focused on products and services to help covered entities and business associates stay in compliance with these regulations.

In conjunction with the strong focus on improving health care in government and industry, in the past few years there has been a surge in venture capital investment in this domain. There is tremendous excitement in the tech community about health care opportunities, with lots of startups and increased innovation in this area. 

 

 

Likely Outcomes

 

Patient Health as a Service (PHaaS): The new Economic model

The continuous and glaring focus on reducing costs is driving the health care system towards a new economic model that generates revenue by focusing onkeeping the patient healthy and out of the system rather than by getting reimbursed for services rendered. Conceptually this is similar to providing regular maintenance for a car rather than services when it breaks down. The biggest advantage of this new model is that it aligns incentives much better between the health care system and the patient.

Some of the characteristics of this cost-conscious model are the following:

• A focus on prevention rather than episodic services
• Concierge health plans, such as a monthly fee for unlimited primary care
• Increased financial incentives for preventive services, screenings, improved nutrition and regular exercise
• A focus on early detection, increased follow-up, and monitoring of chronic conditions to reduce ER visits and hospital admissions
• Increased cost transparency and consumer choice for medical services
• Remote care: Telemedicine, remote dispensing, and other similar approaches
• Continuous monitoring: Sensor data, devices, smartphone apps
• Pushing services down the provider cost chain: PAs, NPs, increased pharmacy services, offshore support, reliance on automation

 

Automation and AI: The Robot will see you now

Hardware and software automation is still in its infancy within the health care system, but with the increased focus on reducing costs we are likely to see a sweeping change in the future similar to manufacturing automation using industrial robots in the 1980s and 1990s.

Improvements in health care automation technology are already under way in many areas:

• Robot-assisted surgery (especially microsurgery)
• Ingestible sensors, smart pills and bottles
• Cheap, rapid, easily-accessible lab results (like a blood test)
• Electronic, semantic patient data for demographic, clinical and fitness parameters
• AI-assisted diagnosis and measurement

 

The Big Data Revolution: Measure Everything!

New technology improvements have made it possible to collect, aggregate, analyze and visualize staggeringly large-scale data in near-real time at a reasonable cost. These new technologies will revolutionize health care both at a macro and a micro level.

At a macro level, it will enable public health systems and health organizations to manage the overall health of their entire patient population, looking for trends and automatically highlighting potential patient issues for proactive intervention.

At a micro level, it will be possible to continuously track an individual patient’s condition, treatment and progress remotely, and compare those parameters with nominal values against the backdrop of the entire population of similar patients.

The Big Data technologies available now for data storage and algorithmic analysis enable researchers to track and analyze exponentially larger data sets for treatment and research; so it is entirely possible to test a new theory against the entire data set, rather than extrapolate from a statistical sample of the population. This could change the way medical science researches diseases and their cures. Machine learning and predictive analytics based on this data could lead to unexpected discoveries.

Finally, automation and data analytics taken together enable physician scientists to make a systematic, objective evaluation of treatments and medications; AI assistance based on these evaluations can be used to enhance and support the traditional approach of physician experience and intuition, as venture capitalist Vinod Khosla argues in his influential paper: 20-percent doctor included.

 

Patient-centered Care System

Another likely outcome is a shift in focus towards a health care system that proactively reaches out and engages with patients on their terms: wherever they are and whenever they are available, as opposed to the current model where patients engage with health providers at specific locations at specified times.

A patient-centered care system will look very different from the current status quo:

• The health system will proactively reach out to the patient wherever they are, through a variety of devices and technologies
• The emphasis will be on localized services: remote care, home health, telemedicine, local pharmacies, ingestible sensors, monitoring devices
• An increased focus on compliance, post-acute care, and managing chronic conditions
• A holistic integrated approach to patient health: health maintenance, not pay per service, as mentioned earlier
• Continuous monitoring with automatic follow-up, similar to a home alarm system
• Personalized medication (genome- and symptom-specific) and customized delivery through a combination of in-person, mobile and automated services
• A focus on what is “normal” for this patient? For example, diabetes may turn out to be many different diseases, all of which have the same symptom of blood sugar issues

 

Integration: Interoperability and the Care Team

As the concepts of Meaningful Use and Health Information Exchange (HIE) take hold across disparate components of the health care system at all levels, widespread integration will make it possible to provide patient data visibility across the entire care team, including patient caregivers.

This broad-based integration will have a big impact not only on how care is provided, but also on the health organizations themselves - leading to the formation of vertically integrated health care systems or groups (ACOs) comprised of hospitals, emergency rooms, ambulatory clinics, labs, home health providers, pharmacies and even payers. These integrated organizations can not only track all aspects of the patient’s health but can combine that data with provider workflow and financials as well. As an example, coordinated care organizations in Oregon have been transforming the health care system in that state.

With the increased prevalence of deep cost consciousness, integrated care organizations and bundled payments, we are likely to see a broad consolidation of many small players into a few large entities that have the resources to deal with the new financial, technology, data and interoperability requirements.

 

Global Access to Healthcare

This to me is the most exciting development of all. The combination of algorithmic approaches, diagnostic support, telemedicine and low-cost tests means that high-quality health care can be made uniformly available in economically-challenged and isolated locations of the world.

We are already entering a phase where, in the near future, practically everyone will have access to a smartphone; these powerful computing devices are low-cost, easily accessible and nearly ubiquitous. At the same time, Cloud technologies with economies of scale are making computing and storage power progressively cheaper. Taken together, this will make it economically possible to provide prevention and monitoring health technologies to everyone, a true global democratization of health care.

 

The Future is Now

 

All of the effects listed above are interdependent and build on each other, which will lead to dramatic sweeping changes in health care within the next decade. The biggest hope is that it will lead simultaneously to better patient outcomes andreduced health care costs.

This type of industry-wide change is not new. The automation and optimization wave that swept through manufacturing in the late-80s and early-90s revolutionized how physical products are made; similarly ERP systems transformed how organizations collect, manage and interpret data from their business activities.

These types of changes not only impact and transform entire industries and economies, but they change the way we live. The current sweeping changes in health care have the potential to do just that on a global scale.

[Cover photo by Army Medicine / CC BY 2.0]

 

Links: Comparing Yahoo! S4 and Storm

The next frontier in Big Data Analytics beyond Hadoop is Real-time processing. The two open source leaders in this space are: S4 from Yahoo! (now in Apache Incubator) and Storm from Nathan Marz (of BackType and Twitter). 

After some research on this topic, I found only a couple of good articles comparing the two, and a question on Quora. The links are given below.

S4 vs Storm

1. Awesome blog post from Gianmarco de Francisci Morales: http://gdfm.me/2013/01/02/distributed-stream-processing-showdown-s4-vs-storm/
2. Slideshow from Richard McCreadie of the University of Glasgow:  http://demeter.inf.ed.ac.uk/cross/docs/s4vStorm.pdf
3. Comprehensive answer by Divye Kapoor to a Quora question on this topic:  http://www.quora.com/What-would-you-choose-between-Flume-Yahoo-S4-and-Backtype-Twitter-Storm-and-why

 

In addition to S4 and Storm both of which are open-source, there are other commercial products in this space such as InfoChimps Cloud::streams .

Big Data and Graph Databases

One of the more interesting upcoming growth areas in Data Science is the use of Graph Databases and graph-based analytics on large, unstructured datasets.

This is a natural next step along the progression we’ve already been on: first raw MapReduce and Hadoop for large-scale data processing, then tools and frameworks (such as Streaming and Cascading), and then the addition of SQL-like layers on top such as Pig and Hive.

Why do we need Graph Databases?

Today applications and devices generate a flood of data. This high volume of data is typically incredibly dense and highly related; it does not fall neatly into pre-defined schemas. Facebook’s Open Graph and twitter’s interest graph are two obvious examples, but there are many other domains where this applies, such as Healthcare, E-commerce and sensor data.

This type of data of highly-connected entities is not easily modeled using traditional relational schemas; instead, using graph data structures makes it easy to represent connected data and to perform rapid analysis on these large datasets.

In a graph database, data is stored as nodes and relationships; both nodes and relationships have properties. Instead of capturing relationships between entities in a join table as in a Relational Database, a Graph Database captures the relationships themselves and their properties directly within the stored data.

To quote Derrick Harris from his GigaOm article:

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are.

Although graphs have recently become more popular because of their applicability in modeling social networks, graph analysis can be widely applied to analyze any kind of relationships.

Modeling the data as graphs allows data scientists to discover localized patterns; i.e. how are specific items related to other items. Even with large datasets, most analytics queries end up acting locally within a graph. All graphs share common patterns - simple examples include the diamond, butterfly and star patterns, however these simple patterns can be composed into arbitrarily complex patterns.

Graph Database: Neo4J

One of the exciting entries in this area is Neo4J, a Java-based open source graph database from Swedish company Neo Technology. Neo4J stores graph data directly and offers large-scale horizontal scalability using replication; in addition it offers ACID transactions and indexes similar to a traditional database. Neo4J also has a REST API and its own graph query language called Cypher.

A good starting point to learn about Neo4J is Robert Scoble’s video interview with Neo Technology’s CEO, Emil Eifrem. To me, the most interesting section is where Emil nails the value of graph databases in general (at the 6:30 mark in the video):

Sophisticated intelligence and reasoning is all about how are things related to one another. … Whenever the value is in the connection between things, that's when a graph database excels.

You can watch the video here:  http://vimeo.com/56040747

Interestingly, he says that the next step for Neo4J is to work on transparent partitioning, to enable the database to automatically co-locate highly-connected nodes. There is lots of good information about graph databases on the Neo4J site, here: http://www.neo4j.org/learn/neo4j .

As a student of data visualization, I also found the following infographic an interesting way to represent Graph databases:

[If the graphic does not display above, click here to view it on visual.ly .]

Other Technologies

Clearly, this is a hot area right now, with lots of different technologies and frameworks coming up. One offering comes from Intel: GraphBuilder is an open source Java library for constructing graphs out of large datasets for data analytics. YarcData, a Cray company, has a graph analytics hardware platform for real-time analysis: an in-memory appliance called Urika . And of course, there is an Apache project: Apache Giraph that addresses graph processing of big data.

This is still very much an emerging space, so it’s not clear which of these projects will evolve into popular successes. But given the explosion of data all around us and the prevalence of relationships among entities within that data, it’s certain that this area of computer science will receive a lot of attention in the near future.