The Software Abstractions Blog

Links: Comparing Yahoo! S4 and Storm for Continuous Stream Processing (aka Real-time Big Data)

The next frontier in Big Data Analytics beyond Hadoop is Real-time processing. The two open source leaders in this space are: S4 from Yahoo! (now in Apache Incubator) and Storm from Nathan Marz (of BackType and Twitter). 

After some research on this topic, I found only a couple of good articles comparing the two, and a question on Quora. The links are given below.

S4 vs Storm

1. Awesome blog post from Gianmarco de Francisci Morales: http://gdfm.me/2013/01/02/distributed-stream-processing-showdown-s4-vs-storm/
2. Slideshow from Richard McCreadie of the University of Glasgow:  http://demeter.inf.ed.ac.uk/cross/docs/s4vStorm.pdf
3. Comprehensive answer by Divye Kapoor to a Quora question on this topic:  http://www.quora.com/What-would-you-choose-between-Flume-Yahoo-S4-and-Backtype-Twitter-Storm-and-why

 

In addition to S4 and Storm both of which are open-source, there are other commercial products in this space such as InfoChimps Cloud::streams .

Big Data and Graph Databases

One of the more interesting upcoming growth areas in Data Science is the use of Graph Databases and graph-based analytics on large, unstructured datasets.

This is a natural next step along the progression we’ve already been on: first raw MapReduce and Hadoop for large-scale data processing, then tools and frameworks (such as Streaming and Cascading), and then the addition of SQL-like layers on top such as Pig and Hive.

Why do we need Graph Databases?

Today applications and devices generate a flood of data. This high volume of data is typically incredibly dense and highly related; it does not fall neatly into pre-defined schemas. Facebook’s Open Graph and twitter’s interest graph are two obvious examples, but there are many other domains where this applies, such as Healthcare, E-commerce and sensor data.

This type of data of highly-connected entities is not easily modeled using traditional relational schemas; instead, using graph data structures makes it easy to represent connected data and to perform rapid analysis on these large datasets.

In a graph database, data is stored as nodes and relationships; both nodes and relationships have properties. Instead of capturing relationships between entities in a join table as in a Relational Database, a Graph Database captures the relationships themselves and their properties directly within the stored data.

To quote Derrick Harris from his GigaOm article:

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are.

Although graphs have recently become more popular because of their applicability in modeling social networks, graph analysis can be widely applied to analyze any kind of relationships.

Modeling the data as graphs allows data scientists to discover localized patterns; i.e. how are specific items related to other items. Even with large datasets, most analytics queries end up acting locally within a graph. All graphs share common patterns - simple examples include the diamond, butterfly and star patterns, however these simple patterns can be composed into arbitrarily complex patterns.

Graph Database: Neo4J

One of the exciting entries in this area is Neo4J, a Java-based open source graph database from Swedish company Neo Technology. Neo4J stores graph data directly and offers large-scale horizontal scalability using replication; in addition it offers ACID transactions and indexes similar to a traditional database. Neo4J also has a REST API and its own graph query language called Cypher.

A good starting point to learn about Neo4J is Robert Scoble’s video interview with Neo Technology’s CEO, Emil Eifrem. To me, the most interesting section is where Emil nails the value of graph databases in general (at the 6:30 mark in the video):

Sophisticated intelligence and reasoning is all about how are things related to one another. … Whenever the value is in the connection between things, that's when a graph database excels.

You can watch the video here:  http://vimeo.com/56040747

Interestingly, he says that the next step for Neo4J is to work on transparent partitioning, to enable the database to automatically co-locate highly-connected nodes. There is lots of good information about graph databases on the Neo4J site, here: http://www.neo4j.org/learn/neo4j .

As a student of data visualization, I also found the following infographic an interesting way to represent Graph databases:

[If the graphic does not display above, click here to view it on visual.ly .]

Other Technologies

Clearly, this is a hot area right now, with lots of different technologies and frameworks coming up. One offering comes from Intel: GraphBuilder is an open source Java library for constructing graphs out of large datasets for data analytics. YarcData, a Cray company, has a graph analytics hardware platform for real-time analysis: an in-memory appliance called Urika . And of course, there is an Apache project: Apache Giraph that addresses graph processing of big data.

This is still very much an emerging space, so it’s not clear which of these projects will evolve into popular successes. But given the explosion of data all around us and the prevalence of relationships among entities within that data, it’s certain that this area of computer science will receive a lot of attention in the near future.

 

Software Development using the SCRUM process

I just recently completed a three-day SCRUM training class, and I have to say that I quite enjoyed it. Normally, the thought of sitting in a chair listening to a corporate trainer for several days in a row fills me with dread, regardless of the quality of cookies and snacks provided; the little participatory exercises feel entirely contrived and artificial. Frankly, I went into the class somewhat skeptical of Scrum ...

But this class turned out to be very high quality and valuable. The presenter, Kenny Rubin of Innolution, a certified Scrum Trainer of the Scrum Alliance, was knowledgeable, informative and interesting; most important, he has enough experience with software development practices and teams to be able to give it a real-world flavor.

That is, the dogma of "the Scrum way" was tempered by the realities of professional software development. (Those of you working on a real-life software development project day to day, or managing software developers, know what I mean - my 20+ years of industry experience have taught me that most software projects are understaffed, have too many requirements, ultra-aggressive deadlines, and repeated requirements changes; it's a wonder we deliver anything at all!)

SDLC: New and Improved with Scrum!

SDLC - the Software Development LifeCycle - is not a new concept. I was first introduced to the concept during my graduate software engineering class at UCSB during the late 80s - by Prof. Richard Kemmerer, he of the red hair and temper to match. (The UCSB of the time was one of the many hotbeds of a radical new movement called the "Free Software Foundation", but that's another story!). The good Professor had lots of industry experience and tried to explain to us the myriad challenges of building and maintaining large-scale software in teams, so very different from the limited computer science coding assignments which we could simply turn in and forget. 

A key exercise in the class was to compare the then-prevalent Waterfall model of software development, with Barry Boehm's new Spiral model; the latter was clearly superior in terms of dealing with the uncertainty of changing requirements. This was in the late 80s; since then, I've seen countless new software processes and programming models come and go: Structured, Functional, Object-oriented, Booch, Rumbaugh, Jacobson, Coad, Yourdon, followed in the 90s by UML, Pair Programming, XP, and then in the 2000s, Lean, Agile, and so on. After seeing so many "great new thing" theories come and go, it's hard to take the flavor du jour too seriously.

After looking at the Scrum process though, I'm optimistic that it is likely to have greater staying power than some of the earlier short-lived fads. Scrum seems to recognize reality a bit more than some of the other recent processes that have generated buzz [e.g. putting two developers on creating a single piece of code when only one will do - is that model going to take over the world? Really?]; Scrum attempts to include this reality into the flow rather than trying to bend it to an artificial process.

Another advantage is that Scrum tries to take out-sourcing and off-shoring into account - relatively recent trends that are likely to continue to grow. Finally, Scrum borrows heavily from the models and processes that came before: not just Lean and Agile, but even earlier. The iterative sprints approach of Scrum is conceptually remarkably similar to Boehm's spiral model, isn't it?

At the same time, there are some glaring deficiencies in the Scrum model that will need to be addressed. 

Scrum Process: Key Points

Having been through the official Scrum training, I will highlight the key points of the Scrum process in this section.

The overall Scrum process definition is broken into Activities, Roles and Artifacts. (A canonical place for definitions is Mike Kohn's Mountain Goat Software web site: Introduction to Scrum. )

Sprint Activities

Scrum is an agile process that relies heavily on specific, prescribed activities. Since development is organized as a series of sprints, scrum specifies a set of activities that are repeated for each one. 

Sprint Planning Meeting: Figuring out which of the high-priority product backlog items can fit into the next sprint.

Daily Scrum Meeting: A quick daily meeting during which team members share their activities for the past day, planned activities for that day, and identify any impediments to progress.

Sprint Review: Conducted at the end of a sprint, this meeting provides an opportunity to demonstrate the functionality added during the sprint, displaying it to product owners, users and stakeholders.

Sprint Retrospective: The whole team participates in this meeting, which is an opportunity to reflect on a sprint that is ending and opportunities to improve for the next sprint.

Overview Image: (c) Mountain Goat software

Roles

Different members of the team are assigned specific roles in a Scrum process; these roles prescribe specific behaviors and tasks for the team members.

Scrum Master: The Scrum master is the team's coach, and is essentially a facilitator rather than a project manager or line manager. The main role of the scrum master is to remove project impediments that affect the team, and provide coaching about the process, enabling the individual sprints and meetings to go smoothly.

Product Owner: The product owner comes up with a vision for the product, and translates it into a prioritized set of features required in the product (the product backlog, in Scrum parlance). The product owner answers questions about the requirements and helps the team make trade-offs between different options.

The team: The team as a whole is the central player in the Scrum process. Each person contributes in whichever way they can to complete the work of each sprint. 

Artifacts

Scrum defines various artifacts that are used at different stages in the process.

Product Backlog: The product backlog is a prioritized list of "user stories" that, taken together, define the functionality that remains to be added to the product.

Sprint Backlog: The sprint backlog is much more limited in scope than the product backlog; it's a list of tasks the team needs to perform in order to deliver the functionality committed to for that sprint.

Release Burndown Chart: This chart shows the amount of work remaining in the release. Similarly, a sprint burndown chart shows the amount of work remaining in the sprint. These charts provide a visual indication of progress towards completion of the project.

Conclusion

One of the key takeaways for Scrum, is the focus on a unified, cross-functional, empowered Scrum Team. This team includes software developers, testers and web designers, as well as the product owner and scrum master; in short, everyone required to specify, build and test the product. And this team is both empowered to make decisions with regard to scope, schedule, features and resources, as well as held responsible for the results. An interesting concept!

And what of those shortcomings of Scrum that I alluded to earlier? Those will follow in a future post. Stay tuned ...

[Note: some of the material for the training class referenced above was licensed from Mike Cohn of Moutain Goat Software.]

Crowdsourcing Charity: Sustainable Innovations

Paul Buchheit (founder of Gmail and Friendfeed) wants to donate money to charity. He's trying an interesting experiment in collaborative charity, where he's planning to crowdsource the list of ideas he should consider funding. The ideas he gets are only recommendations, he says, and he will ultimately choose the best idea that appeals to him and that makes sense.

In that spirit, let me make a suggestion for Paul and others like him in Silicon Valley, one that takes a different perspective. If you're a successful tech founder or VC who likes philanthropy, consider this a free "business idea for giving back" that you can adopt as your own if it interests you. [Now I'm no VC, nor am I a rich, successful tech founder, so I have no idea if the approach below has any appeal to these folks; nevertheless it is an interesting thought exercise.]

Silicon Valley skills

In general, anyone with a bit of time or money and the inclination to help, can donate to charity, and many people do (bless 'em!). The most effective donations, however, leverage the unique skills and perspectives that one can bring as an individual, using one's own talents and gifts in addition to money, to help others.

So what could Silicon Valley entrepreneurs do? How about this: instead of donating money directly, why not use it as seed funding to build an organization that can keep on giving? After all that is what tech founders (and VCs) do best: they fund startups that take some initial amount of money and then leverage it massively to create a disproportionate impact on a target market - or die trying. Why not take the same approach to charity? Build an organization that leverages other people's time and money, and creates massive impact, much bigger than the pure monetary donation would create.

But what type of organization? There are already many, many charities, of many different types. How do you make a difference?

Third level of Abstraction

Give a man a fish, it is said, and you feed him for a day; teach him to catch fish, and you feed him for a lifetime. Extending the metaphor, what if you could create a construct that continuously spawns ideas, technologies and approaches for teaching everyone to catch fish?

What the world needs more than direct charity, is locally-sustainable ideas that allow populations in underdeveloped areas to take care of their own core needs - clean water, sustainable food production, cheap education and economic development - by themselves. As a philanthropist, what if you were to establish an organization to crowdsource innovations and ideas to bring about these very changes? You could create a mechanism to encourage such attempts and to reward successes.

The idea: So the idea I'm proposing, in a nutshell, is this: instead of making a direct monetary donation to charity, set up a recurring contest or start an organization that encourages and rewards innovations in technology, design, reuse, financial models, human factors or ecology conservation, for the social good. Most important, these innovations must lead to local, self-sustainable solutions that do not depend on ongoing external help to keep going.

Real-Life Examples: Business Models and Innovations

There are many examples of business models for encouraging innovation through crowd-sourcing:

• Competitions, such as the Netflix prize for improving their recommendation algorithm (which seems to be beaten) or NYC's design competition for post-disaster provisional housing
• Paul Graham's Y-Combinator micro-funding model for seeding startups
• Social incubators like socialedge

I'm sure there are many others that are not listed here, feel free to list them in the comments if you know of any.

What kind of innovations should such an organization look for? Ideally, these would be locally-managed, self-sustainable or reuse-based product design ideas, original approaches or creative financial models.

Here are some examples of innovations of this type:

• Clean water powered by playgrounds: Playpumps International
• SHRIMP Refugee Housing, which folds up so small that four of these can fit into a standard shipping container, and each one provides shelter and fresh water for a family of 4. [A slide show from Wired Magazine has a whole list of innovative shelter designs for providing emergency housing to disaster areas and refugees.]
• Buy a pair of chickens for AUS $10 and help a family get started in sustainable chicken farming. [via Darren Rowse of Problogger]
• Lighting up remote villages in India, using technology and innovative financial models: SELCO [via Sramana Mitra]

The world needs many more innovations of this type. Why not explicitly encourage them?

---

Why You Should Market Your Brand on twitter - NOW!

twitter has been an emerging phenomenon for a little while now. There were some very public performance problems and outages in the beginning, but the company seems to have moved past those now. twitter usage seems to be gaining critical mass, crossing over from the technology early-adopters to non-technology leaders, companies and celebrities (imagine Stephen Colbert and Jon Stewart of Comedy Central bantering about twitter on Election night).

There has been much discussion and debate recently about whether it is appropriate and worthwhile for businesses to spend time marketing their brands on twitter. In this article, we take an in-depth look at the advantages of doing so.

Given the ongoing dramatic growth of twitter usage, and if twitter adoption has indeed crossed over to the mainstream, then it seems inevitable that brand marketing will eventually catch up with it on a large-scale; adding twitter marketing to company budgets will become an increasingly easier decision once "everyone is doing it". However there are compelling reasons why you should consider starting your twitter marketing efforts right now rather than wait until there is a herd.

So if you need to convince your boss to let you spend time on this activity - promoting your brand on twitter - here is a list of reasons to get into twitter-marketing now:

Early mover Advantage

Years ago Seth Godin's book, Permission Marketing, launched a revolution in brand-customer interaction. Imagine if you had gotten into Email Marketing at that time, when it was first getting started - it would have been so much easier to get and hold consumer attention, make your brand stand out, and establish a solid position. In contrast, getting the same level of attention today using email is almost impossible, since practically every company practices email marketing; consumers face email fatigue and get deluged by spam, and are reluctant to give out their real email addresses to companies.

For twitter marketing it's not too late, however - it is still in its early stages, and this time you do have a chance to jump on it before most of your competitors (as @Southwest Airlines, @Whole Foods, @Baskin Robbins and @Zappos have already done), gaining momentum as it becomes mainstream.

Influencing the Influencers

A common argument against the twitter marketing strategy is that many of the users of your non-technical product may not yet be twitter users, and you should always market where your users are. This seems like common sense. However, it overlooks a fundamental truth: users often tune out the direct messaging coming from a company anyway, and instead use indirect sources to help them make decisions.

In Malcolm Gladwell's influential book, The Tipping Point, he describes how word-of-mouth influences brand popularity, trends and buying decisions. If you can get some of the super-connectors and influencers to adopt and evangelize your product, then interest in your product spreads like wildfire and demand soars.

How do you reach these influencers? In today's world, it's a safe bet that the influencers are heavy users of new media methods like blogs, social networks and twitter, even if your field of interest is not particularly technical. twitter is especially effective in encouraging and responding to these highly-connected users; the medium gives you direct, instant access to even the most popular of these folks at all levels. (For example, here are the twitter accounts for publishing guru Tim O'Reilly: @timoreilly, VC Fred Wilson: @fredwilson, VoIP pioneer Jeff Pulver: @jeffpulver, and even celebrities like Shaquille O'Neal: @the_Real_Shaq and Lance Armstrong: @lancearmstrong ). By connecting with them and interacting directly, you have a much better chance of getting mentioned on blogs, in forums and discussions, and being highlighted in their media outreach.

Customer Engagement

More than any other medium, twitter allows you to connect directly with your users and engage them in a direct, immediate, two-way information exchange. You can get timely feedback: users with negative comments can be heard and their concerns addressed; users with positive comments can be encouraged to become evangelists. You can even learn of alternative ways your product is being used, or discover new markets that can be targeted.

This strategy of connecting with individual users and building a "long tail" following nicely complements the previous strategy of targeting "influencers". Some studies contradict Gladwell's conclusions, showing instead that users trust recommendations from peers and friends far more than those of a few super-connected individuals; the safest bet is to court both types.

Low Cost

In general, twitter marketing is very cheap and easy, and shouldn't have any significant impact on spending for your other Marketing programs; important, since cost is a major concern in the current challenging economic climate. Success in connecting via twitter takes time and authenticity, not a large budget!

And if your company still doesn't buy it? Create a personal account on twitter (hey, it's free) and start connecting! You can begin engaging others in your field and thought leaders in other fields - by engaging at a personal level, you can start to build your own network of followers.

Oh, and if you'd like to connect with me, you can find me as @nitink on twitter - I always follow everyone back!

For further reading, here are some great articles on twitter Marketing:

Great Examples of Corporate twitter use - Jennifer Laycock
Using Twitter for Brands or Corporate Identities - Dawn Foster
How to use Twitter as a Twool - Guy Kawasaki
Looking for Mr. Goodtweet - How to pick up followers on twitter - Guy Kawasaki
Free (but welcome) advice on the corporate use of twitter - Elliot Ng
A User's Guide to Twittering - Wall Street Journal
Professional Branding Comes to Twitter - Svetlana Gladkova
Why I Love Twitter - Tim O'Reilly

---

Should you invest in New Media Marketing in a down economy?

Over the past couple of years, companies and brands have been increasingly engaging with New Media, which can be broadly defined as blogs, feeds, wikis, micro-blogging (twitter), social networks (such as LinkedIn and Facebook) and social bookmarks.

As the economy goes sour and budgets decrease, however, does it make sense to continue to invest Marketing dollars into these word-of-mouth, engagement programs?

Here are some of my initial thoughts.

As with everything else, it comes down to a cost-benefit analysis, comparing these types of programs with alternatives such as direct marketing, trade shows, magazine advertisements and so on.

Benefits

On the benefits side, some of the core value propositions of New Media Marketing are pretty strong:

Increased Brand Awareness
Research has shown that consumers most often buy brands that they are already familiar with; generating brand awareness and familiarity is thus as important as pure direct conversion rates.

Engaging with Customers and others in the Ecosystem
The first simple step to irrelevance for any brand is when a company stops listening to those who are most affected by its products. By engaging with customers, users, suppliers and others, a business can get direct, rapid feedback and potentially head off problems before they turn into major crises.

Getting Product Ideas
Discussions about your product and its use in social media are a fertile environment to get new product ideas, especially involving novel settings or uses of your existing products or discovering gaps that can be filled.

Lead Generation
In the same way, social media can be mined to harvest meaningful, highly-qualified leads, significantly reducing the cost of customer acquisition.

Find, Encourage, Enable and Reward Advocates
Who are the users who really like your products and are most vocal about it? How can you provide encouragement and rewards and remove barriers? By engaging directly with users, you can find the answers to these questions.

Participate in the Conversation
If your product is at all popular, you can be certain that customers and users are already talking about it somewhere, somehow - whether you know it or not. If you can participate directly in this conversation, you can at least react to what they are saying, even if you cannot and should not directly steer the conversation.

Monitor => Engage => Influence => Monitor
This is the virtuous cycle New Media enables. By its very nature, social media is interactive; it is a dialogue, rather than the outward-bound one-way Marketing messages of the past.

Costs

By its very nature, social media marketing is low-cost compared with traditional approaches. What it takes to be effective in this field is time (which, of course, also translates to cost) and authenticity.

The Marketing Mix

So is it worth investing in this type of user engagement? Obviously, the decision depends on your particular products and your environment, but increasingly in the future, the answer to that question is going to be another question:
Can you afford not to be a part of the conversation, especially if your competitors already are?

---

Terrorist Attacks in Mumbai, India

As I write this, the terror situation continues in Mumbai, India. The Times of India first reported the initial news sometime around 10 am this morning, which means it has already been going on for over twelve hours. Although Mumbai has suffered from other significant terrorist actions in the past few years, this is the most violent and long-running attack that I can remember. According to the latest reports, at least 101 people have been killed and hundreds more have been injured. The beautiful Taj hotel building has flames shooting through its top floors and terrorists are still holed up in the Oberoi Trident hotel.

One tragic outcome is that the Mumbai police have been hard hit by this attack, having lost three of their top people: ATS Chief Hemant Karkare, Encounter Specialist Vijay Salaskar and ACP Ashok Kamte, have been killed. Our thoughts and prayers are with their families.

Atypically for terrorist activities in this region, this particular attack seems to have specifically targeted foreigners. That, and a high level of coordination across multiple targets, points to a new level of sophistication for the local terrorists.

The best way to monitor the situation from afar, I've found, is via twitter. The catchall twitter tag #mumbai (thanks to @fredwilson for pointing this out) is currently getting multiple updates every minute. Although CNN is much more visual and includes more analysis, it's hard to beat twitter for real-time updates of breaking news. Gaurav Mishra provides some great commentary on the role of real time citizen journalism in covering these attacks (as does TechCrunch).

"The purpose of terrorism," Vladimir Lenin once said, "is to terrorize." Perhaps, but I think that these terrorists have underestimated the resilience of spirit of the people of Mumbai. In the same vein as New Yorkers, the Mumbaikars will survive this brutal act of terrorism and eventually emerge the stronger for it. Terrorism has become a scourge worldwide, but in the end, it cannot succeed; by simply going on with their daily lives, the regular folks - unsung heroes all - will eventually prevail.

---

"Doctor, I read it on Google"

As I observed earlier when writing about Emerging Trends in Healthcare, the relative knowledge levels are changing rapidly in the physician-patient relationship. In the past, the physician was clearly the expert and the patient simply accepted the doctor's judgement and counsel without question.
 
Today's web-savvy Healthcare consumers, however, are adept at finding comprehensive and detailed information about their own condition and disease. In many ways, in fact, they have an advantage over the doctor - a patient only has to learn about one particular condition (or a few); whereas a physician, even a specialist, must necessarily cover a far wider range of medical knowledge.

By and large, most doctors are uncomfortable with this sudden shift in patient knowledge level and the potential for second-guessing the doctor's opinion, that goes with it. New research from Microsoft suggests that the doctors may be right!

Microsoft Research has just published an article on this topic: Cyberchondria: Studies of the Escalation of Medical Concerns in Web Search. The study authors found that patients who perform Web Searches about their symptoms, tend to have an exaggerated sense of their illness. Says the article:

However, the Web has the potential to increase the anxieties of people who have little or no medical training, especially when Web search is employed as a diagnostic procedure.
...
We show that escalation is influenced by the amount and distribution of medical content viewed by users,

Overall, though, this finding is hardly surprising. As any layperson who has ever perused a medical encyclopedia knows, just reading about the specific symptoms for any random disease makes one start to start imagine those very same symptoms - it is part of the human condition! Obviously the same thing happens when patients read medical information online.

At the same time, there is a distinct upside for patient health in having all of this medical information easily available and accessible to all. Informed patients are in a better position to understand what is happening to them; they can handle it better and can take better care of themselves in addressing their own health problems.

In any case, this is a one-way street. The genie is out of the bottle now and patients will be increasingly better-informed in the future. Whether this is preferable or not, physicians had better gear up to deal with this change.

---

Shared Searching from Microsoft - a new form of Social Search

The New York Times reports that Microsoft is developing a tool to enable users to collaborate in real-time on web search. This free IE7 plug-in allows users at different computers to divide up a search task among themselves, and then pool their results and recommendations in a shared space displayed on their browser.

 

Meredith Ringel Morris, a computer scientist at Microsoft Research in Redmond, Wash., has created one of these collaborative tools, SearchTogether, now available in a test version as a free download at http://research.microsoft.com/searchtogether. The program is designed to work within the Internet Explorer 7 browser.

 

“Web search is usually considered a solitary activity,” Dr. Morris said. “But many tasks can benefit from joint searching.”
...
The system also has a “peek and follow” feature that lets a group member watch another member search.

This is a fascinating idea that hasn't been tried before. Although search is commonly an individual activity, there are many scenarios when being able to directly share the search activity and discuss the results is invaluable; e.g. in a business context, geographically distributed team members could collaborate on project tasks that include internet search activities; similarly, in a family context, you could remotely assist less-computer-savvy adults or children in completing tasks that include web search.

Coincidentally, earlier this week Google also launched their own take on social search. The Google SearchWiki enables you to customize search by re-ranking, deleting, adding, and commenting on search results. These results only affect your own searches, but you can share your annotations and changes with others; although as Marshall Kirkpatrick notes on the ReadWriteWeb, viewing these annotations leads to a poor user experience. No doubt that will change soon; the area of social search is very important to Google, as Marissa Mayer noted in a VentureBeat interview in January of this year.

Of course, there are many other social search engines - Delver, OneRiot, Scour, Tusavvy, Yoono, et al (I still remember Lijit, but it seems to have turned into a publisher application); Mashable has a has a list .

---

"Connecting With Your Intimate Bot" - by Sramana Mitra

I've just started reading Sramana Mitra's new book, Entrepreneur Journeys. More thoughts on this to follow, but in the meantime, here is an excerpt.

---

Connecting With Your Intimate Bot

We've all had this thought: When will technology slow down enough for me to catch up? Just managed to get a grip on Web 2.0 (or did I?), and already there are rumblings about Web 3.0.

But the Web's evolution slows for no one. Here's how to get ahead of the next wave.

There are numerous definitions of Web 3.0 floating around. Tim Berners-Lee, a father of the World Wide Web, talks about the "Semantic Web," a way that computers employ the meaning of words – not just pattern matching – along with logical rules to connect independent nuggets of data and so create more context for information. The formula that makes the most sense to me is this: Web 3.0 results from combining content, commerce, community and context, with personalization and vertical search. Or, to put it in a handy phrase: Web 3.0 = (4C + P + VS).

Here's what it means.

Web 1.0 was all about driving online commerce and trying to find "anything" in the tangled jungle of the Web. It produced companies like Yahoo!, Amazon.com, eBay, Netflix and Blue Nile. The rush for dollars also resulted in the dot-com meltdown. Even so, people's habits of searching, buying and selling genuinely changed.

Web 2.0 has been a relatively niche phenomenon, with hundreds and thousands of tiny companies primarily focused on social networking through online communities. MySpace, Facebook and Digg, have been the most notable companies to emerge. But there are a plethora of others where you can "meet," "connect" and "make friends" online these days – habits no longer considered weird.

At the same time, we've seen a great deal of investment in vertical search companies. If you are looking for a job, you can go to a site like Indeed.com and search across various job portals and career sites. Or go to Kayak if you've got travel questions, or TheFind if you're seeking shopping advice. In each case, the sites have carefully customized search parameters (job seekers, for instance, can search on salary ranges, locations, job levels and so on). Therein lies the big difference with Google, a generic horizontal search engine.

Finally, Web 2.0 has brought an onslaught of user-generated content in the form of blogs, podcasts, appending comments at the bottom of articles, posting reviews of restaurants, movies, stores, and hotels. Media has become truly interactive, as opposed to the one-way world we were used to. Many more voices are being raised, and heard. The media industry, as we have known it, has been shaken to its roots.

The next wave – Web 3.0 – will organize itself around two different elements: context and the user.

By "context," I mean the intent that brings you to the Web, your reason for surfing. Looking for a job is "context," as is planning a trip or shopping for clothes. Fundamental to context is the user. And when you fuse a specific user with genuine context, you wind up with truly personalized service.

Imagine this: You are planning a trip to Rome. You are looking for a hotel around Piazza di Spagna, but not something large and impersonal – which rules out the Hassler Villa Medici. You like smaller bed-and-breakfasts, with charm, warmth, character. You want an online travel agent who can understand your needs and preferences, and find you not only the right hotel but really interesting restaurants, boutiques and shows all aligned with your taste. Normally, you use Guide du Routard as your travel guide, but today there is still a gulf between travel guides and online travel-booking sites – in other words, content and commerce are fragmented. In Web 3.0, you will see content and commerce finally come together in a big way, no longer forcing you to hop from site to site to get one job done.

On this same trip, you would love to meet local people who share your interests – say, cooking, jazz, opera. In Web 3.0, you will see the community elements of Web 2.0 pulled into context, making it as easy to find new friends with common interests, even in a distant city, as it is to book a hotel room.

Some user-generated content is already evolving into an integral part of travel planning today. At TripAdvisor, for instance, travelers report back on their experiences at hotels around the world. The missing element, however, is the notion of the individual user and his or her personal needs. You don't want to read reviews from anyone. You want to read reviews by people whose taste and judgment you trust.

In a Web 3.0 world, then, a personalized travel agent will help you find and book a highly customized itinerary, leveraging all the power of previous generations of Web technology –searching (both generic and vertical), community building, content and commerce. That's how I get Web 3.0 = (4C+P+VS) – the sum of content, commerce, community and context, with personalization and vertical search.

This is complex technology, requiring sophisticated artificial-intelligence algorithms. After all, your Web 3.0 travel agent will not be a "person" but a "bot", or intelligent agent.

But I suspect you will like your travel bot. And your career bot. And your shopping bot.

So be patient with the technology entrepreneurs around the world, who are working through these generations of the evolving Web, trying to bring about a dramatically better user experience. After all, they - and their bots - are working for you.
 :)

                      Entrepreneur Journeys
                           

---