I ran into this problem recently of wanting a truly distributed method for gathering loosely structured input from everyone in an open and community organized way. How do you best go about gathering lots of opinions and getting some forms of consensus online in a way that software can make sense of, and doesn't rely on single points of authority? That may sound difficult, but it really is just the essence of the web itself, we're surrounded by it.

The first and obvious instinct is to "just use a wiki" and I was heading down that path, but I started having concerns that a single group mindset may form around that wiki, or that the administration itself could play too important of a role. Not only were there these long term organizational concerns, but scale was also troubling me, I don't want to be dealing with one monster database and a huge volume of changes.

The next thought, a much more distributed and decentralized approach, would be to use microformats. This would allow any page anywhere to openly contribute and a bot to do some type of automation and together form their own kind of consensus. I usually love everything about this model, but I also really like the instant gratification of having a wiki, and sometimes microformats tend to focus on individual contributions whereas wikis foster communities.

The answer I arrived at was really quite simple, combine the two, a wikimat :) Just use some simple recognizable keywords that people can plug into any wiki anywhere on the web. Based on that I've come up with these guidelines for a wikimat, which is just a wiki page with some common patterns like:

- First, any wikimat page must be easily identifiable, for instance by putting a unique tag or code word in the title/name/url of the page.
- Second, within the wiki's human editable area there should be another unique word to know where software can start looking.
- Third, the markup used by the wikimat (if any) should be as minimal and compatible across all wikis as possible, bold, lists, breaks, etc.
- Finally, any wiki that gets used to support a wikimat should be listed or identified in some common area for that wikimat (to ease discovery).

It may not make sense what I'm talking about yet, so I'll use an example (perhaps a little US-centric), renewable energy and communities: Lots of individuals and growing numbers of communities are experimenting with renewable energy, but there is very little coordination or structured exchange of success/failures/lessons/etc. A simple renewable-energy wikimat might define keywords for the common types, wind, solar, geo, bio, etc. Then also a bunch of important attribute keywords like funding, start date, status, homepage, people served, and so on. After there's some basic dictionary of these code words, and one important primary unique keyword to identify the wikimat itself (say "Renewmat"), any community can use any existing wiki and start sharing a semi structured experience with the world.

In the example above, a simple renewable energy project wikimat page might be:

http://wiki.example.org/Renewmat:Cedar_Rapids,_IA,_US

Name: Indian Creek Nature Center
Date: Saturday, October 7, 2006
Community: Cedar Rapids, IA, US
Org: Iowa Renewable Energy Association
Type: PV
Description: The Nature Center's buildings are a model of efficiency and use a photovoltaic system to produce 40% of their electricity.
...

Then either someone has listed wiki.example.org as one of the sites that has Renewmats to some central place, or web search/bot can discover these automatically. Then third party tools can collect, aggregate, and process the semi-structured information.

Of course, I have a specific wikimat I will be proposing (to the grub-dev list) soon relating to grub and crawling :)

Sorry for the lack of posts here, I've been busy :)

Here's the summary of Atlas, join the mailing list to partake in the discussion and help evolve the idea:

Atlas - Internet Search Infrastructure

This is a brief overview of a large vision: enabling search to become a part of the Internet's infrastructure. Building on Atlas as an open protocol, search can become a fully distributed and interoperable world-wide community. All of the participants can interact openly and in any role where they believe they can add value to the network.

A search engine can be constructed from many entities serving different roles instead of one monolithic system. These entities are exchanging aggregate information, or knowledge, and can decide with whom they want to work with. To design this working economy based on knowledge, there must be balance between these various entities. Each actor must have incentive to act both for their own benefit and for the benefit of the whole, and enough information to make and validate those decisions. Reputations and relationships are the essential fabric of Atlas, just as they are in a real-world free market.

There are three primary roles within Atlas:

Factory - Responsible to the content.
Collector - Responsible to the keyword.
Broker - Responsible to the searcher.

Each of these actors must interact with the others to complete any search request. Any two roles could be performed by a single entity (whereas if all three are performed by one entity, the result would be a traditional, monolithic search engine).

A Factory is akin to a crawler in today's search engines. An Atlas Factory must fetch and process the content as intelligently as possible, performing analysis (such as Natural Language Processing) and normalizing it into distinct units. A Factory shares its highly refined and processed output with one or more Collectors based on who they believe is best utilizing it.

A Collector absorbs and indexes output from one or more Factories, with one primary goal: ranking. An Atlas Collector must provide the most intelligent ranking and relationship analysis possible. A Collector has to compete for the output of a Factory, as well as compete to provide the best ranking quality for Brokers.

A Broker must provide a searcher with the best possible results. It does so by combining diverse ranking results from Collectors and also by retrieving content from the original Factories. This last step, a Broker interacting with a Factory, is critical to maintaining a balanced ecosystem. All Factories must be aware of and approve how their results are being used and by whom.

Reputation and reward is bi-directional between all parties (Factory-Collector, Collector-Broker, and Broker-Factory). Each entity may choose to interact on principle (free, Commons), attribution (results provided by), or commercially (as a paid service), the Atlas protocol is purely a facilitator and does not restrict how the relationships between any entities are formed. In considering these motives for the various entities, it's likely that the free-based networks will tend to become more specialized, commercial ones will compete on quality, and attribution based networks will mature in both directions.

This simple yet powerful division of roles, responsibilities, and relationships will result in a distributed economic foundation for an Internet Search Infrastructure. The wire protocol and further definition of the interactions between these entities is openly evolving, anyone interested is welcomed to join the discussions and see the initial proposals at http://lists.wikia.com/mailman/listinfo/atlas-l over the coming weeks.

Thanks, looking forward to a radically different search ecosystem in the coming years :)

Meaning, the process of converting Information into Knowledge. To give meaning to information, is to make it useful, to have context, to enable understanding, to empower. Information simply exists, a commodity, dimensionless. When information has meaning it can become knowledge, and that is perhaps the most important process humankind has ever practiced, to learn.

Why is it then that our current most modern Meaning Economy is a text box dictatorship? Why in such an advanced civilization have we become Knowledge Peasants whom are so easily placated by the black magic of our Goovernor? Am I the only one wondering why these commercial boxes own such an important social function: what everything means?

We're safe because it's a free world marketplace on the Net, and anyone can compete if something goes wrong, right? Not quite, 'compete' itself tells you why, the competition will just be another commercial box, how else do you pay for all those servers and bandwidth it takes? I'm glad you asked!

Open open open! Open source, open distributed grids, open algorithms, open rankings, open networks of people cooperating to provide resources. The future of search is in open cooperation (and competition) based on a Meaning Economy, create meaning, exchange meaning, serve meaning.

My vision begins with an open protocol, allowing independent networks of search functions (crawling, indexing, ranking, serving, etc) to peer and interop. All relationships between these networks are always fully transparent and openly published. Networks exchange knowledge between them, each adding new meaning to the information, each of them responsible for the reputations of their participants and peers. This is the very foundation of a Meaning Economy.

Tomorrow now has a meaning that we can all help build.

Doing some blog software updating, new theme, and in the process I wanted to make special note of the now-missing turtle:



The picture was an important depiction of how I used this space, begrudgingly and out of my element. I don't expect that to change, I may blog (a little), but I'm hardly a blogger.

I'm getting indigestion reading all of these complaints about Internet Radio dying. My acid reflux isn't due to the fee hikes, it's the short-sightedness of everyone bitching about them.

Folks, we're ON THE INTERNET here. What happens when a network goes down? You route around it. If the big music licensing conglomerates can't understand the market, route around them. This could be the best thing to ever happen to Internet Radio, and music in general, a new marketplace and new systems will form in the wake of the dying beast(s). A fallen tree may support more life than a living one.

Independent and forward-thinking artists have an enormous opportunity now to get in front of listeners. Online stations have a chance to become next generation leaders, to provide rich new music, and most importantly to engage the listeners with them in their battle to find new sources of content.

Please, people, stand up and do the right thing, MOVE ON.