12 Nov 2015 libre decentralized software

Libre software has been very successful. However, i feel that there is a lot of duplicate efforts and efforts that don't combine into a whole.

The the GNU manifesto, the goal is to provide libre software "like air" to everyone. But lots of functionality that people apparently want, like from Facebook, Twitter, Reddit, could still be filled by libre software.

Decentralized versions of (portions of) the functionality exist, many with libre licenses and way of use; i.e. Zeronet, ipfs, maidsafe, gnunet, Freenet. I think there are too many and to different degrees, they don't interoperate enough. Likely partially because they focus on end-users. I think might be is better to focus on the developers.

My suggestion

My suggestion is to have a (virtual)bus system, and statements across them. Programs attaching to it can put out statements or listen to them.

To set some things into stone, perhaps we can compress a lot of the formats into one; all statements start with a tree, and then optionally come with a (one way)stream. (somewhat like http, except headers are just key-value, not tree)

An obvious initial choice for the format for a tree is JSON.1

The top will have things we define for everything;

protocol Mandatory; Protocol that this message uses. People have to pick *specific* names, or get the definitions accepted by the rest of the community.
data The data is branched down one so nothing in it has to be reserved.
once The statement is to be processed once; only the first through the (below mentioned)filters.
version, description Version and perhaps other similar information. Optional, things with specific names we can move from data to the toplevel if they are similar enough.

The hub program moves around statements. They could move around with, for instance, dbus or ZeroMQ. Filters are pieces of data that specify a function returning yes/no determining what statements get where.

Two kinds of filters can be sent to the hub for getting statements.

  1. Query filters, requesting for data in the DB.

    for the DB to work with performance, having queries in a nice subset would be good.

  2. Subscription filters, requesting any data that comes in and passes through the filter.

For mandatory access control, each program also has in- and out filters. (this actually happens in the hub program)

Furthermore, you might want to delete information you are done with from the DB, hence deletion-filters should be sendable. Here MAC is very important too, so again another mandatory filter is needed.(preventing deletion)

The filters provide a way of user control aswel. Each fixed(mandatory) one could be edited independently. Furthermore the hub has some, For instance, it could have its own running-deletion filter, and enter-into-DB filter.2

once is there to allow things to be processed once. Many programs responding producing more statements, and those statements being responded to, could cause too much resource use. Don't quite see all the sort of considerations like these. It might be useful for the bus system to provide some facilities like these. (alternatively, you make this up to the programs to figure out)

Where do the statements come from?

As you see above, programs can produce statements. This is how it connects to the outer world aswel. I should say explicitly, the statements are mostly for between computers controlled by the same person. Implementations like IRC, Tox, TOR(chat), etcetera can take statements and send them, or receive messages and put them in as statements.

It could include a mount-script that runs and looks for statements on a SD-card on a carrier-pigeon.(though that would have too high latency for many things!) Statements instruct programs to look at bittorrent or ipfs magnet links for more statements. All programs for which the latency/bandwidth is okey should automatically benefit from the new connection method.

And it is not necessarily straight-up just-receive-message; Programs might fish for data based on statements from other programs which are asking for some data. I.e. you ask for all comments from friends and friends-of-friends-etc about some link. Different ways of doing that can be implemented.

Open for developers on all ends

Developers can write programs that are good at one thing;

  • Using any channel of communication that, receives or sends statements, or "fishes for them" based on statements.

  • Programs that receive-send a set of protocols, like chat clients.

  • Adding new protocol, making defining what data means is a matter of not infringing on each' other namespace, and trying to work together defining and implementing something coherent.

  • Programs can also pick up statements and push out new statements based on it. For instance a chatterbot.

Furthermore, configuring and selecting the programs attached, and ability to change/add-to mandatory filters, or using multiple hubs gives the users more power over the system. Though filters on the DB might need to suit the DB for speed, in principle other ones could be programs returning yes/no.

Some ideas

For one, we could make a local server or proxy3 that serves static pages, but also provides Remote Procedure Calls(backs)(RPC) to the pages. That way, people can make applications for it in pure HTML, CSS, JS plus specifying filters.4

Chat is probably a good example. I think it needs a origin, destination, and a message in the most basic form. This in effect replace libpurple, for instance.

Another idea is to also replace mpd, but now any non-music program could use statements to program music, podcast, video, a chore after the next song, or after the next item in a radio program, etcetera.

Yet another is a forum-like thing. Each post would have a target which would be an URL. The post should itself get an URL, thus defining comment threads. It would have a content, optionally a title, origin, tags Could additionally imagine voting(weighted by reputation, for instance) or title-changing votes. It is always annoying when a link is highly-voted, good, but just that misleading title...

Conclusion & how i got here.

For quite a while now i felt things were missing, often regarding the web. Of course i looked at decentralized things like the ones mentioned in the third paragraph, but they weren't quite it.5 I didn't feel like i could see the right track, but I started working on luakit and later a lua Tox client anyway. However, as i came to the point where i wanted write the decentralized-reddit and other things the feeling of being off still was not quite right.

I did have the idea that "channels for flowing data over" was a solution to try not-depend on Tox in particular, and the latest version of tox_comms does this.(only having Tox as an implemented channel, but leaving it open) But i was still missing the last step, making it open on the ends, where people can define protocols and implement things to get such things as decentralized reddit. However, i was annoyed by lua and or web-tech being "special" in it.

Then i came to this idea of using a bus and statements. Really this is not a solution of decentralization, the solution is to generalize where statements are coming from. Decentralized software is software that makes some grander structure out of individual nodes, but it stiffles innovation if those structures are directly tied to applications.

Perhaps i am technically not (entirely)making sense, and the above is pretty much a sketch. Perhaps even, it basically exists, and i didn't see it, in which case, maybe this will help people see it. I hope that in the very least the general idea, that our programs need to be able to communicate more liberally can make it.


  1. To suit json, we need to limit the tree; all branch-keys are sequential-as-a-list or a suitable string.

    Later-on we can take this same idea, but make a binary version. (storebin.. which is "build from parsing unsigneds", or bson? other?)

    Perhaps it would be nice to have them followed by streams of sorts.

  2. This would be superfluous though, since a program with no mandatory filter on deletion can delete anything to clean it up.

  3. Reverse proxies, can we do them better? Maybe we can make the browser program do less.

  4. Suggest AGPL for this sort of thing?

  5. The consensus stuff like Bitcoin, Namecoin, Ethereum and others are a bit in a different category. Though you could imagine statements-injectors for these aswel.

Other Blog Posts