[PP-main] Technical/implementation matters

[Joakim Ziegler]

On Fri, Mar 03, 2000 at 11:02:59AM +0930, Nick Lothian wrote:

>>> * How do we exchange them?
>>> There are two or three basic ways to do it. First of all, should it be
>>> centralized or not? That is, should there be a single point all the data
>>> passes through? I think so. This is dependant on timely delivery, so a 100%
>>> distributed system would be too slow, and would also consume a lot of
>>> excess bandwidth. The other question is, do you poll for new items once in
>>> a while, or do you open a connection and get all the stuff pushed to
>>> you as it happens? Or maybe you do a combination. I don't really have an
>>> opinion about this, input would be appreciated, as usual.

>> I think that ultimately, it needs to be centralized to some extent. This
>> doesn't mean there can't be failover and redundancy built into the
>> architecture, and IMO, there should be.

> I think it is fairly important to separate the XML/HTTP based protocol for
> publishing stories, and the "value added" type stuff for the trust matrix,
> rating of stories (dare I say moderation?)

Not that we necessarily use HTTP, though (see my other mail). A more rich
protocol could carry all the data over the same link, and thus be a lot more
unified and easy to handle.


> I quite like the way RDF works - site publish the files, and sites that are
> interested in them retrieve them.

I honestly think we need something more centralized. One of the reasons RDF
and friends-based syndication hasn't taken off more, is that there's no way
to get an overview of what's out there, you need to know about it to get to
it.


> I could see the value of a central site to register sites that publish
> stories, but I feel that it should work the way something like UserLand
> does - sites ask it to check their site every 30 minutes or so, and then if
> there is a change, it publishes the stories.

Well, obviously, this is more or less how it will work. Although if we want
to get in and compete with other, commercial wire services, 30 minutes is not
at all often enough. You'd have to poll once every minute, at a minimum, and
at this point, provider-push-to-central-server becomes a much more feasible
method (not to mention that push when you have something is much more elegant
and clean than poll at intervals, as a model).

Time really is of the essence here. I spent a year working in an IS
department that provided services directly to a wire bureau (a Norwegian
financial wire service), and the time limits there are daunting. We're
talking maximum 5 minutes from the news comes in, for the journalist to write
up the piece and get it on the wire. I'd like to have a model that can at
least provide this sort of speed, even though the vast majority of our stuff
won't require it.

This is why I want the providers to push items. I'm actually toying with the
idea of giving consumer sites the option of having the central server push
items to them, if they want them. This would be as close to zero-latency as
we can get, and I don't think it's significantly heavier on resources.


> The advantage of this is that it is fully open - if another person wants to
> start up a site that competes with UserLand, they can, and it can do what
> UserLand does without any changes on the part of all the RDF publishing site
> owners.

> If we have a client push, central site based architecture, it basically
> locks power into that central site. Imagine if, in a couple of years, say,
> PeerPressCentral.org goes the way of Slashdot - it begins to get trolled
> (somehow), and moderation is introduced (or whatever - just for the sake of
> argument). Now no one can come and produce their own, competing site,
> because it has to take the feed from PeerPressCentral, unless they can
> persuade all the sites in the PeerPress network to push their news directly
> to them, too.

This is true. However, I really trust the community to take care of these
issues. If Peer Press does something really stupid, the community will
respond in kind. I'll be the first to make sure that the tools to set up a
competing site are readily available.

Also, I think those problems are the ones we're aiming to solve with our
approach, so the only possible problems I could see are the "administrator
gone evil" scenarios, and I trust the community to take care of those.

-- 
Joakim Ziegler - simplemente r&d director - joakim at simplemente.net
 FIX sysop - free software coder - FIDEL & Conglomerate developer
      http://www.avmaria.com/ - http://www.simplemente.net/