aka: HOME -   - STRATEGY - project management - technology - design - tools - Blue Sky Radio - klogs - community - staffing - shortage watch -   - LIFE - events - food - Bloggers for Hire - shrub - public policy - books - Obituaries a la Blog
Click to see the XML version of this web page.
a klog apart
Phil Wolff's subversions...


Sunday, July 13, 2003 Go to this day's page

community   design   klogs   strategy   technology  


The syndicated blogosphere will reach 300 million feeds in 3 years. 

This is very early in the adoption of RSS feeds. Very few publishers. Even fewer readers. How will this change?

Assume growth.

In two years:

  1. Every blogger will publish a main feed.
  2. Each blog's category or topic will have a mirror feed.
  3. Every business system requiring a user ID will customize feeds for each user.
  4. Every major media outlet will drive traffic and affilliation by publishing feeds.  
  5. Some consumers will add editorial value by blending existing feeds into new, focused feeds.

I assume AOL, Microsoft, Yahoo!, and Terra will turn on blogging tools in the next 18 months, and 10% of the online community (70 million people) become bloggers.

So, many feeds.

Feed payloads will grow 100 to 10,000 times.

TiVo users often record more programming than they can possibly watch (assuming employment and sleep). This assures freedom and choice. There is every reason to believe that newsreader users will behave likewise.

Which brings us to bandwidth...

A picture is worth a thousand words. Literally.

If so, what is audio and video? Moblogging and photoblogs will exacerbate this.   

In bandwidth terms, text is nearly free over land lines. Images, sounds, and video will comprise a growing share of bandwidth costs.

Each reader may consume 1000 feeds. 

We'll also grow in our ability to read them.

Newsreaders will help us filter and prioritize our reading.

  • Security filters (I required special authorization to read these posts)
  • Social filters (people who read this post also liked these)
  • Social network proximity (rank higher posts from family, friends, colleagues, FOAFs, my bowling team)
  • Commercial spam filters (black lists and bayesian filters)
  • Manual parameter filters (like the ones in most email clients)
  • Categorizers and Threaders (these posts from different sources belong together)
  • Pre-processors (these posts in German are interesting to you in English)
  • Content filters (flagging or filtering-in/out "offensive" pictures or phrases) 
  • Geographic proximity filters (these posts are by people in your neighborhood, those posts are about events near you)

So our capacity to follow more feeds will also grow by at least one to two orders of magnitude. Most people follow under 100 feeds in their newsreaders now. I follow nearly 1000, 50 religiously, 200 regularly. But all of them are searchable on my hard drive and they all pop-up in a balloon when they update.

And we don't have useful filters now. When the tools start to do more, the number of feeds consumed per reader will grow.

Syndication Growth = Denial Of Service

Lets assume that for every blogger there are two non-bloggers reading. That puts feed readership around 200 million. So you have 200 million people reading probing a thousand feeds an hour for updates. That's 200 billion probes an hour. Don't get me started on how many terabytes of flow that represents.

Do people only probe hourly? How often would you update for the latest scores during the World Cup or SuperBowl? For election results? For your medical report? Some fraction of services must support updates at closer intervals.

What architectures will support this scale?

Peer-To-Peer (P2P) and caching by intermediaries (communal aggregators) have helped others to scale. Both add delays to distribution while absorbing publisher bandwidth costs and connections. There is no reason why we shouldn't apply both architectures to this problem.

So, I ask the Echo community, What changes if:

  1. The physical location of the feed is not the feed's original source?
  2. A client must choose from among multiple sources of the same feed? 
  3. The publisher, while not abdicating a feed's authoritative url, wishes to redirect some or all consumers to any of a list of alternative locations? (Think mirrored downloads or bitstream)
  4. The copy of a feed file has been passed on ten times before you receive it?

P.S. I welcome challenges to my assumption, estimates, and conclusions. While I'm pretty confident in the shape of this analysis, details matter.

Click here to send an email to the editor of this weblog. ( comments) # 2478 9:35:56 AM G! DayPop!

 



Phil Wolff's
a klog apart
What's the next question?

Home

Click here to send an email to the editor of this weblog. AIM Y! @Ryze




 ?

Updated: 8/1/2003; 4:16:33 PM

Recent Posts


Previous post or Next

dijest rss Radio coffee mug
Phil's dijest


My Sources (600k)
My Neighborhood (700k)