A little bit of PubSubHubbub background

Like rssCloud, PuSH adds a line to your feed to let clients know where they can send PuSH subscription requests. In the RSS2 feed this looks like:

<atom:link rel="hub" href="http://josephscott.org/?pushpress=hub">

and in Atom looks like:

<link rel="hub" href="http://josephscott.org/?pushpress=hub">

The href attribute contains the URL that subscribers can send requests to. Details on what that request looks like are in the PubSubHubbub Spec. The short version, it’s a simple HTTP POST call with some specific field names.

Here’s an example PHP subscription request asking that updates to http://example.com/feed/ be sent to http://example.com/push/ via the hub at http://example.com/?pushpress=hub

$curl = curl_init( );

curl_setopt( $curl, CURLOPT_URL, 'http://example.com/?pushpress=hub' );
curl_setopt( $curl, CURLOPT_POSTFIELDS, 'hub.callback=http://example.com/push/&hub.mode=subscribe&hub.secret=not-telling-you&hub.verify=sync&hub.topic=http://example.com/feed/&hub.verify_token=133t-$7r1n9' );
curl_setopt( $curl, CURLOPT_VERBOSE, 1 );
curl_setopt( $curl, CURLOPT_POST, 1 );

curl_exec( $curl );
print_r( curl_getinfo( $curl ) );
curl_close( $curl );

The hub at example.com will then send new content (in feed format) right after it’s published to URL provided in hub.callback.

Why the PuSHPress plugin?

I love that there are large, open PuSH hubs available for anyone to use. There’s a list at http://code.google.com/p/pubsubhubbub/wiki/Hubs. In PuSH since most of the hard work (figuring out what’s new in a feed and sending that out to subscribers) is done via the hub this is very handy and really great for testing.

I also think there’s a role for a readily available simple hub that anyone can put up and use, in this case built on top of WordPress. And by leveraging WordPress the hub part can be greatly simplified. The real bonus for users is choice, they can relay the updates through 3rd party hubs, or use the built in hub in PuSHPress.

It’s worth noting that PuSHPress has a WordPress filter on the array of hubs; pushpress_hubs. This allows other plugins to easily add to or replace the hubs mentioned in the feeds by PuSHPress.

To help keep things simple and limit potential abuse the PuSHPress plugin will only allow subscriptions for the RSS2 and Atom feeds of the WordPress blog that it is installed on.

My thanks to Brett Slatkin for helping test PuSHPress. His tests revealed a few bugs that were quickly addressed.

How fast?

When talking about these realtime-ish update features this question often comes up. Fair enough, I’d want to know too :-)

The PuSHPress plugin schedules pings to go out right away via the wp_schedule_single_event function in WordPress. I suspect for most people this will be quite fast. If your server is under tremendous load and really slow, then this will be really slow too, just like the rest of the server ;-)

On WordPress.com the details are a bit different. Instead of scheduling pings to be sent out with wp_schedule_single_event it adds them to our asynchronous jobs system ( more info available at http://code.trac.wordpress.org/ ). The jobs system is setup to do exactly these kinds of tasks really, really fast. Don’t be surprised if the ping shows up before you get a chance to finish the first Mississippi.

The future

So where is all this going? Who knows, but it is fun to watch! :-)

Further reading

If you have an interest in this area there are plenty of resources:

• PubSubHubbub project site
• Google Group
• Spec

The spec is a quick read and to the point, worth reading.

And there you have it, a little more in-depth discussion of PuSHPress.

Related posts:

1. RSSCloud For WordPress
2. mod_ping, Maybe I Should Have Called It PubSubHubbub
3. rssCloud WordPress Plugin Update – 0.4.1

The only change was the addition of 2.6.27 to the list of libxml versions that the plugin looks for. I’ve had a few reports that this version also has problems with stripping brackets.

For those that haven’t been following this problem here’s the short version of the story. Some combinations of libxml and PHP don’t play nice with each other, the result being that brackets get stripped out of XML content. This is a particular problem for XML-RPC requests in WordPress. This plugin injects a small hack into the XML-RPC data before it gets processed in an attempt to preserve the brackets.

Related posts:

1. WordPress & libxml2 Episode IV: A New Plugin
2. Plugin – LibXML2 Fix
3. Update On libxml2 Issues

Version 0.4.1 of the rssCloud WordPress plugin is now available. The biggest change is adding support for the domain parameter in notification requests. This means that rssCloud updates processed by the plugin are no longer limited to being sent to the IP address that the request came from. Support for the domain parameter is live on WordPress.com as well.

When a domain parameter is included with a notification request the verification process does the following:

• Sends an HTTP GET request to the {domain}:{port}{path} URL
• That HTTP GET includes to pieces of data: url and challenge. The url field contains the URL of the feed that we’ll been sending pings about. The challenge field contains a random string of characters
• The response back must have a status code of 2xx and the body must contain EXACTLY the contents of the challenge field. If both of those conditions are not met then the verification process will consider this a failure

For notification requests that have no domain parameter the verification process is unchanged from before.

Another item that some may find helpful is a new constant – RSSCLOUD_FEED_URL – if that is defined they it will be used as the feed URL of the blog instead of determining it via get_bloginfo( 'rss2_url' );. For plugin authors that provide options for an alternative feed URL note that can override the default in WordPress via the feed_link filter. That filter can be used instead of the RSSCLOUD_FEED_URL constant and will bubble up through the get_bloginfo( 'rss2_url' ); call.

Related posts:

1. RSSCloud For WordPress
2. Plugin – RSSCloud
3. WordPress Plugin Update: LibXML2 Fix

So what does this really mean?

From the point of view of WordPress and this plugin there are three main additions:

1. Adds the <cloud> element to your RSS2 feed (more details here and here) which tells clients where and how to sign up for notification requests.
2. Registers a URL handler with WordPress to process the notification signups.
3. Sends out notification updates when a new post is published.

The cloud element looks like this:

<cloud domain='josephscott.org' port='80' path='/?rsscloud=notify'
  registerProcedure='' protocol='http-post' />

The domain, port and path attributes combined to form a URL, http://josephscott.org:80/?rsscloud=notify in this case, where others sign up for notifications. The registerProcedure attribute is the XML-RPC method to be called if the protocol attribute was xmlrpc. Since the plugin uses http-post for the protocol the registerProcedure field is blank.

Using this same example here is a small chunk of PHP code that uses the cURL library to sign up for notifications:

$curl = curl_init();

curl_setopt($curl, CURLOPT_URL, 'http://josephscott.org/?rsscloud=notify' );
curl_setopt($curl, CURLOPT_POSTFIELDS, 'notifyProcedure=&protocol=http-post&port=80&path=/~joseph/rsscloud/&url1=http://josephscott.org/feed/' );
curl_setopt($curl, CURLOPT_VERBOSE, 1);
curl_setopt($curl, CURLOPT_POST, 1);

curl_exec( $curl );
print_r( curl_getinfo( $curl ) );
curl_close( $curl );

This code sends an HTTP POST request to http://josephscott.org/?rsscloud=notify asking to get a notification when the http://josephscott.org/feed/ feed is updated. The notification is to be sent to the remote IP used in the request (this means notification requests must be sent from the IP that will be receiving the notifications), port 80 with a path of /~joseph/rsscloud/ and it will be given the update data via an HTTP POST. The notification script will get $_POST data that looks like:

Array
(
   [url] => http://josephscott.org/feed/
)

It is then up to notification script to turn around fetch the updated feed.

How fast does all this happen?

It depends :-)

On WordPress.com the notifications happen through the jobs system, which means it will be sent out very, very quickly. On a WordPress.org powered blog with the plugin it schedules notifications to get sent out as soon as possible with the wp_schedule_single_event( ) function. Scheduled events in WordPress are checked on each page load, so if you publish a new post and then view it on the front page of your blog the notifications will get sent out in pretty quick.

I think for most blogs these approaches will work fine and send out notifications with very little delay.

What does this mean for feed readers like Bloglines, Google Reader, etc.?

I believe that many (most?) public feed readers like Bloglines and Google Reader already listen for feed updates via pings (like those sent to Ping-O-Matic). With an RSSCloud enabled WordPress blog they can register for updates to specific feeds. Why would they do this if they are already getting ping updates? Unfortunately the ping updates are similar to email, they have massive amounts of spam in them. Since RSSCloud isn’t a stream of everything, but a specific request for specific updates they could sign up for updates to those feeds that they believe are more likely to be legitimate.

Signing up is simple for a feed reader (or anyone/thing) to do:

1. Look for the <cloud> element in the RSS feed
2. Sign up for notifications using the data from the <cloud> element
3. Process notification that are sent to it from WordPress

Right now I believe the only feed reader that supports RSSCloud is Dave Winer‘s River2.

If you are working on RSSCloud support in your feed reader let me know, I’ll be watching the RSSCloud stats on WordPress.com. And of course if you run into problems with RSSCloud on a WordPress blog (ORG or COM) I’m happy to help track down any bugs in our implementation.

Related posts:

1. rssCloud WordPress Plugin Update – 0.4.1
2. Plugin – RSSCloud
3. PuSHPress: A PubSubHubbub Plugin For WordPress

Further details and history are at my LibXML2 Fix plugin page.

Update: Rein caught a typo in version 0.2, so make sure that you get version 0.2.2.

Related posts:

1. Conclusion of libxml2 Issues – Use PHP 5.2.9 & libxml2 2.7.3
2. Plugin – LibXML2 Fix
3. WordPress Plugin Update: LibXML2 Fix

Just as I was ready to throw in the towel and look at getting a work around into core WordPress Peter Westwood suggested a method that would allow this “fix” to work in a plugin. His idea was spot on and I wrote a small WordPress plugin, LibXML2 Fix, the next day and tested it on a WordPress install with this problem. It fixed the stripped brackets problem for XML-RPC requests, so I’ve submitted it to the plugin directory and it’s now available at http://wordpress.org/extend/plugins/libxml2-fix/.

To be clear, the best way to fix this is to upgrade to PHP 5.2.9+ and libxml2 2.7.3+. But if you are stuck in a hosting situation where that’s not possible then the LibXML2 Fixed plugin will fix XML-RPC requests so that brackets don’t get stripped.

I’ve closed ticket 7771 and will be pointing people to this plugin if they can’t upgrade PHP and libxml2.

Related posts:

1. WordPress Plugin Update: LibXML2 Fix
2. Plugin – LibXML2 Fix
3. Problems With libxml2 For WordPress XML-RPC Users