Last week I began discussing Yahoo Pipes, a service provided by (as you might have guessed) Yahoo.
This service is designed to route and transform data taken from one or more online sources and generate an output in RSS 2.0, RSS 1.0 (RDF), JSON and Atom formats.
I discussed the range of input source choices (generically referred to as “feeds“) for Pipes, now I want to cover what you can do with the data.
Pipes provides a number of operator modules that do things such as counting the number of items in a feed, filtering out feed items according to string matching rules, removing items with duplicate strings, performing a Regular Expression operation on feed items and creating RSS feeds.
There are also modules for building URLs, manipulating strings (including one that extracts the linguistically most important terms from feeds and another that will translate items into other languages which will surely annoy the American Translation Association).
As if that weren’t enough, there are also modules to define and format dates, convert text strings to geographic locations and work with numbers.
As I mentioned last week, you create a Pipe using the Pipes Web-based IDE by dragging modules onto the canvas, setting any parameters the modules might have, and then “wiring” them together by dragging from the output of one module to the input of the next. To test the Pipe you just click on Refresh at the bottom of the IDE screen and the Pipe’s results are displayed.
So, what can you use the Pipes service for? You might want to use Pipes, for example, to filter the flood of IT news and track the main topics at the top of the industry news.
I created a pipe (after much wrangling; the Pipes documentation could be better) that starts by ingesting the ITBusiness.ca Top Stories feed using the Fetch Data module (I couldn’t use the Fetch Feed module as the feed is in XML format). I had to give the path or branch of the XML data I wanted to analyze, which the Netflash feed names “rss:item”.
The pipe then sends the item branches to a Loop module that, in turn, contains a Term Extractor module. The Loop module is configured to extract the data from the child nodes named “rss:description” which the Term Extractor then strips down to just the significant phrases.
The phrases are sent to a Create RSS module that makes a very simple output feed where the titles of the feed items are the extracted phrases. Voila! A very simple RSS formatted list of all of the terms used in the Netflash feed. You can try out the pipe here.
Because some of the phrases will be duplicated (for example, as of this writing the pipe produced 457 phrases which included one instance of “network neutrality”, three of “net neutrality“, and nine of “apple” along with 12 that included the text “fcc”) the importance of phrases can be estimated making the output suitable for building a tag cloud.
Now, you may be wondering what are the limitations of Pipes? First of all, and in common with most Web services, Pipes doesn’t offer any kind of performance guarantees. In fact you may occasionally find that the service responds very slowly or might not respond at all. Such are the downsides of free.
Next, only online data can be used as inputs so if you want to use desktop data sources you’re going to have to get clever using reverse proxies and other hacks. There is also no support for Extensible Stylesheet Language (XSL) transformations which, while not critical, would make dealing with XML data a whole lot easier.
Finally, the whole system is proprietary so any effort you put into creating a Yahoo Pipe won’t be portable and, should Yahoo ever decide to pull the plug on this experiment, you’ll be left high and dry.
Those limitations aside, Yahoo Pipes is an amazing achievement in both its functionality as well as its user interface.
If only Yahoo would provide a guaranteed service or, better yet, make Pipes available to run on your own servers, it could be a great service. Yahoo Pipes gets a rating of 4 out of 5.
Gibbs is done with pipework in Ventura, Calif. Your plumbing to [email protected].
