Sometimes the flood of information that we get from the Internet can be overwhelming. But you can turn your flood into a more manageable trickle that focuses on what you want to know. Yahoo’s Pipes provides a graphical approach to building custom news sources by letting you snap together RSS feeds, Web pages, and other sources, and then filter the results. The service combines the familiar action of drag-and-drop manipulation with the metaphor of flow charts to aggregate and process results.
My introduction to Pipes was when I wanted to create a “Glenn feed.” I write for many publications, and I wanted a way to have a single RSS feed that incorporated results from all the outlets I work with. Yahoo Pipes let me take many different RSS feeds and Web page search results, collapse them to a single news feed, and then offer that on my Web site for those who care. (It also lets me to track when my own articles appear. Yes, I often don’t know precisely when an article I wrote will appear-such as this one!)
I’ve also used Yahoo Pipes to extract custom feeds from sites that don’t offer per-author RSS. At one technology site, I like to read a particular writer, and a Pipe lets me scan the feed for his name, filter his items into RSS, and then aggregate that into my newsreader, NetNewsWire.
Related stories
Gmail Priority Inbox filters frivolous e-mail
Seven browser tweaks to reach Web productivity nirvana
The name Pipes refers to one of the remarkable ideas incorporated into Unix in its very early days. A pipe (written as a vertical bar or |) means “take the output of this one program and send it to the next.” This program workflow is like a pipe connecting machines: Put milk, sugar, and flour into the mixer; batter drains into the extruder, which pushes it onto waffle irons; out come ice-cream cones!
Let me walk through a project with a reasonable test case: You want to know whenever hurricanes are mentioned at any of several major news sites. Go to Pipes, and create a Yahoo login account if you don’t have one. Then, click the Create A Pipe link to begin. Note that Pipes doesn’t appear to auto-save, but it does. You can also click the Save button after making significant changes to safeguard against loss of work from a browser crash.
Add RSS feeds
Start by deciding what raw material you want to dump into your pipes-the inputs. Pipes can work with RSS feeds, HTML in a Web page, XML-defined data, and even information from a Yahoo Search or a Google Base database.
Since it’s most likely you’ll work with RSS and HTML, we’ll make this pipe examine feeds from CNN, The New York Times, The Los Angeles Times, and USA Today. In the Source heading in the list of categories at left, drag Fetch Site Feed into the main window, or click the plus sign to right of the item’s in the Sources list. Paste the first of the site’s URLs into the first URL field. Click the plus sign next to URL to add more fields, and paste each site’s URL in turn into the field.
Each site’s favicon-a square Web site icon, usually customized for a site’s design-should automatically appear to the left of each URL if Pipes was able to retrieve the site’s main RSS feed correctly. You’ll see a question mark if no icon is available. If you see a yield sign, you need to double-check the URL.
Another way to add feeds to use the Fetch Feed option under Sources. Use this item when you want to add a specific feed from a site with many feeds. For example, many news sites have tailored feeds filtered for different topics. These may include more stories than the main RSS feed.
The Fetch Feed item requires you to find a specific URL for a feed. Look for an RSS link on the site’s home page or, if need be, use the site’s search tool to search for RSS. Make sure that the final page you find is actual RSS feed. (Some sites, for instance, make you click a button labeled XML to get to the final URL.) Once you have a URL copied, paste it into the Fetch Feed URL field. Click the plus sign to add more fields.
Filter the results
Now that we have a set of feeds queued up, we can filter the feeds’ content. Yahoo Pipes offers a lot of possibilities under the Operators link to the left, but for our purposes, the Filter tool will do the trick.
Drag Filter into the layout or click the plus sign next to its name. Filter examines all the feed input and passes along only the parts that match your criteria. For our hurricane example, I set Filter’s Permit/Block pop-up menu to Permit. (Alternatively, you could select Block to pass through everything but items you don’t want to read.) I set the All/Any menu to All, though this isn’t critical because we’re only going to add one condition.
Click in the field under Rules to bring up a menu of tags used in the RSS feeds. Here you can select what descriptive information-for instance, author, pubDate (for the date an item was published to the feed or in a publication), and title-you want to focus on. In this case, I’m trying to cast a broad net, so I select item.title, item.description, and item.y:title. (Depending on what feeds you use in your pipe, you might not see an item.y:title entry. This is a special definition in one of the feeds. The other item entries are more generic RSS field names.) For each field, I set the middle option to Contains and then enter “hurricane” in the text field to its right.
Connect your pipes and test it out
Now we can connect the pieces. Hover over the circle at the bottom of the Fetch Feed selector and drag it to the circle at the top of the Filter item. A curvy line appears, connecting the items. Note that an item called Pipe Output has appeared at the bottom of the page. That’s the piece in the chain that produces your custom feed. Connect Filter to Pipe Output.
You can test how your filter is working by using the Debugger pane at the bottom of the window. Make the window bigger by dragging the resize bar (centre of the separator line) higher.
Click any item in the layout or hover over the Debugger label and choose an item from a pop-menu that appears. The debugger runs and shows what output has reached that stage. That can help you troubleshoot problems.
Finally, click Save and name your custom feed. An option to Run Pipe, or generate the output, appears at the top middle of the screen. Click that link, and Yahoo opens a new window with the results.
Use your custom feed
The Yahoo Pipes output window will let you choose to see your custom feed as an RSS feed or even get the results via e-mail. (Click the More Options link for e-mail and other choices). If you click the Get As A Badge link, Pipes will provide code you can use with blogging tools or Websites to embed the current results of the feed, too.
The late senator, Ted Stevens, was widely ridiculed for describing the Internet as a series of pipes, but this wasn’t an inaccurate summary, in many ways. Yahoo has taken that metaphor to new lengths. Use this tool to turn the Internet into a set of filtered pipes that filter the mass of information for you, delivering only what you need.
Glenn Fleishman has written several ebooks in the Take Control series about file sharing, remote access, and Wi-Fi. He contributes regularly to Macworld.