This example uses the Content Analysis and Unique modules to whittle down the feed and eliminate similar stories. The Content Analysis module evaluates a feed and tries to reduce the gist of the item to a few words. We first run our political news feed through Content Analysis, and it creates a y:content_analysis element containing those key words.
Then we run the feed through Unique, and configure it to remove any duplicate y:content_analysis items. What's left is a smaller feed with fewer repetitive articles.
Pipe Web Address: http://pipes.yahoo.com/pipes_team/BITULOFS3BGGymzjODY80A