Pipe Dreams: The life and times of Yahoo Pipes

That is the laggiest animation I've ever seen :( Totally unusable (Chrome on Android, Pixel 7). Is there a text version?

I miss Yahoo Pipes so much. And YQL. In 2009 I implemented an effectively cross-domain XHR (ajax) plugin on top of jQuery so I could grab random webpages on the client-side and query them via CSS selectors (converted to XPath). YQL would give me JSONP back so it all felt quite magical. Ah the old days...

I have to plug one of my favorite workflow automation tools that is a namesake and was fairly recently developed: https://pipedream.com/

Would definitely give it a try if you’re looking to automate Yahoo Pipes style.

I have no affiliation to them, just a happy user

wow. what a great mobile read xp to match a great story. reminds of reading "in the beginning was the command line."

Kudos to the retool design team and Glenn on a really nice execution of this story. The Pipes inspired illustrations are gold.

(Hi, Retool employee here who worked on this piece.)

This is the second installment[1] in what we hope will be a regular series telling the stories of novel programming environments that had a lot of influence on developers.

Creating these is a labor of love—we get to talk to the original teams who built products that were formational for us. We have an incredible in-house creative team that finds a way to sneak these fun mini-sites in amongst their other work.

For me, the most exciting part of working on a product like Retool is that it's an evolution of lot of ideas from the past. We're always looking at prior art to spark ideas and challenge our assumptions as we're building. There are many great ideas in old computing papers and products that were either before their time, or lost when there were major shifts in the industry. Pipes was a touchpoint for us as we built out one of our newer products[2].

We have lots of ideas for future installments in this series, but if you have any programming environments that were near and dear to you heart, let me know!

[1] Our first deep-dive was on Visual Basic. I'm very biased (I wrote it!), but if you like Glenn's Pipes piece, you'll probably enjoy this one too. :) https://retool.com/visual-basic

[2] Retool Workflows: https://retool.com/workflows

Is it still useful today? I skipped the rss time.

Proud Pipes user.

Does the web need something like this today and in what form? All the replacements that are offered pale in comparison to excitement it was to use Pipes 15 years ago. Are we simply past that point?

I’m interested in how an article like this comes to be. I can tell that it was made with a lot of care and I’m sure it took a while to polish to the final result (high quality content + polish = high cost). I’m still not sure what’s the purpose, it doesn’t seem to be a marketing piece, although the underlying message is clear between the lines: if you liked pipes, you’ll like retool. Or something along those lines.

This brings back soo many old memories of mine @ Yahoo! . We had soo many tooling - YTS, YCS benchmarks, Vespa, SherpaDb, Pipes, shimproxy the list goes on. Its unfortunate leadership didn't have the foresight to saasify them in a way AWS did. Tooling at Yahoo!! was soo ahead of its time.

Readers may enjoy Flowpipe [1], an open source cloud scripting engine we launched this week. It allows creation of pipelines in HCL to connect HTTP queries, run containers, execute lambda-compatible functions, query databases, etc all from your own machine / CLI. It can also be combined with Steampie[2], our open source project to query cloud resources (139+ plugins) with SQL. We're more DevOps focused, but Yahoo Pipes was part of the inspiration for our naming of these "pipes" :-)

1 - https://github.com/turbot/flowpipe 2 - https://github.com/turbot/steampipe

I used Pipes to get a feed of specific categories of used video games and game systems from Craigslist in college and beyond. I got a lot of stuff that way, and visited many parts of the cities I lived in that I otherwise wouldn't have.

I loved Yahoo Pipes. I had so many pipes set up to filter RSS feeds and Podcasts down to just the types of content I cared about. Insanely useful, I miss it so much.

A big example, I listen to a podcast where they have standard episodes and episodes where they play D&D and I just don’t care about the D&D ones so it was really nice to be able to filter them out.

Just this week, I was thinking if I could recreate my use of Pipes...

From each track in the RSS feed of tracks scrobbled on last.fm, do a google search and pipe the output back to an RSS feed into my reader.

Neither chatGPT nor Google could make sense of my request. Any ideas?

It took me a while to realise that AWS Steps is like Yahoo Pipes.

It isn’t as simple, and there aren’t as many clickable components, but it was weird that I was so slow to realise it.

I guess someone with more insight can point out where I’m wrong.

Web 2.0 times were great. It was just taken as a given that the future was open standards, interoperable web services, and creative tools like Pipes that let users and developers easily leverage them. Social media and the drive to monetize users at all costs destroyed that potential.

When Pipes died I saw all the comments and wondered why no one built a successor project. When after a few years that still hadn't happened, and after an idea on how to get it to work had popped into my head, I built it myself: https://github.com/pipes-digital/pipes is the source code and https://www.pipes.digital/ a hosted version.

To call it a successor project is maybe arrogant and misleading - it can not do half of what the Yahoo project did, and is sorely missing time and work from my side to make it into what it should be. But if you are nostalgic for the concept and especially if you just want to filter or combine a few RSS feeds, or create one for a HTML page that is missing one, it's worth a try I still think. I personally use it regularly to get specific YouTube videos from a few channels that have many other videos I do not care about.

The article mentions one of the big problems for this kind of project: The APIs are missing or not open anymore. For example, a comment here mentions doing a google search for a song title and putting this in a feed - but you can't realistically parse Google. So you have to use the API, which I didn't know existed until looking it up now, and calculate the price tag attached to it very carefully. Maybe it would be a realistic integration to provide with a different search engine. Or a bigger project should run of the local search engines for the pipes block.

The article is also right in that there is a lot of influence of Pipes in different projects. I was always wondering why that influence was often so little - to react to output on a page in a wizard for example, instead of having blocks. I think I understand that a little bit better now - it is really a strange abstraction level. You are working with a flow of structured data, without necessarily having the correct data structure for that, and then that needs to be transformed somehow. The idea that you might have JSON on a page somewhere you want to extract and manipulate, but then that data is inside of your outer feed format - it feels wrong to confront the user with these details, it makes sure the target groups is developers, but it also does not seem like something that can be reasonably universally solved. But maybe I'm wrong here, and this is more a problem with my implementation (where RSS is piped from block to block) than with the concept itself.

The idea of a real Pipes 2 is very exciting to me though. Give it some initial funding, built all of the modern APIs into it that can be added (and in doubt just give users the option to use their own API keys) - especially things like ChatGPT, after all those LLMs are great in converting structured data - solve the data format question and crucially keep the interface block based (but maybe built with something like Flutter, so it can be better). Could work, not to win the world, but as something that can justify its existence next to ifttt and zapier. At least I regularly get the impression that the concept still fascinates people a lot, though their specific use case is very often very specialized and then often enough not something my small pipes project can cover. But if it were broader...

Yahoo Pipes was the spiritual father of Node RED

I never used Yahoo Pipes. Was it primarily for automating for manipulating stuff grabbed from the web. Or could you also usefully use it to wrangle/ETL local data files and databases?

i have a few (new, unworn) pipes tshirts left. mostly XL thou. and a ton of pipes stickers.

If you're more handy with code, I'd also recommend checking out https://val.town which lets you quickly create automations and endpoints.

Great article by Retool! Yep, Pipes was something different a the time!

The best part of Pipes was that no one blocked it or it was difficult to block. I used it to scrape data from web pages that I served up via Android widget apps.

I almost wrote my thesis on a system using Pipes, until my professor changed his mind and I did something else. Good times.

These days there's stuff like Make.com (formerly Integromat), IFTTT, Zapier, etc. to do some low code piping. Airtable also got automation and pipeline support.

Still miss Pipes though. It was awesome!

I have no idea how to use this website, but if anyone wants to read up on pipes: https://en.wikipedia.org/wiki/Yahoo!_Pipes

I had my first taste of "automation" with Yahoo Pipes - scraping sites, generating alerts etc. Did a lot of workflows back in the day.

My favorite “project awareness” technique for work was through pipes and I’ve never had something as quick as way back then.

I had a yahoo pipe that combined and deduped about 10-15 jira projects and about 30-50 GitHub commit streams into a single rss feed that I read through google reader.

It was glorious because it let me know quickly who changed what and when and where. And it was rss so I didn’t have to search through email or look at multiple notification pages.

Another amazing node-based app that was deprecated is Quartz Composer on Mac. It was extended by Mike Matas' team when he was at Facebook: https://medium.com/the-year-of-the-looking-glass/introducing...

although this was also later deprecated in favor of Origami Studio, which retains the node-based UI: https://origami.design/

I thought this was going to be about https://en.m.wikipedia.org/wiki/Pipe_Mania

Yahoo Pipes was full of XSS vulns. I used to play with cookie grabbers

I love Pipes, I also think it is the last best thing from yahoo

Everytime I see Yahoo Pipes I remember the video [1] from Tom Scott about programming. ( The video title view count is updated in as close to real time as possible )

[1] This Video Has 69,187,812 Views

https://www.youtube.com/watch?v=BxV14h0kFs0

A lot of what I thought were good, accessible, productive, entry to middle level technology ( HyperCards. Yahoo Pipes. Flash, Delphi etc ) never gained enough market. They were all the sort of middle ground good enough options. Instead technology is often filtered from the top end extreme. So everyone tries to use what Google, Facebook, Amazon, Microsoft are using. Because those tools are deemed as what Professional should use. The only rare exception that is powering most of the word is Microsoft Excel.

I feel like theres a potential for B2B solutions in this space. Especially for workers who want to bootstrap something quick, avoid something like Teams or other behemoths, create something for a one time survey, campaign, shorter process etc.

So much was lost transitioning from web 2.0 to whatever walled dystopian garbage we have now.

The concept of pipes are way back from the Unix days, and the idea of this modularity and later building stuff from a flow based interface like this somehow never translated into anything on the frontend side.

I've been met with simple specs that had to be built with custom web solutions a few times now where i've almost started building a generalised model like this that could scale because most apps could be defined like this, as entities related with filters between them in sequences related to each other and web views where people could interact.

The closest i found back then was something like Airtable/Zapier combinations that was extremely limited.

I think you could make the case that Zapier is the monetized version of Pipes. It's a 5 billion dollar company now.

I was just telling someone about this. Good old days.

Congrats to the authors for taking a potentially great article and wrecking it with insane distractions and useless web BS. Pipes was an amazing piece of software and deserves better than that.

I used pipes back in college to randomize our roommate ordering in picking dorm rooms. I had it take the top headline on The NY Times website, hash it, and pull out numbers from 1-6 in a deterministic way. It was the best way I could think of to do it from afar over the summer, and convince everyone it was fair without needing to run code ourselves. It was useful and fun to use!

When people believed the internet was a series of tubes

I used this service quite a bit in combination with drupal 5, shortly before it shut down. So 1.5 decades ago you could build quite large systems just by clicking stuff together.

For anyone interested in visual programming, I would recommend having a look at Flow Based Programming[0] better known for being the inspiration for Unix pipes, which in turn were name giver for pipes in yahoo pipes.

[0]=https://jpaulm.github.io/fbp/

I 'member it used to be a thing to blogspam AdSense with Y! Pipes via RSS subscriptions to source sites → translate to non-English language → translate back to English → auto-post to AdSense-encumbered Wordpress blog.

I miss the internet circa 2013. I had a blog, and twitter would fetch content from the blog feed and automatically post it. Simple and elegant.

Pipes is dead, at least we still have ifttt.

Pipes was awesome for sure. It fit so nicely into a golden web 2.0 era sort of period where there was still loose open interoperability thanks to prevalent RSS and similar tech. Effective at cobbling together simple things to make simply useful things.

What's with this epic design and work put into this deep dive tho? What an effort from Retool. Suppose maybe just wondering why they didn't do it earlier! Ha