The Downfall of the Open Web, And What Comes Next

In 2019, I left my job running the best damn interaction team in all of media – The New York Times Community Desk – looking for a new challenge, to venture out into what I called “the private sector.” 

I took over editorial for a place called Canopy, and became the editor-in-chief of a little slice of joy: An app that would simply recommend five articles a day from across the Internet. We were an aggregator, linking directly to external sites, working off our own homebrew personalization techniques. 

One very important aspect of aggregation: Lots of good content to aggregate that is not behind a paywall, imprisoned within a social media feed, or at the top of Apple News. So, I set out searching far and wide across the internet for websites. Day after day, month after month, I’d follow rabbit holes looking for interesting people making cool things of decent quality: art, journalism, essays, poetry, science and anything else that didn’t require academic-level skills to understand. 

I liked to think my impossible goal would be to re-discover the entire open internet. You know, the one I remembered from so many years ago, when I had a Myspace band, and Myspace friends, and message board beefs and custom .wav files to alert me when the girl I had a crush on logged into AIM. 

When I started collecting websites, I came to understand how much of my attention had been nibbled away by feed products. Reading the World Wide Web was eye opening! I saw the subtle ways that we all write and create for algorithms, we reframe our intentions for algorithms and pervert our creative fire in the process. 

To see talented people just... writing, taking cool pictures and geeking out about them, drafting clever, horrifically-SEO-unfriendly headlines that tell us something ineffable about the creator’s spirit. My job was to discover all of it, and find creative ways to use machine learning to deliver it to the right people. We were just 12-ish people working out of rented spaces in Brooklyn and Boston trying to find a way to get another round of funding – but still, it was humbling. The internet was never some magical fairy tale land, but this is what it was supposed to be. What an incredible mission! 

As time went on, though – only months, really – maintaining my list of sites and scrounging the internet for more, I began to see my “incredible mission” in a different light. I was documenting the decline and the eventual death of the open web. Our app was a eulogy. 

Site after site died or was in hospice – from art mags, to academic blogs, cultural collectives and niche music zines – they were all withering away. These are the kind of people who couldn’t or wouldn’t pay for SEO optimization, so their intended audience could never find them. They did some advertising, but even the local flower distributor’s spend had long since moved over to Facebook. They may have tried podcasting for a while, but absent a powerful network, they were lost and never found. They may have decamped to social themselves, but it probably wouldn’t even deliver posts to their followers for free.

I’d come to a sobering realization: If our app would ever reach its full potential, we’d have to start making our own content. The collapse of our editorial framework seemed only a matter of time. We had tried to use machine learning for good, but like the 90s Green Bay Packers, we were just a fart in the wind. 

I actually intended to write this piece about LinkedIn’s new-ish tooling that lets recruiters send you AI-generated messages, and allows you to send recruiters AI-generated messages.

Thankfully Andrew Hutchinson from Social Media Today rather gently (a polite fellow, no doubt) alludes to my obvious point that a system in which automated candidates reply to automated agents inquiring about automatically ranked lists of resumes is destined to collapse upon itself at scale. But hey, the fine folks at LinkedIn probably know that already. 

My point here is that we’ve been taking baby steps toward this GPT moment for a while, haven't we? SEO agents reverse-engineering algorithms, social media companies perfecting the short term cause and effect of reaction, all of us at home trying to make our caption sound a little bit more like an Upworthy headline. Current-gen AI is simply what comes next. So why the hell do I, an old newspaperman with a cigar in my mouth pounding the table demanding pictures of Spiderman as Erica rolls her eyes, want to have anything to do with this AI business? 

Because it is inevitable. Because the Internet is on the verge of a great reset as search engine AI cannibalizes the economies of its source material. Because after the sea of AI-generated bullshit subsides, and we all take stock of the reams of ill-conceived products rusting out under a sand dune in the desert of forgotten tech, there will still be people looking for interesting culture, science and news. 

Over the next few months, Erica and I were thinking we’d spend a bit more time on this site building AI tools that demonstrate how we might use AI to rebuild the dream of the Internet when the madness subsides. What if you could simply create something, and the AI simply helped you find your audience the same way that a local newspaper boy could find you by shouting the right headline on the right street corner?

We probably won’t build that here at MoP, but we’d like to create a few simple tools that help point the way there. More on that soon! 

Also coming soon: A researched explanation of why I’ve come to believe my newspaper boy analogy just might be the most powerful use for AI in society.