Inside the Brotherhood of the Ad Blockers
For the advertising industry, ad blocking is an existential crisis. For the Pi-hole community, it’s a sport.
Anyone who works in the $200 billion digital advertising industry should be scared of people like … Mark Drobnak, because the ad blocker he uses is way more powerful than yours. The college freshman says it feels as though everyone at Rochester Institute of Technology, from his roommate to his professors, has installed some way to ward off online ads. Drobnak is one of the die-hards who goes further, working with a handful of comrades to build what they call “a black hole for advertisements.” His parents say the one he built them works great.
Pi-hole (as in “shut your …”) is a free, open source software package designed to run on a Raspberry Pi, a basic computer that’s popular with DIYers, fits in the palm of your hand, and retails for about $35. Most ad blockers have to be installed on individual devices and work only in web browsers, but Pi-hole blocks ads across an entire network, including in most apps. (Two big exceptions, both for technical reasons, are YouTube and Hulu.) It can’t block ads inside Facebook, but it can stop Facebook from following you around the web. It’ll let you play Bejeweled without seeing ads between games, watch Mr. Robot ad-free in the USA app, stream NPR with silence in place of the sponsor messages, and avoid the banner ads that have become common on internet-connected TVs. If friends come over and connect to your Wi-Fi, it’ll block ads for them, too.
Drobnak discovered Pi-hole in high school in 2015, after he and his siblings had already used their Raspberry Pi to play tic-tac-toe, program an elaborate light show, and monitor their respective addictions to electronics. The ad blocker, created by a Minnesota programmer named Jacob Salmela, was 2 years old and still fairly rudimentary. Less than a month after installing it at home, Drobnak hacked together a web interface to let users more easily block or whitelist sites. Two months later, Salmela invited him to join a tiny, all-volunteer development team. “Ads are annoying,” Drobnak says. “Pi-hole gives you control over that.”
About 18 percent of U.S. web users have an ad blocker, says PageFair Ltd., a company that helps advertisers find technical ways to work around the software. (Its estimates are among the more conservative ones.) Outside the U.S., the numbers are more dramatic. Desktop ad-blocker penetration is 24 percent in Canada, 29 percent in Germany, and 39 percent in Greece, according to PageFair. The practice is growing fastest on mobile devices in Asia, where data allowances are typically lower. In Indonesia, 58 percent of users block mobile ads. “In the early days, it was privacy activists and people who had an objection to capitalism in principle,” says Sean Blanchfield, chief executive officer of PageFair. “These days, it’s just average people.”
Only a few years ago, even people who hated ads saw ad-blocking software as akin to stealing. But online advertising has grown so predatory that while blocking is estimated to cost publishers billions of lost revenue a year, it’s started to seem less like robbery than self-defense: Ads slow devices, eat up data plans, and sometimes deliver malware. Meanwhile, the industry is building ever-more-detailed dossiers on every user based on web habits.
Among other things, the online advertising business model has incentivized clickbait—and worse—at enormous scale. Facebook Inc. and YouTube LLC figure out how to make people spend more time on their sites to maximize ad inventory. This has abetted the spread of fake news, violent children’s content, and Logan Paul.
Enforcement of the European Union’s General Data Protection Regulation (GDPR), which requires companies to get consent before tracking users, is set to begin on May 25. And with members of Congress grumbling, Silicon Valley, which has ignored complaints about invasive ads for decades, is beginning to acknowledge the scope of the problem. Google and Apple Inc. have added features to their browsers that limit the most intrusive and invasive ads. In response to its Cambridge Analytica scandal, Facebook has taken steps to limit its rampant data-sharing and also ended partnerships with companies that combine online profiles with offline credit card transactions, public records such as voter registration and home purchases, and store loyalty programs. Even the Interactive Advertising Bureau, an industry trade group unsurprisingly hostile to ad blocking, now says the practice is “a crucial wake-up call to brands and all that serve them about their abuse of consumers’ goodwill.”
Pi-hole is installed on only 140,000 networks. Unlike more popular ad-blocking browsers (Brave, which claims 2 million users) or browser extensions (Adblock Plus, 105 million), it requires a dedicated computer and some tech savvy to set up. Still, it has assumed an outsize role in the ad-blocking movement. Its 22,000 true believers on Reddit help a lot, says Drobnak, who’s spending 5 hours to 20 hours a week working on Pi-hole between computer science classes. The developers have discovered spying by internet-connected TVs (which collect data for ad targeting), lightbulbs (users have reported some LED bulbs mysteriously connecting with the manufacturer’s server every 2 seconds), and printers (including one that sent out 34 million queries in a day).
Drobnak’s fellow core developers, all volunteers, say what unites them most is resentment of just how far the advertising industry has overreached in building its online empire of distraction and surveillance. There’s a corollary motivator, too: Puzzling out ways to frustrate the industry’s efforts—to zap millions upon millions of ads—can be really, really fun. “There’s a huge community behind it,” Drobnak says. “It’s just tinkerers trying to figure stuff out.”
The rise of ad blocking mirrors an explosion in online advertising technology. Barely 100 digital-only ad-tech companies operated in 2011; today there are about 2,000. Most arose with what’s known as programmatic advertising, automated systems run by the likes of Google and Microsoft Corp. that promise to match every ad with the person the ad is most likely to influence.
Dissect how such systems work, and it’s easy to be outraged. When you load a website, it sends a series of requests to other web domains to auction your eyeballs to the highest bidders. The number of intermediaries involved changes with every page load, but on a recent visit, the homepage for one popular U.S.-based news site sent 20 requests to 10 ad exchanges, each of which likely offered the space to hundreds of advertisers. It also set 47 cookies with unique tracking IDs, many of which log user data such as location, gender, age, and likes and dislikes based on browsing behavior. These data give advertisers a sense of how valuable you might be as a customer, and therefore how much to bid to show you an ad. When one of the advertisers wins the auction, an ad appears on your screen. The whole process takes less than a tenth of a second.
As a side business, every company involved in any step of the process may also try to place a cookie or tracker to collect more data on you for later use. Such companies often swap data to try to identify users they have in common, and they may pull in your email address, name, public records, and credit card history. “Ad blocking has grown in response to a lot of legitimate problems,” says PageFair’s Blanchfield. His previous venture, Jolt Online Gaming, collapsed because 30 percent of its users were blocking ads. He and his co-founder were, too.
By acting as a traffic cop at the network level, rather than in a browser, Pi-hole can cut off the nested bidding and tracking processes from the start. It takes on the role of your network’s Domain Name Server (DNS), meaning it translates IP addresses into URLs and vice versa. So if a website tries to contact what the Pi-hole knows to be an ad server, “it sends a request to the Pi-hole for the ad, and the Pi-hole is like, ‘Hah, I’m just going to return an empty page to you,’ and the ad server is never contacted,” Drobnak says. In the ad slots, the user typically sees blank white rectangles.
Installing Pi-hole took me about an hour with the help of a friend who’d done it before, and much of that time was spent setting up my new Raspberry Pi. The Pi-hole can run on any computer, but in general you probably want it to be a cheap model that can be left powered on and online. Even the Raspberry Pi 3 Model B+, the most full-figured Pi, ships naked and blank: no case, no operating system, no apps, just a single green circuit board with components and ports sticking out of it. Spring for a case if you like, plug in a monitor and keyboard, and install an operating system. Then it’s simply a matter of connecting the Pi to the internet, installing the Pi-hole software with a single line of code (curl -sSL https://install.pi-hole.net | bash), and setting it as the DNS server for the network. In case of trouble, Pi-hole fans tend to respond promptly to questions on their forums.
Turn the Pi-hole off after you get used to it, and you find your brain engaging more with the ads than it used to because it’s forgotten how to glance past them. Were they always so garish? Who needs that many razors a month? And why would anyone set a video to autoplay with sound?
After three weeks, my Pi-hole logs reported the system had blocked more than 39,000 requests, 29 percent of the total on just two devices (a phone and a laptop). They were all ads or ad-related tracking from places such as analytics.localytics.com, static.doubleclick.net, googleadservices.com, graph.facebook.com, app-measurement.com, sb.scorecardresearch.com, and capture.condenastdigital.com. It’s fascinating to look at Pi-hole’s dashboard, a colorful layout of numbers and graphs, and think about what the network is doing—and just how many entities are keeping watch.
Salmela, the creator of Pi-hole, is a 33-year-old Linux administrator who lives outside Minneapolis with his wife and son. He has buzzed brown hair, favors black T-shirts, and rarely volunteers information. He worked at a Target for 12 years before he got bored enough to go to college, where he studied computer networking. With a bachelor’s in hand, he got a job in the IT department of a high school, quickly automated everything he needed to do, and got bored again.
In the summer of 2014, facing three months in an empty school, Salmela massively upped his internet time. Lifehacker, OS X Daily, and Macworld were among his go-to sites, and he started to notice the ads more and more. He doesn’t remember what the pop-ups or autoplay ads were for, just that they were annoying. On Kickstarter, he’d backed a device called AdTrap that sounds a lot like Pi-hole: blocks ads at the network level, shows you what it blocked if you want to know. After using it for a while, “I thought, ‘I could probably make something better with a Raspberry Pi,’ ” he says.
Salmela spent the rest of the summer writing the code for Pi-hole. A few months later, Lifehacker wrote about it. He published a 6,000-word install guide on his blog, and Lifehacker wrote about it three more times. By fall 2015, Salmela was getting a lot of bug reports and feature requests and needed help. Luckily, he’d hosted Pi-hole on GitHub, a website that allows programmers to collaborate on code. GitHub is popular for open source projects, because it allows anyone to submit suggested changes, which means it also functions as a recruiting tool. Salmela began to assemble some extremely dedicated and like-minded volunteers.
“It really is a project of love for me,” says the first recruit, Dan Schaper, who claims to spend 50 hours to 80 hours a week on Pi-hole in addition to his work as a consulting network engineer. He wouldn’t divulge personal details, except that he lives on the West Coast and is obsessed with the pervasiveness of tracking. “I’m the tinfoil-hat guy of the group,” he said in an email.
Another Pi-hole developer, an Australian who consented to a video interview on the condition that he be identified only by his internet handle, WaLLy3K, is more of a scorched-earth type. He distinguished himself among Pi-hole fans by curating huge block lists of domainsassociated with ad servers, trackers, and malware. One of the other devs took to calling him “Mr. Insane Lists.” If you take all of WaLLy3K’s recommendations, you will find yourself blocking about 2.6 million domains. His objection to ads includes an aversion to “visual clutter” as well as a desire for privacy. He’s the kind of guy who will go up to a photographer at an event and ask that she not take his picture. “It comes down to consent,” he says. “I didn’t consent to giving out this information to people.” He estimates he spends 10 hours to 15 hours a week on Pi-hole.
Along with Drobnak in Rochester, Salmela’s other core comrades are Blayne Campbell in Canada, Adam Warner in the U.K., and the pseudonymous DL6ER in Germany. Together they put in a lot of late nights, mostly focused on talking new Pi-hole users through setup and the occasional bug. To support that labor, donors provide $1,000 to $2,000 a month. Salmela also collects an extra $20 to $100 a month through sales of Pi-hole T-shirts, hoodies, and mugs, and about $20 to $30 through affiliate links on the Pi-hole website, which pay a tiny commission when someone clicks on them.
This is obviously a pittance for a team of skilled developers putting in serious hours. After a user survey found that 69 percent thought Pi-hole was “worth paying for,” Salmela set up a one-time fundraiser asking for $100,000. He says he’d need $160,000 to $180,000 a year to truly support the project, including some full-time staff, but he’s been shy about asking for donations. “If your product is actually good, your consumers will sell it for you,” Salmela said in an email. “We have paid $0 in marketing and advertising, and look what we’ve grown into. It’s not easy and not currently sustainable, but it’s the way it needs to be done.”
It’s common for Pi-hole users to screenshot their stats (how many ads blocked, from where) and hold informal competitions with one another. Some wear T-shirts and drink from mugs emblazoned with replicas of the software’s colorful graphs; at least one displays his stats on his internet-connected mirror. “Get to see how many ads I’ve blocked any time I look in the mirror now!” he wrote. (It was 227.) Another asked the community how to set up an audio alert every time an ad is blocked in real time. “Could be super satisfying,” they wrote.
Pi-hole has attracted fans from mainstream tech companies such as Microsoft and AsusTek Computer Inc.—plus, its developers believe, the occasional self-loathing adman. In 2016 someone showed up in the team’s chat room with a carrier pigeon avatar, a number of sophisticated technical suggestions, and what sounded like inside information. “He was like, ‘You know, this isn’t going unnoticed,’ ” Drobnak says. “Or, ‘The advertising business, this is something that concerns them.’ ” The carrier pigeon was active for about two to three months, then disappeared.
The ad industry hasn’t taken any official shots at Pi-hole, likely because setup remains a significant barrier, says Jeremy Gillula, tech policy director at the nonprofit Electronic Frontier Foundation. Yet some 30 percent of the internet’s top 10,000 sites now use software designed to subvert browser-level ad blocking. Publishers will target Salmela’s software if it becomes anywhere near as popular as AdBlock Plus, says Nicole Perrin, an analyst at researcher EMarketer.
Still, the popular support for Pi-hole and other ad blockers may signal changes. The scandal around Cambridge Analytica, the political advertising firm that got hold of as many as 87 million Facebook users’ data and used the information to try to influence an election, shows that people still value privacy more than Mark Zuckerberg long claimed. In April, Democratic Senators Richard Blumenthal of Connecticut and Ed Markey of Massachusetts introduced a bill that would allow ad targeting only with users’ clear consent.
While the U.S. legislation has built little momentum so far, the EU’s privacy law, which affects any company doing business in Europe, is already changing the industry’s practices. The letter of the law requires publishers to get explicit permission from users to share data with every party that’s asking for data each time one of them makes a request. That could mean clicking “OK” perhaps hundreds of times just to get to one webpage, a status quo that would at least force the consolidation of data collection and reduce the ad industry’s data drain.
For publishers struggling to survive even with maximum ad surveillance, the Pi-hole team …