Over the last few years, the Recording Industry Association of America (RIAA) and other organizations looking to eliminate the illegal swapping of digital media files have attacked the problem through the courts, publicity campaigns, and other means.
But while they’ve managed to close down some peer-to-peer operations, and have successfully (and not so successfully) sued individuals who were uploading movies and music to the Web, there is one part of the Internet that has, until now, been operating under their radar: Usenet.
Usenet was once big — as big, in its day, as blogging is today.
In the 1980s, before the Web made the Internet the focus of everyone’s attention, Usenet tied the messaging and communications of local BBS systems into the distributed networking of the Internet. The result was a mass of user communities (called newsgroups) devoted to almost every conceivable topic, from software support to alien spacecraft.
But as Usenet nears 30, it has become, instead, the conduit for a rising tide of binary-file traffic that threatens to swamp the Internet. While it’s not easy to upload and download files from via the Usenet binary groups (large media files must be transferred in chunks and then stitched together again), savvy file exchangers with little respect for copyright law have found it a relatively safe place to operate.
All this activity isn’t only a copyright issue for ISPs. The resources taken up by large numbers of people uploading large numbers of files is significant — and one that many ISPs may no longer be able to ignore. In fact, in recent weeks, major ISPs have stopped providing open access to the hundreds of thousands of newsgroups distributed via Usenet.
These actions have been driven by New York State Attorney General Andrew Cuomo’s crusade against child pornography on the Internet. Cuomo’s actions, in turn, may have given ISPs an excuse to cut back on their increasingly costly support for Usenet.
Usenet isn’t one thing, but many
As with anything that has been around as long as Usenet, and been so important to so many people, there’s a great deal of folklore that’s grown up around it, reflected in terms like “Big 8” and “The Great Renaming,” “Netiquette” and “Never-ending September.”
Usenet’s technological underpinnings predate its association with the Internet, resting on dial-up-based store-and-forward e-mail BBS systems and UUCP protocols and programs.
Although its name makes it sound monolithic, Usenet is perhaps best described as a huge, loose collection of informal information-exchange communities that have little in common beyond their naming convention and their reliance on the Network News Transfer Protocol used to manage Usenet messages.
The basic unit is the newsgroup, a threaded discussion devoted to a topic. Newsgroups are organized by topic into hierarchies. Google Groups, which provides access to Usenet, lists more than 1,000 top-level hierarchies.
Many of these are named for a country or city, company or product. The Microsoft hierarchy, for example, includes 3,337 newsgroups, such as microsoft.public.mac.office.entourage, microsoft.public.scripting.vbscript and microsoft.public.outlook.calendaring.
Internet folklore says the first three newsgroups created in alt.* were alt.sex, alt.drugs and alt.rockandroll.
Nine of these hierarchies are particularly important. Eight of them (comp.*, humanities.*, misc.*, news.*, rec.*, sci.*, soc.* and talk.* — the so-called Big 8) are administered by a board of directors.
The Big-8 Management Board administers the creation of new groups and the structure of the hierarchies with an eye to maintaining order and promoting the usefulness of the groups. The Big 8 hierarchies include about 3,000 newsgroups.
The Big 8 (originally the Big 7; humanities.* was added later) were created in 1987, when the explosive growth of Usenet and the proliferation of newsgroups forced a reorganization that came to be called the Great Renaming. It systematized the names and structures of the newsgroups to make it easier for system administrators to manage the groups they carried.
The ninth major hierarchy, alt.*, was created as a protest to the Great Renaming, and was specifically intended to provide a less controlled alternative to the Big 8. Fittingly, Internet folklore says the first three newsgroups created in alt.* were alt.sex, alt.drugs and alt.rockandroll.
Usenet had already developed its own culture and standards, often referred to as netiquette.
“Back then Usenet was a conversation, and a fairly safe one. It was well behaved, pretty much everybody posted using their real names, and it was courteous,” says Steve Yelvington, principal strategist at Morris DigitalWorks, a subsidiary of publisher Morris Communications Co. in Augusta, Ga.
Back in the 1980s, Yelvington wrote software for his Atari ST so he could access Usenet and participate in a newsgroup for Atari owners. “Everybody read Brad Templeton’s rec.humor.funny joke every day,” he recalls.
Then came Never-ending September. Usenet had its origins on college campuses, which is where the computer networks were in the 1980s, and it went through an annual turmoil every fall as an influx of new student users learned the finer points of netiquette.
“When AOL connected its users to Usenet in 1993, there was a sudden flood of people who hadn’t read Emily Post,” recalls Yelvington.
Some Usenet veterans claim that the masses of new users who engaged in personal attacks, flame wars, fanboy diatribes and the like did permanent damage to Usenet standards of behavior — hence the feeling that September 1993 never ended.
The next year, 1994, Usenet civility suffered even more when a husband-and-wife lawyer team, Laurence Canter and Martha Siegel, hit Usenet with the notorious “Green Card Lottery” spam. This first recorded example of large-scale commercial Internet messages was soon followed by a flood of spam.
“A lot of people were driven away by that, and they never came back, despite the fact that we won the spam war and basically fixed that problem,” says Jeremy Nixon, a member of the Big-8 Management Board who formerly worked for newsgroup service provider Supernews.
Since 1994, of course, the Web has happened, and there has been less reason to go back.
“Usenet is old technology, and as such, it’s less pretty and it’s harder to use,” says Nixon. “You don’t just click on it in a Web browser, or if you do, it’s a lousy Google interface. Blogs, which are basically a re-invention of Usenet, look nice, and are easy to use and to get to.”
The Web eclipsed Usenet, but it didn’t extinguish it. Over the past decade, Usenet has gone in different directions. The Big 8 hierarchies have continued to march along the high road in a kind of comfortable obscurity, supporting communities of discussion in the Usenet tradition.
When Usenet was talked about at all, it was usually because of the alt.* hierarchy, which has acquired an unsavory reputation as a free market for digital content with a provenance of questionable legality or morality. Over time, Usenet has become a major distribution platform for pornography, illegal media and broken software.
Usenet in the spotlight
Usenet may have operated under the radar while the Web was in its ascent, but that changed this past June, when Cuomo announced a deal with Time Warner Cable, Verizon and Sprint to “eliminate access to child porn newsgroups … [and] purge their servers of child porn Web sites.” In July, he added AT&T, AOL and Comcast.
(Time Warner, Verizon, Sprint, AOL and Comcast are the top five ISPs in the U.S., with an aggregate share of more than 50 per cent of the market. Sprint is not an ISP, although it does provide Usenet access to some wireless customers. AOL was cited by Cuomo even though it stopped providing Usenet access for its customers in 2005.)
The actions taken by the ISPs in response have varied widely. Time Warner Cable quickly stopped offering Usenet access altogether. Other companies took a more surgical approach: AT&T announced it would cease offering its customers access to the alt.binaries.* hierarchy, while Sprint said it would drop the entire alt.* hierarchy.
Verizon went further, announcing that it would continue offering its customers access only to the so-called Big 8 hierarchies. Comcast hung back through the summer, but finally announced in late September that it, like Time Warner, would amputate Usenet access entirely.
However, even though Cuomo’s actions may have targeted child porn, there is some doubt as to whether this was the only thing motivating the ISPs to drop large portions of Usenet access. Significantly, while many of the responses to Cuomo’s child porn initiative involved curtailing access to Usenet, Cuomo never attacked Usenet, according to the Big-8 Management Board’s Nixon.
“Other providers were contacted by the attorney general’s office before any of this ever hit the news, and quietly complied with the request. The only reason it ever became a news issue was because some ISPs used it as an excuse to cut their Usenet service,” says Nixon.
Antiporn — or antibinary?
Although there has been speculation that equated the ISPs’ changes with censorship, Nixon doesn’t think the issue is one of free speech. It is very expensive to provide Usenet service, he says, and ISPs aren’t making any money on it. “When the New York Attorney General came knocking, they saw it as an opportunity to drop a money-losing service while shifting the blame elsewhere,” he says.
Today, there are two sides of Usenet: text and binaries. The text side of Usenet is dominated by the Big 8 and other widely read hierarchies (like k12.* for the education community).
But as active and important as text newsgroups continue to be, they represent only the smallest fraction of the resources devoted to Usenet. It’s the binaries — everything from pornographic images to illegally posted movies — that chew up most of the resources devoted to Usenet.
“Almost all of the resources devoted to Usenet are consumed by the binary groups and the users of those groups,” says Nixon. “If you said 99 per cent, you would still be understating it. However, if you further break down the binaries, the biggest usage of bandwidth is nonporn video content, followed by music and software. Pornography is a relatively small portion of it. Years ago, most of the binaries on Usenet were porn. Not anymore.”
This has resulted in a major change in the cost to ISPs of providing Usenet access to their customers.
“If you remove the binary newsgroups and keep the rest — and take care to not take feeds with misplaced binaries in them — the resulting text-only service could be run with modest resources,” says Nixon.
“Verizon’s Big 8-only service could probably run on one or two reasonable PCs. If you add in the rest of the non-Big 8 text groups on Usenet, it still wouldn’t add enough to require a more complex infrastructure. Even given the service availability requirements of a major ISP, you’d be talking about thousands of dollars [for a text-only service] versus millions for a service with all the binaries.”
Kathy Morgan, co-chair of the Big-8 Management Board, chimes in: “I seriously doubt if any of [the ISPs that have blocked or limited Usenet service] have cut their prices to make up for no longer offering news service. Any money previously spent on server maintenance and personnel is an increase in profits.”
A legal mess
Of course, ISPs aren’t the only means to access Usenet newsgroups. Because Usenet is actually a separate part of the Internet, it can be accessed through Newsgroup Service Providers (NSPs).
Curt Welch, who operates an NSP called NewsReader, offers another reason for why ISPs might want to exit the business of providing Usenet access.
“At the same time as the binary traffic has come to dominate Usenet, it has made Usenet a sticky legal mess,” he says. “It’s a slippery slope for ISPs. Today it’s kiddie porn, and tomorrow it may be the copyright people coming after you.”
But that doesn’t mean, says Welch, that there is a shortage of available access to Usenet. Many consumers who only need limited access to Usenet can use Google Groups, which provides free access to much of Usenet (though not the binary groups) and maintains an archive of messages that goes back to 1981.
If you want more than that, says Welch, “There are probably hundreds of providers, of which I’m just one.”
He explains that “on the text side, you can get access to all the great text content for $10 or  $20 a year, and that will continue.”
The binary side has become a very competitive business, says Welch, with the number of providers consolidating rapidly. In fact, many of the providers that had formerly sold bulk services to ISPs are now focusing on the consumer marketplace.
“A lot of people may have lost access to Usenet and just haven’t done enough research to find out how to get it back,” Welch says.
The Big-8 Management Board maintains a list of NSPs on its wiki. The AnchorDudes list makes a useful effort to track the mergers and combinations of trade names and services, according to Welch.
Because of the existence of these NSPs, the ISPs’ changes aren’t a major blow to Usenet, in Welch’s view. “A lot of people may have lost access to Usenet and just haven’t done enough research to find out how to get it back,” he says.
But others feel that the ISPs’ shift away from Usenet will likely have a negative impact on its future. Morgan says she expects the ISPs’ actions will “further marginalize Usenet in general.
A newbie who has never used newsgroups and doesn’t know what they are is not likely to care enough or know enough to search for a competent NSP to obtain access.”
Usenet may never again be what it once was. Still, if you look, you can find the old Usenet.
“Small groups are as active as they’ve always been,” Nixon says. “But the average age of the people posting is probably high, while the thing that keeps Usenet alive, what the younger people are doing, is binaries.”
If the ISPs’ actions really are driven by simple economics, it’s those younger Usenet users who will be most affected. And there is evidence that that’s the case.
“It is important to make the distinction that the ISPs are not blocking Usenet service from their users,” says Nixon. “They are simply choosing not to provide it themselves. They are not preventing their users from obtaining it elsewhere and using it via the Internet connections they sell.”
Slowing Usenet traffic or blocking access entirely may be a shoe that is yet to drop, but so far, that’s something the regulatory agencies aren’t permitting, as illustrated by the FCC’s recent decision prohibiting Comcast from throttling binary traffic (in this case BitTorrent) on its network.
With Usenet access readily available from NSPs, and with many ISPs still freely passing Usenet traffic, the text-based communities may be affected very little.
Usenet may be technology from the past, but despite the recent changes, it still has a future.
