Adam Prescott

Sending huge files with torrents

I had to send and receive a lot of photos recently. At least a few hundred megabytes in total, and it was far too painful trying to get them all transferred. So I made copyfile.net.

I’ve faced this type of problem repeatedly, and it’s never been enjoyable. I don’t think I’m alone, either. Two main options I think people go for are:

  1. Through an instant messaging system.
  2. A large file hosting site à la Megaupload.

(I hope you aren’t now thinking, “Why not just email them?”)

For one-off file transfers, file sharing sites feel wasteful. Twice the filesize has to be used in bandwidth, first for the upload, then the download.

The instant messaging solution — which is the one I’ve used most often — can sometimes be fine, but past a point, the amount of data makes it cumbersome. If all the files go into an individual zip file, chances are pretty good that the transfer will die before the file finishes, at least in my experience. If you bundle them into groups of 10 or 20 (manually — do most people even know splitting files into multiple archives exists, let alone use it?), that’s a lot of time wasted checking transfers have finished before you can start the next one, and the transfer can still die.

Torrents have solved this problem already, of course. They deal with transferring huge files, connection interruptions, and even rate-limiting in torrent clients. So it should be a no-brainer, right? Why reinvent the wheel?

Except, it’s not so simple. The primitives of creating and sharing torrents are too technical for the average person. If you search the web for how to send huge files privately, you arrive at articles and how-to pages for creating a torrent file. But even if you learn how to do that, where do you go from there? How do you share the torrent so that it’ll work? Do you use a tracker? If you don’t use a tracker, how do you get the correct node information into the torrent file so that you can do it all tracker-lessly?

There should be a very simple set of requirements to transfer a large amount of data from person A to person B: nothing more complicated than a file sharing site would ask of you. If it uses torrents, then there should be no setup required except for the client you’ve already got. Because, again, why reinvent the wheel?

If this is done with torrents, then the transfers are peer-to-peer, which saves bandwidth and avoids having to trust a file hosting site, and they’re much smarter about resuming interrupted downloads.

Solving it

So it’s this problem that I went about trying to solve a few weeks ago, mostly for fun and my own occasional use. Privately sharing a large amount of data with a friend, without needing to install a custom file sharing application, without any registration, and with no setup. The end result is copyfile.net.

Technically, it’s really simple: a single-page site that uses the HTML file APIs through a drag-and-drop interface to produce a torrent file that you can share with someone else through a link. Both of you add the file to your existing torrent client, then just wait for it to start transferring.

Behind the scenes there is a custom torrent tracker running which periodically purges the peer lists for each torrent, as well as the torrent data itself. This is simply to try and enforce the principle that this should it’s really about personal one-off file transfers, and not a free-for-use open tracker. I’ve got a few plans to make this very strict using some kind of authentication mechanism, to keep transfers genuinely one-to-one.

Some downsides

There are some genuine problems, though.

Adding the torrent file to your client as the sender means you have to set the location of the file you’re trying to send, so your torrent client can seed it. This isn’t too intuitive, but once you get the hang of it, it’s simple.

Since torrenting has become centered around the idea of keeping a good ratio, your client will almost definitely record the leech as part of the global ratio, even though it doesn’t have much meaning in this particular case.

Having gone through a few transfers using copyfile.net, it tends to not really be clear whether something has finished transferring. Short of writing a custom torrent client to handle this single use case, though — which would defeat the point — it’s hard to avoid.

Even if I’m the only one who uses it, it was fun and I got a caching library out of it.