Aria2 and GNU Paste: The perfect team

I have been having internet speed problems lately (as always :P), which kept me from updating my Arch Linux system. Today I decided to go ahead and update the system anyway, only to find that the download speeds were excruciatingly slow. As always, whenever I need to update my system with a bad internet connection, I fall back to aria2.

Downloading from multiple servers with aria2

Somehow, I was getting decent speeds with concurrent outbound connections, so I decided to use multiple mirrors to download the packages. aria2 supports downloading a file from multiple sources (and protocols!) at the same time.

So, I decided to generate three download lists using three of my favourite mirrors. I first did:

# pacman -Su --print > mirror1.list

And, after editing pacman's mirror file: /etc/pacman.d/mirrorlist, I generated mirror2.list and mirror3.list in the same fashion.

The problem at hand

Now came the real problem. When using an input file with aria2 (as I was about to), aria2 expects all the sources for a file to be specified on the same line and separated using a <Tab> character.

So, I had three input files containing urls, and they needed to be merged linewise in the above mentioned fashion.

Here's a little illustration of what was needed, download.list being the final file containing all the urls:

mirror1.list
------------

line1: url-11
line2: url-12
line3: url-13

mirror2.list
------------

line1: url-21
line2: url-22
line3: url-23

mirror3.list
------------

line1: url-31
line2: url-32
line3: url-33

download.list (the urls should be separated using a literal <Tab>)
-------------

line1: url-11  url-21  url-31
line2: url-12  url-22  url-32
line3: url-13  url-23  url-33

I had 60 urls per file, and merging them by hand was going to be a real chore :P. Also, I am not too fond of real tabs.

Where does GNU Paste fit in

After searching the net, I found this very helpful link: http://stackoverflow.com/questions/3806874/how-to-concatenate-two-files-line-by-line-using-bash.

As you can see, paste, a part of GNU Coreutils does exactly what I wanted to do! Its man page reads:

Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.

Note: You can use delimiters other than <Tab> with paste as well. Read its man page for details.

So, I simply did:

$ paste mirror1.list mirror2.list mirror3.list > download.list

and the resulting file was in the exact format that I needed! No effort required, period.

Finally, using:

$ aria2c -i download.list

I got much better download speeds, as I had expected.

Thanks to aria2 and paste, I had a great experience, and they make a great team. As always, aria2 proved to be a trustworthy companion and paste was a great new find. GNU Coreutils always has something surprising up its sleeve.

Comments

Comments powered by Disqus