I have been having internet speed problems lately (as always :P), which kept me from updating my Arch Linux system. Today I decided to go ahead and update the system anyway, only to find that the download speeds were excruciatingly slow. As always, whenever I need to update my system with a bad internet connection, I fall back to aria2.
Downloading from multiple servers with aria2
Somehow, I was getting decent speeds with concurrent outbound connections, so I decided to use multiple mirrors to download the packages. aria2 supports downloading a file from multiple sources (and protocols!) at the same time.
So, I decided to generate three download lists using three of my favourite mirrors. I first did:
# pacman -Su --print > mirror1.list
And, after editing pacman's mirror file:
/etc/pacman.d/mirrorlist, I generated mirror2.list and mirror3.list in the same fashion.
The problem at hand
Now came the real problem. When using an input file with aria2 (as I was about to), aria2 expects all the sources for a file to be specified on the same line and separated using a <Tab> character.
So, I had three input files containing urls, and they needed to be merged linewise in the above mentioned fashion.
Here's a little illustration of what was needed, download.list being the final file containing all the urls:
mirror1.list ------------ line1: url-11 line2: url-12 line3: url-13 mirror2.list ------------ line1: url-21 line2: url-22 line3: url-23 mirror3.list ------------ line1: url-31 line2: url-32 line3: url-33 download.list (the urls should be separated using a literal <Tab>) ------------- line1: url-11 url-21 url-31 line2: url-12 url-22 url-32 line3: url-13 url-23 url-33
I had 60 urls per file, and merging them by hand was going to be a real chore :P. Also, I am not too fond of real tabs.
Where does GNU Paste fit in
After searching the net, I found this very helpful link: http://stackoverflow.com/questions/3806874/how-to-concatenate-two-files-line-by-line-using-bash.
As you can see,
paste, a part of GNU Coreutils does exactly what I wanted to do! Its man page reads:
Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.
Note: You can use delimiters other than <Tab> with paste as well. Read its man page for details.
So, I simply did:
$ paste mirror1.list mirror2.list mirror3.list > download.list
and the resulting file was in the exact format that I needed! No effort required, period.
$ aria2c -i download.list
I got much better download speeds, as I had expected.
Thanks to aria2 and paste, I had a great experience, and they make a great team. As always, aria2 proved to be a trustworthy companion and paste was a great new find. GNU Coreutils always has something surprising up its sleeve.