Standard Mischief

Here I’m writing a big file full of zeros that I will, on my next step, mark as deleted. All the space that was taken up with that huge file will still exist, but as Gigabytes and Gigabytes of free space filled with zeros. It should all gzip down to a compact archive. I’m using the Linux command line to “Ghost” or “clone” an exact image of my laptop as a baseline for incremental backups. I had no idea I’d need to leave the thing all day though.

You can probably do almost any device I/O thing in the world with dd, but you have to be careful. Despite what anyone might tell you, dd stands for “destroy data”. Screw up the odd syntax and you’ll copy the factory fresh formatted partition all over top of your backup file.


root@StandardMischief:~# time dd if=/dev/zero of=/media/Preload/zerofile2 bs=1024 count=58970800
58970800+0 records in
58970800+0 records out
60386099200 bytes (60 GB) copied, 44091 s, 1.4 MB/s
.
real    734m51.051s
user    2m8.116s
sys    19m59.531s
root@StandardMischief:~#


Look at that speed! The SATA was running at a staggering 0.0109375 Gigabits/second, which is about 137 times slower than the max rate of 1.5 Gigabits/second for first generation SATA . Something is way wrong here.

So it took 12+ hours to create the file and only a second to erase it. I’m pretty sure at least part of the issue here is that I used a ridiculously small block size of 1 KB and did that 58,970,800 times in a row. We’ll see, as next run will use half of my RAM as a block size.

Update: Compression on the fly, with a larger block size


root@StandardMischief:~# time dd if=/dev/sdb1 conv=sync,noerror bs=512K | gzip -c > /home/backup/lappy486-20090905.img.gz
138767+1 records in
138768+0 records out
72754397184 bytes (73 GB) copied, 4765.18 s, 15.3 MB/s
.
real 79m25.198s
user 52m54.734s
sys 21m10.651s
root@StandardMischief:~#


So 15.3 (MB / s) = 0.11953125 Gb / s, which is both much faster and still a less than a tenth of advertised speeds. If this was my broadband connection I’d be on the phone to customer support by now.

I see I did have an error for the second run. I used a blocksize of half a MB instead of half a GB ( bs=512K instead of bs=512M). Well that’s something to try for next week.

If you’ve ever had to hand carry a package to the FedEx drop-off point and you have half a brain you’ll know that as soon as the staff scans the barcode on the label, the package becomes their responsibility. You would also assume that if someone dropped off an item and it somehow failed to be scanned, and that package dropped into a black hole, FedEx would assume that such a package never even existed. Therefore if the package is especially time-sensitive (like perhaps you are willing to spend a bit of scratch to get it there overnight), or valuable, you’d probably want to make sure that the staff actually scan the package before you let it out of your sight. Unfortunately at all the drop-off locations I frequent, this is harder than it looks. In fact, in my experience none of these work:

Hi. Could you scan my package?
Would you scan my package please?
Please scan my package. No, I don’t mind waiting my turn.
I need you to scan my package first before I leave.
No, don’t get up out of your chair. Just lift that wireless scanner that is by your right hand, point it in the general direction of this very important package that I’ve placed right here in front of you on the scale, and pull the trigger.

I’ve had a package that I insured for several thousand dollars that had to go out that night for an early AM delivery. Because of the hour, I drove the package right to the airport’s staffed FedEx point and got into a shouting match with fscking asshole behind the counter because he did not care to scan the package in front of me. Then a “manager” swung by and told an obvious lie to cover for his minion. By that time I was on the phone with FedEx. I asked if there was any special magic words I needed to say to get the hired help to do their jobs without making the whole deal a pain in the ass. They were no help.

On a different trip, I discovered the secret. Someone in front of me in line said the special magic words, which are as follows:

I need a receipt for this package.

That’s it. They need to scan the package to give you a receipt. Absolutely no stress, no issues, and no attitude.

While comments are open, I’d especially like to invite comments from those who work at FedEx, either as management/customer service/public relations employees, or front line workers. Maybe there’s another side to the story on the “i don’t wanna scan the packages in the presence of my customer” issue.

Just a little something I whipped up one night before the long drive home. Yea, after losing my 5 minute commute, I’m starting to get into pre-recorded talk shows. I’d have started this post a little differently, but 2003 called and they wanted their podcasting howto post back…

Anyway, here it is:

curl -LO $(curl "http://www.blogtalkradio.com/gunnuts.rss"|egrep -o "http://[^ ]*mp3"|uniq|head -1)

The first curl command uses the L option to allow the request to be redirected with a 3XX response code. Blogtalkradio.com does this.

The O option just assigns a filename based on the original rss feed URL (sans path). This works well unless you get someone clueless who names their files like this:

I’m not going to explain the regex and remainder except to say that the -1 can be changed to a higher number to get more shows unless everything goes foobar because Karl named his files with exactly the same name.

Oh, and the files just end up in your current working directory, and you’ll have to get out of it if you want to cleanly unmount your flash card, but you knew that already.

You’ll need to get curl though. N00buntu users could use something like

$ sudo apt-get install curl

but if they really are n00bs they’ll probably use the GUI tool to get the job done, and then never touch the command line and get their podcasts using Amarok or something. Nothing wrong with that, it’s just that you are not the audience I’m writing for. The audience I am writing for will find their way here using the “search engine friendly” URL.

Oh and since I’m new to this podcasting stuff, I have to ask. With the proliferation of mp3 players like the iPod shuffle that don’t have a display, shouldn’t we be starting our broadcasts with “gun nuts #54 for May 27th” instead of “Goooood evening ladies and gentlemen…“? That would allow a user to shuffle through a bunch of shows to easily find the one he has not heard yet. Much nicer if the shows don’t all sound the same until a half a minute in.

Thanks for making my commute a bit more interesting, everyone.

I did the BWI loop trail the other day. It’s detailed here and elsewhere. Because working at the startup really cuts into my workout time, I had the same second-day nausea and loss of appetite from having a hugely calorie deficient bike ride. With regular exercise this does not happen, but you gotta bike and bonk a bit after slacking off.

I’m trying to get some motor-vation today but all I can manage is this post. The problem with the first link above is that there’s no data that would let you know how old the info is. Blogs, or at least mine are immune because every post is dated.

I did this trip about a year ago, and I’m happy to report that someone has put up signs so it’s a bit easier to keep on the loop. I’ve felt the urge before to spray-bomb the entire loop in orange marker paint. Still, in my GPS notes that follow, I’ve tried to mark out the major turns to keep you on track.

It was a nice day yesterday. The average “serious” speed cyclists on the trail were as rude as ever (I swear, it’s like the bastards think you are fscking Lance Armstrong and the only way they can beat you is to sneak up and pass you silently and suddenly). These idiots are probably only eclipsed by the typical Young Hippie Bicycle Fanatic (of Critical Masshole fame). Motto: “share the fucking road except for those god-damn pedestrians!”

Now you know why I like to keep off the beaten track.

Anyway, I ought to make a clue sheet or something. Tough. Assume the following is going around the loop clockwise:

39.16260,-76.66355 – Start and end point. Parking. Good plane-spotting area.

39.15938,-76.68412 – A 7-11 with gas pumps. All the usual stuff here but be extra careful crossing the high-traffic street. Better yet, stop here first, before parking at the trailhead.

39.17782,-76.69032 – Left turn here

39.17822,-76.69305 – Right turn here

39.19095,-76.69146 – Left turn here

39.19035,-76.68082 – Bear left. Do not bear right and cross the bike-bridge over Aviation Loop unless you want to do the dead-end spur that goes to the main terminal.

39.19132,-76.68034 – If you want, you can go off the trail at this point, and go left on Elkridge Landing Road for 550 feet, (there’s a trail on your side of the road, no need to cross), to a McDonalds (food, free refills, tap for water at the soda fountain) You are about half-way around the loop at this point.

39.19349,-76.67067 – Bird watchers might see an Emu or two behind the chain link fence.

39.19389,-76.66678 – Not a lot of plane action, because of a less popular runway, but otherwise an outstanding overlook.

39.19626,-76.66509 – Farm. Horses. Sculpture in stainless steel.

39.19478,-76.65619 – Rail track crossing

39.16141,-76.64547 – Decision time. If doing the loop clockwise, veer off to the right at this point. If going on to the Baltimore and Annapolis Trail, then go ahead and cross the bridge across I-97 (a/k/a Glen Burnie Bypass).

If you’ve done any bash shell scripting, you may have come across the need to write data to a temporary file. However, a well-written script should take in the possibility that it will be run more than once at the same time, (the unices being multi-tasking and all that), so you will want to make sure that different processes don’t all write the same data to the same hard-coded /tmp file.

One of the standard ways to do this is to use the bash built-in variable “$$“, like so:

echo $data-to-log >> /tmp/"$$"log.file.txt

And that usually works fine, as long as only one person is running that script per login account. You will, however, have issues if you run the script from the console at the same time that the script is run from your crontab. Ask me how I know.

The common misconception here is the idea that the bash built-in variable “$$” holds the PID (process ID) of the script itself. That is not true. It’s actually the PID of your bash shell. Therefore running your carefully crafted script more than once at the same time can fsck-up your script output.

A better way, and my standard mischief, is to do the following. First, somewhere up near the top of your shell script create a variable that is unique, like so:

uq=$(date +%s-%N)

backtick people can use this:

uq=`date +%s-%N`

but I like to use the “$()” out of habit because it makes it easier to nestle the commands. Besides, after too much caffeine and staring at a LCD all day, those backticks can be a bit hard to see.

Then, we use this variable in place of “$$“, like so:

echo $data-to-log >> /tmp/"$uq"log.file.txt

What we are doing here is running the date command, and having it show the number of seconds since the Unix epoch back in 01jan1970. Then, we add a dash and the number of nanoseconds. We assign the results of that command to the variable “$uq“. The end result is that unless the two scripts run this command at exactly the same time (probably impossible even if you were trying), the variable $uq is going to be a wholly unique number. When you use this unique number in a filename you can be pretty sure you won’t be clobbering existing data by overwriting an existing file.