hypergeometric: a place for me

One of the benefits of the pandemic for me personally is that I was able to uninstall several work apps from my phone. There are a few things that made this possible, the first of which is I moved into an individual contributor position, and the second of which was that I was not longer and on-call employee. The strange thing about doing this was it immediately has improved my quality of life, especially when I was away from work. Its made me think a bit more if I’ve ever truly needed them.

The “productivity” things like slack, email, and calendar on my phone have brought me, cost me my ability to focus. It made me much more reactive, since now I was bombarded with notification, and constant interruption for a quick question. These interruptions prevented me from being present in meetings, stopped me from thinking deeply, and impacted my personal and professional relationship. Without them, I have clear boundaries, and I feel I’ve regained a much better work routine.

I’ve wondered why I ever configured them in the first place. I felt compelled to install them like somehow it would improve my performance, or I had to demonstrate constant availability.

I wonder if I’ll install them when we head back to the office.

So, if you haven’t seen this page you should take a look. It has a whole bunch of interesting techniques you can use to manipulate strings in bash. If you end up working with bash a lot you might find yourself doing this quite a bit, since it can save a lot of time.

Lets take a pretty typical task, stripping the domain off of an email address.

So this poorly written program will split an email address at the first @ and print the first portion:

Its counterpart, which does not fork, uses a bash built-in to remove everything after the @:

So, whats the execution difference?

Its a 100x faster to skip the fork.

Now, granted this is a pretty dumb example, and its easy to rewrite this to perform better than the bash example (i.e. don’t use a loop and just use awk which is 3x faster than the pure bash solution). So, think about what your doing, use a pipe over a loop, and if you can’t do that, try to find a built-in that can take care of your needs.

Two colleagues of mine ask a very similar question for interviews. The question is not particularly hard, nor does it require a lot of thought to solve, but its something that as a developer or a ops guys you might find yourself needing to do. The question is, given a log file of a particular format, tell me how many times something occurs in that log file. For example tell me the number of unique IP addresses in an access log, and the number of times each IP had visited this system.

Its amazing how many people don’t know what to do with this. One of my peers ask people to do this using the command line, the other tells the candidate they can do this anyway then want. I like this question because its VERY practical; I do tasks like this everyday, and I expect the people I work with to be able to do.

A More Concrete Exmaple

I like the shell solution, because its basically a one liner. So lets walk through it using access logs as an example.

Here is a very basic sample of a common access_log I threw together for this:

Lets say you want to count the number of times a unique IP addresses who’ve visited this system. Using nothing more than awk, sort, and uniq you can find the answer. What you’ll want to do is pull the first field with awk, then pipe that through sort, and then uniq. This isn’t fancy, but it returns the result very quickly without a whole lot of fuss.

Like so:

This gives you each hostname or IP, and the number of times they’ve contacted this server.

Upping the Complexity

Now for something more complex lets say you want to get the most commonly requested document that returns a 404. So, again we can do this all in a shell one-liner. We still need awk, sort, uniq, but this time we’ll also use tail. This time we can use awk to examine the status field(9), then print the URL field(7) if the status returned was 404. We can then use sort, uniq, and sort to order the results. Finally we’ll use tail to only print the last line, and awk, to print the requested document.

So here is what this looks like:

Of course there are many other ways to do this. This is a totally simple way to do it, and the best part of this is that you can count on these tools being on almost every *nix system.

Follow @gpapilion