A consensual stance towards the shell scripting language is that this tool is not suited for the preparation of complex, reliable and evolutive software. Statements according to which shell scripts should be replaced by programs written in a more capable language can frequently be heard but these replacement programs are rarely seen.

We should rethink this consensus because it denies reality. Is the shell scripting language a difficult, idiosyncratic and occasionally obscure language? It certainly is, but there must be many reasons why shell scripts are still in use today. Actually “still in use” is a bit of an understatement: we do not see shell scripting slowly fading away and being replaced by some other more capable tool. Instead, what we see is a steady stream of new shell scripts being written today. We do not see shell scripts surviving in some old arcanes and dusty grymoires, instead we see the application realm of shell scripts expanding. A subtle but strong sign of the relevance of shell scripting in today’s IT-industry landscape is the mention of ShellCheck in the trial section of ThoughtWorks tech radar in october 2020.

It is therefore important to any team active in the IT-industry today to cultivate their shell scripting skills and we present here evidences supporting this claim before providing useful learning material in later contributions.

Summary

Shell scripts application realm expanded in the last decade

This movement is parallel to the continuous development of free software production and the adoption of convenient virtualisation and cloud-computing technologies. The related application fields where shell scripting is widely used today are:

  • Distribution of free software and curl-su-isms – Many free software projects with a large audience are using and popularising the approach of providing installation instructions that can be curl-ed and piped through su. Some of these projects are NodeJS or PHP Composer. This practice opens strong security risks but nevertheless enjoys some popularity.

  • Preparation of Virtualised Environments – The popular container platform Docker is using so-called Dockerfiles to define the initial contents of a container’s file system, and this Dockerfile is mostly a succession of shell commands. These commands can and should call shell scripts if they must perform tasks outsmarting the expressivity of one-liners. Similarly, Vagrant virtual machines or systems provisioned in a cloud infrastructure are usually initialised with a shell script prepared by the provisioner.

  • System Configuration Managers – Systems like Ansible are all providing an interface to the shell, so that it is easy for the user to add customised steps to the system which are not well covered by off-the-shelf functionality. (I once wrote a more complete comparison of Ansible vs. shell-scripting.)

  • Customisation of Interactive Shells – The “Oh-my-zsh” effort is one of the signs showing a vivid activity in the realm of interactive shell customisation. Of course, that customisation involves writing shell scripts.

Shell scripts are still widely used in their historic field of application

Shell scripts have a long going record of adequacy in several historic applications. This list is not exhaustive.

  • Starting and stopping daemons on Unix systems – Writing shell scripts to perform this task and related housekeeping activities remains popular even on Linux systems in the age of systemd.

  • Building software with make or similar systems – Makefiles are really shell scripts in disguise and virtually every complex build system on a Unix system relies on shell scripts at some stage. Building the Linux kernel or FreeBSD universe are perfect example for this.

  • Portably installing software – Shell scripts are used for this by some major software distributions like TeXlive, Maple, VirtualBox or Python Anaconda among others.

Shell scripts fill new space as it is uncovered

Rather than making here reckless generalisations I would rather share the experience of my colleague Philip Peter who could see in a few IoT-projects the important and pervasive usage of shell scripts supporting the software development lifecycle:

  • Building complex applications – Compiled languages and especially C and C++ being the most prominent. This means usually building the application is a multi-step process, often with at least some Make and CMake involved. This multi-step process usually ends up being wrapped in a small shell script

  • Setting up cross-compilers – It requires a bit of setup to cross-compile for different architectures, and this is easily wrapped in some shell script.

  • Setting up virtualised environments – It is common to use virtual machines for testing or to replace cross-compilation. While Vagrant can be configured and extended with Ruby provisioning a virtual machine often ends up chaining a lot of shell commands for the setup.

  • Setting up and running tests on a target device – This activity usually requires a list of invocations of different command line utilities.

According to security experts from Pentest Partners, the firmware update system for Tesla cars also used shell-scripts to function.

Shell scripts are a actually good choice for these activites

Now that we are confident that in many contemporary situations the shell scripting language is still nowadays preferred over more modern and capable languages we need to understand the unique characterstics of the shell scripting language that balance this choice.

One of the reasons for this preference is that the shell is one of the few pieces of software which are guaranteed to be there on any Unix system, with no major differences from one version to the other. (In my experience most differences or bugs distinguishing implementations or versions of the same implementation are to be found in the lexer-parser or in job management.)

Another often overlooked reason is that the shell scripting language, while being crude, is superficially simple. It is one of the few languages in which many people interacting with Unix systems gain at least a rudimentary literacy. There is little or no other language from which this could be said and none of the shell-replacement pretenders that are regularly mentioned can stand the competition on this point: wether it is TCL, Perl, Python, Ruby or JavaScript/NodeJS, all these languages require much more language knowledge to get easy things done than the shell does. It is trivial to recycle an interactive command into a shell-script: we just need to save that command in a file. No language is challenging this simplicity.

The generalised use of shell scripts in contemporary build and delivery systems should be understood from the perspective of the DevOps transition. Many integrated development environments come with systems able to coordinate the build of very complex software solutions. The generalised use of shell scripts in contemporary build and delivery systems rather than these “off-the-shelf” solutions is a tangible artefact showing how important it is for delivery teams to define and adopt their own workflows, protocols and conventions. These “off-the-shelf” solutions might be useful in the simplest and easiest cases but are unable to adapt to changing requirements of delivery teams as fast as shell scripts can: off-the-shelf solutions are only good to solve off-the-shelf problems.

The fourth and last point we should observe is that no other language has a really convenient interface to start external programs. Most languages expose functions reminiscent of the system(3) ISO C function or the exec(3) POSIX C functions but none of them are hardly as efficient as the shell language itself to describe complex workflows mostly based on external programs. Some would say that the shell language is a domain specific language, the domain being Unix tasks and this statement helps us to understand why the shell is here more successful and convenient than many advanced programming languages. (To alleviate this situation I’have written a library called Rashell for OCaml and Common Lisp.)

We should cultivate shell scripting skills in our teams

To support this cultivation, I am preparing a series of resources to be published on this blog. Here is a glimpse at my writing pipeline:

  • A shell language primer
  • Writing and testing functions in shell scripts
  • Extract, transform and load in shell scripts
  • Using temporary files and directories in shell scripts
  • Dynamic programming techniques in shell scripts
  • Resources and methods for shell scripting self-studies