I've written, used, and seen a lot of loose scripts in my career. They start with someone that needs to semi-automate some task. After a while, they grow. They can change hands many times in their lifetime. I've often wished for a more command-line tool-like feeling in those scripts. But how hard is it really to bump the quality level from a one-off script to a proper tool? It turns out it's not that hard in Python.
In this article, I start with a little Python snippet. I'll drop it into a
scaffold module, and extend it with
click to accept command-line arguments.
All non-inline code samples in this article refer to a specific version of the code you can find at https://codeberg.org/ofosos/rotoscope. Every commit in that repo describes some meaningful step in the course of this how-to article.
This snippet does a few things:
- Check whether there are any text files in the path specified in
- If it exists, it creates a new filename with the current timestamp and moves the file to
- Delete the current
ARCHIVE/latest.txtlink and create a new one pointing to the file just added
As an example, this is pretty small, but it gives you an idea of the process.
Create an application with pyscaffold
First, you need to install the
tox Python modules.
scaffold, change to the directory where the example
rotoscope project resides, and then execute the following command:
Pyscaffold overwrote my
README.md, so restore it from Git:
Pyscaffold set up a complete sample project in the docs hierarchy, which I won't cover here but feel free to explore it later. Besides that, Pyscaffold can also provide you with continuous integration (CI) templates in your project.
- packaging: Your project is now PyPi enabled, so you can upload it to a repo and install it from there.
- documentation: Your project now has a complete docs folder hierarchy, based on Sphinx and including a readthedocs.org builder.
- testing: Your project can now be used with the tox test runner, and the tests folder contains all necessary boilerplate to run pytest-based tests.
- dependency management: Both the packaging and test infrastructure need a way to manage dependencies. The
setup.cfgfile solves this and includes dependencies.
- pre-commit hook: This includes the Python source formatter "black" and the "flake8" Python style checker.
Take a look into the tests folder and run the
tox command in the project directory. It immediately outputs an error. The packaging infrastructure cannot find your package.
Now create a Git tag (for instance,
v0.2) that the tool recognizes as an installable version. Before committing the changes, take a pass through the auto-generated
setup.cfg and edit it to suit your use case. For this example, you might adapt the
LICENSE and project descriptions. Add those changes to Git's staging area, I have to commit them with the pre-commit hook disabled. Otherwise, I'd run into an error because flake8, Python style checker, complains about lousy style.
It would also be nice to have an entry point into this script that users can call from the command line. Right now, you can only run it by finding the
.py file and executing it manually. Fortunately, Python's packaging infrastructure has a nice "canned" way to make this an easy configuration change. Add the following to the
options.entry_points section of your
This change creates a shell command called
roto, which you can use to call the rotoscope script. Once you install rotoscope with
pip, you can use the
That's that. You have all the packaging, testing, and documentation setup for free from Pyscaffold. You also got a pre-commit hook to keep you (mostly) honest.
Right now, there are values hardcoded into the script that would be more convenient as command arguments. The
INCOMING constant, for instance, would be better as a command-line parameter.
First, import the click library. Annotate the
rotoscope() method with the command annotation provided by Click, and add an argument that Click passes to the
rotoscope function. Click provides a set of validators, so add a path validator to the argument. Click also conveniently uses the function's here-string as part of the command-line documentation. So you end up with the following method signature:
The main section calls
rotoscope(), which is now a Click command. It doesn't need to pass any parameters.
Options can get filled in automatically by environment variables, too. For instance, change the
ARCHIVE constant to an option:
The same path validator applies again. This time, let Click fill in the environment variable, defaulting to the old constant's value if nothing's provided by the environment.
Click can do many more things. It has colored console output, prompts, and subcommands that allow you to build complex CLI tools. Browsing through the Click documentation reveals more of its power.
Now add some tests to the mix.
The test sits in a method of a testing class. Most of the conventions follow what I'd use in any other Python project very closely, but there are a few specifics because rotoscope uses
click. In the
test method, I create a
CliRunner. The test uses this to run the command in an isolated file system. Then the test creates
archive directories and a dummy
incoming/test.txt file within the isolated file system. Then it invokes the CliRunner just like you'd invoke a command-line application. After the run completes, the test examines the isolated filesystem and verifies that
incoming is empty, and that
archive contains two files (the latest link and the archived file.)
To execute these tests on my console, run
tox in the project's root directory.
During implementing the tests, I found a bug in my code. When I did the Click conversion, rotoscope just unlinked the latest file, whether it was present or not. The tests started with a fresh file system (not my home folder) and promptly failed. I can prevent this kind of bug by running in a nicely isolated and automated test environment. That'll avoid a lot of "it works on my machine" problems.
Scaffolding and modules
This completes our tour of advanced things you can do with
click. There are many possibilities to level up a casual Python script, and make even your simple utilities into full-fledged CLI tools.