tarsum: checksum utility for tar files

description

tarsum generates and validates checksums for files encapsulated within common archive file formats. It supports a format specification grammar based on the BSD stat(1) utility for specifying the checklist and reporting formats. This makes it easy to interoperate with different checksum utilities (e.g. GNU coreutils md5sum or OpenBSD sha256), which utilize different line formats. This also effectively provides a way to programmatically query file metadata from a shell without expanding the archive.

Why? Because for various reasons I prefer tar for archiving data. But examining tar archives can be slow, especially on media like Blu-ray discs. In most cases what I'm attempting to do is compare archived content against manifest files of the source file trees generated by common checksum utilities, and to do so in a way that is easily scriptable.

news

2022-04-07

Change checklist report status word "MISSED" to "MISSING", matching the OpenBSD sha256(1) spelling as intended when checklist verification was originally committed two days ago. Other utilities (e.g. GNU coreutils sha256sum, Perl shasum) seem to report missing files as failures ("FAILED").

2022-04-06

Add -R verification report format specification option.

2022-04-05

Add -C checklist verification option.

2019-08-16

Project page.

todo

Support POSIX pax listopt format strings in addition to (or instead of) BSD stat(1) format strings. I didn't realize pax had such a feature until after I'd written the existing format parser. I have half a mind to simply submit patches to various pax implementations to support SHA-2 and SHA-3 checksums, though at least for BSD pax I'd also need to submit support for listopt itself, which is lacking.

Implement tarmap, a utility to index tar files and compressed tarballs for random access and file extraction. Compression formats are blocked based so an index can simply map block boundary offsets to the position in the uncompressed stream that begins at that block. tarsum could either use a map directly or build one internally when generating manifests; offset mappings could be printed with new format specifiers. Ideally the generated mappings would be simple enough to be parsed by shell scripts and used to extract individual files using regular utilities like dd(1), gzip(1), xz(1), etc, without having to build or install tarsum or tarmap.

usage

Usage: tarsum [-0a:C:f:R:s:t:h] [TARFILE-PATH]
  -0          use NUL (\0) as default record separator
  -a DIGEST   digest algorithm (default: "SHA256")
  -C PATH     checklist for verification of archive contents
  -f FORMAT   format specification (default: "%C  %N%$")
  -R FORMAT   verification report format (default: "%N: %R%$" )
  -s SUBEXPR  path substitution expression (see BSD tar(1) -s)
  -t TIMEFMT  strftime format specification (default: "%a %b %e %X %Y")
  -h          print this usage message

FORMAT (see printf(1) and BSD stat(1))
  \NNN  octal escape sequence
  \xNN  hexadecimal escape sequence
  \L    C escape sequence (\\, \a, \b, \f, \n, \r, \t, \v)
  %%    percent literal
  %$    record separator (e.g. \n or \0)
  %A    digest name
  %C    file digest
  %N    file name (full path)
  %R    verification status (OK, FAILED, MISSING)
  %T    file type (ls -L suffix character; use %HT for long name, %MT for single letter)
  %g    GID or group name (%Sg)
  %m    last modification time (%Sm: strftime formatting)
  %o    file offset (%Ho: header record, %Lo: end of last file record)
  %u    UID or user name (%Su)
  %z    file size (%Hz: header record(s), %Lz: header and file records)

Report bugs to <william@25thandClement.com>
		

contributions

I originally threw together tarsum.c in a few hours. Some parts are organized as-if they were to be built as a shared library, others assume a simple binary.

The format specification grammar should be straightforward (tastes notwithstanding) for someone with experience hacking on common Unix software providing similar facilties (e.g. libc). Most of that logic is kept encapsulated in the `print` routine. What's somewhat unique is that for parsing checklists the format specification is translated into a regular expression, a task performed by the `format2re` routine.

license

Copyright (c) 2017, 2019, 2022 William Ahern

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

source

git clone https://25thandClement.com/~william/projects/tarsum.git

Or visit the GitHub mirror

other projects

airctl | bsdauth | cnippets | libarena | libevnet | authldap | streamlocal | libnostd | zoned | dns.c | delegate.c | llrb.h | lpegk | json.c | cqueues | siphash.h | hexdump.c | timeout.c | luapath | luaossl | lunix | phf | runlua | autoguess | tarsum | prosody-openbsd | AnonNet