Implementation:MaterializeInc Materialize Shlib Bash

Knowledge Sources	MaterializeInc_Materialize
Domains	Build_System, CI_CD, Shell_Utilities
Last Updated	2026-02-08 00:00 GMT

Overview

A shared Bash utility library providing common shell functions used across Materialize build scripts, CI pipelines, and developer tooling.

Description

shlib.bash is a foundational shell library sourced by numerous scripts throughout the Materialize repository. It provides error handling (die), verbose command execution (run), version comparison (version_compat), Git helper functions (git_files, git_empty_tree), CI/Buildkite integration helpers (try, try_status_report, ci_collapsed_heading), architecture detection (arch_gcc, arch_go), colored output utilities (red, green, white), environment detection (in_ci, is_truthy), and secret-scanning filters for TruffleHog (trufflehog_jq_filter_common, trufflehog_jq_filter_files, trufflehog_jq_filter_logs).

Usage

Developers and CI scripts source this library at the top of their Bash scripts to gain access to its utility functions. It is particularly important for Buildkite CI pipelines where the try/try_status_report pattern provides structured test execution with pass/fail tracking and collapsible log sections.

Code Reference

Source Location

• Repository: MaterializeInc_Materialize
• File: misc/shlib/shlib.bash

Signature

# Error handling
die() { echo "$@" >&2; exit 1; }

# Verbose command execution
run() { echo "\$ $*" >&2; "$@"; }

# Command existence check
command_exists() { hash "$1" 2>/dev/null; }

# Version comparison (checks sorted order)
version_compat() { printf "%s\n" "$@" | sort --check=silent --version-sort; }

# Git helpers
git_empty_tree() { git hash-object -t tree /dev/null; }
git_files() { git diff --ignore-submodules=all --raw "$(git_empty_tree)" -- "$@" | awk '$2 != 120000 {print $6}'; }

# CI try/report pattern
try() { ... }              # Run command, track pass/fail
try_status_report() { ... } # Exit with aggregate pass/fail status

# Architecture detection
arch_gcc() { ... }  # Returns x86_64 or aarch64
arch_go() { ... }   # Returns amd64 or arm64

# Environment detection
in_ci() { [ -z "${BUILDKITE-}" ] && return 1; return 0; }
is_truthy() { ... }

# Colored output
red() { echo -ne "\e[31m$*\e[0m"; }
green() { echo -ne "\e[32m$*\e[0m"; }
white() { echo -ne "\e[97m$*\e[0m"; }

# TruffleHog secret filters
trufflehog_jq_filter_common() { ... }
trufflehog_jq_filter_files() { ... }
trufflehog_jq_filter_logs() { ... }

Import

# Source the library in a Bash script
source "$(dirname "$0")/../misc/shlib/shlib.bash"

# Or using the REPO_ROOT pattern
source "$REPO_ROOT/misc/shlib/shlib.bash"

I/O Contract

Inputs

Name	Type	Required	Description
BUILDKITE	Environment variable	No	When set, indicates script is running in Buildkite CI (used by in_ci)
Command arguments	Strings	Varies	Arguments passed to functions like die, run, try, version_compat
PREFIX_N variables	Environment variables	No	Numbered env vars read by read_list (e.g., CMD_0, CMD_1, CMD_2)

Outputs

Name	Type	Description
ci_try_passed	Global integer	Count of commands that passed when using the try pattern
ci_try_total	Global integer	Total count of commands executed with try
try_last_failed	Global boolean	Whether the last try command failed
result	Global array	Output array populated by read_list
Stderr output	Text	Diagnostic messages, CI section headers, colored output

Usage Examples

#!/usr/bin/env bash
source misc/shlib/shlib.bash

# Exit with error if a required tool is missing
command_exists docker || die "docker is not installed"

# Check minimum version requirement
version_compat "1.70.0" "$(rustc --version | awk '{print $2}')" \
    || die "Rust 1.70.0+ required"

# Run a series of CI checks with aggregate reporting
try cargo build --workspace
try cargo test --workspace
try cargo clippy --workspace
try_status_report

# Detect architecture for cross-compilation
ARCH=$(arch_gcc)
echo "Building for $ARCH"

# Use colored output for status messages
green "All checks passed"
red "Build failed"