fsutil: browsing Marin's buckets¶
fsutil lists, reads, sizes, and copies objects across every bucket declared in
config/*.yaml — GCS, CoreWeave AI Object Storage, and R2 — from one command, and opens
an interactive browser over all of them.
uv run fsutil # interactive browser, starting at the bucket list
uv run fsutil buckets # what is declared, and whether credentials are set
uv run fsutil ls -l gs://marin-us-central2/scratch
uv run fsutil cat s3://marin-us-east-02a/iris/my-job/config.json
uv run fsutil du s3://marin-us-east-02a/iris/my-job
uv run fsutil cp -r s3://marin-us-east-02a/iris/my-job/logs /tmp/logs
Paths are always full URLs — gs://bucket/key, s3://bucket/key, or a local path.
There is no current bucket to keep track of, and a copy may name a different backend on
each side.
Credentials¶
GCS uses application default credentials:
CoreWeave object storage uses an access key pair from the CoreWeave console:
R2 buckets use R2_KEY_ID / R2_KEY_SECRET the same way. Either backend also accepts
the generic AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair, which is enough when only
one of them is in play.
fsutil buckets reports which backends are reachable, so an unconfigured one shows up
before a command fails against it. A backend with no credentials only breaks commands
that touch its buckets; the rest keep working.
Commands¶
| Command | What it does |
|---|---|
ls [URL] [-l] |
Immediate children; with no URL, the declared buckets. -l adds size and modification time |
cat URL [--raw] |
Print a file. Tabular JSON and JSONL render as a table, and --raw writes stored bytes to stdout |
head URL [-n N] |
Print the first N lines |
stat URL |
The object's metadata as the backend reports it |
du URL |
Total bytes and object count beneath a prefix |
find PATTERN |
Paths matching a glob, e.g. gs://marin-us-central2/x/**/*.json |
cp SRC DST [-r] |
Copy between any two locations, including across backends |
browse [URL] |
The interactive browser |
cat, head, and the browser decompress .gz, .bz2, .xz, and .lzma files by
suffix. A data.json.gz preview uses the JSON table view. cat --raw writes the compressed
bytes.
Formatted file previews are capped at 10 MB after decompression. cat --raw is capped
at 10 MB of stored bytes. Use cp to fetch a whole object.
The browser¶
fsutil browse starts at the bucket list, so descending into GCS or CoreWeave is the
same keystroke.
| Key | Action |
|---|---|
j / k, arrows |
Move; g / G jump to top and bottom |
enter / l |
Open a prefix, or preview a file |
backspace / h |
Up one level |
/ |
Filter the listing by name |
s |
Cycle sort: name, size, modified |
d |
Total the highlighted prefix |
y |
Show the highlighted entry's full URL |
r |
Re-list |
q |
Quit, or close the file viewer |
Where the bucket list comes from¶
Buckets and backends are declared in config/*.yaml under data.region_buckets, and
each S3-compatible backend's endpoint and credential variables under data.stores.
fsutil builds one filesystem per backend from that config, so adding a bucket is a
config change rather than a code change. The same routing is available to library code
as rigging.filesystem.filesystem_for(url), which is the way to reach two S3 backends
from one process — the process-wide AWS_* / FSSPEC_S3 variables can only describe
one at a time.