# tinker_cookbook.stores.FsspecStorage

## *class* [**tinker_cookbook.stores.FsspecStorage**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L345)()

Storage backend wrapping any `fsspec.AbstractFileSystem`.

Supports S3 (via `s3fs`), GCS (via `gcsfs`), Azure (via `adlfs`), and any other filesystem that fsspec supports.

**Append strategy:** Cloud backends don't support native append. This class stages append-only files locally using POSIX atomic writes, then uploads them to cloud on `flush`. Reads check the local stage first, so [`IncrementalReader`](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/stores/incrementalreader/index.md) sees appended data immediately.

Call `flush` at checkpoints or training end to persist staged data to cloud. The context manager calls `close` on exit (which flushes and removes the staging directory).

**Data safety:** Unflushed staged data lives in a local temp directory. If the process crashes before `flush()`, unflushed appends are lost. Flush at every checkpoint to minimize data loss on crash.

Pickle-serializable — stores protocol, root, and kwargs. Local staged data is NOT included in pickle (each process starts with an empty stage).

### [**url**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L403)(*path*)

Return a URI like `s3://bucket/prefix/path`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L403) (*str*)

**Returns:** *str*

### [**read**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L409)(*path*)

See `Storage.read`. Reads from local stage if available.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L409) (*str*)

**Returns:** *bytes*

### [**write**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L415)(*path*, *data*)

See `Storage.write`. Writes directly to cloud.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L415) (*str*)
- [**data**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L415) (*bytes*)

**Returns:** *None*

### [**append**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L426)(*path*, *data*)

See `Storage.append`.

Stages appends locally using POSIX atomic writes. The first append for a given path pulls existing content from cloud (if any), then all subsequent appends are local. Call `flush` to upload staged data to cloud.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L426) (*str*)
- [**data**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L426) (*bytes*)

**Returns:** *None*

### [**exists**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L447)(*path*)

See `Storage.exists`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L447) (*str*)

**Returns:** *bool*

### [**stat**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L453)(*path*)

See `Storage.stat`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L453) (*str*)

**Returns:** *[StorageStat](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/stores/storagestat/index.md) | None*

### [**read_range**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L470)(*path*, *offset*, *length*)

See `Storage.read_range`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L470) (*str*)
- [**offset**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L470) (*int*)
- [**length**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L470) (*int | None*)

**Returns:** *bytes*

### [**list_dir**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L483)(*prefix*)

See `Storage.list_dir`. Returns immediate children names only.

**Parameters:**

- [**prefix**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L483) (*str*)

**Returns:** *list[str]*

### [**remove**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L499)(*path*)

See `Storage.remove`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L499) (*str*)

**Returns:** *None*

### [**remove_dir**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L508)(*path*)

See `Storage.remove_dir`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L508) (*str*)

**Returns:** *None*

### [**flush**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L513)()

Upload all locally staged files to cloud.

Call this at checkpoints or training end. The context manager calls this automatically on exit. Staged files are removed after upload so that subsequent appends pull fresh data from cloud, avoiding re-uploading the entire file on every flush cycle.

**Returns:** *None*

### [**close**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L531)()

Flush staged data and clean up the local staging directory.

**Returns:** *None*

### [**aread**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L556)(*path*)

Async version of `read`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L556) (*str*)

**Returns:** *bytes*

### [**awrite**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L560)(*path*, *data*)

Async version of `write`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L560) (*str*)
- [**data**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L560) (*bytes*)

**Returns:** *None*

### [**aappend**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L564)(*path*, *data*)

Async version of `append`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L564) (*str*)
- [**data**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L564) (*bytes*)

**Returns:** *None*

### [**aexists**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L568)(*path*)

Async version of `exists`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L568) (*str*)

**Returns:** *bool*

### [**astat**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L572)(*path*)

Async version of `stat`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L572) (*str*)

**Returns:** *[StorageStat](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/stores/storagestat/index.md) | None*

### [**aread_range**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L576)(*path*, *offset*, *length*)

Async version of `read_range`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L576) (*str*)
- [**offset**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L576) (*int*)
- [**length**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L576) (*int | None*)

**Returns:** *bytes*

### [**alist_dir**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L580)(*prefix*)

Async version of `list_dir`.

**Parameters:**

- [**prefix**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L580) (*str*)

**Returns:** *list[str]*

### [**aremove**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L584)(*path*)

Async version of `remove`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L584) (*str*)

**Returns:** *None*

### [**aremove_dir**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L588)(*path*)

Async version of `remove_dir`.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/storage.py#L588) (*str*)

**Returns:** *None*