Shard Lifecycle

An Edge Shard is backed by a directory on local disk. The lifecycle of a shard is:

1. Create a new shard with EdgeShard.create (Python) or EdgeShard::new (Rust), or load an existing one with load.
2. Use the shard to update and query data.
3. Flush pending changes to disk, and close the shard to release its resources.

Because the shard owns the files in its directory, only one EdgeShard may be open on a given directory at a time.

Create a New Edge Shard

Creates a new Edge Shard at path using the supplied configuration.

@staticmethod
def create(path: str, config: EdgeConfig) -> EdgeShard

pub fn new(path: &Path, config: EdgeConfig) -> OperationResult<EdgeShard>

Returns a new EdgeShard instance.

Creation fails if the shard’s segments directory already contains any segment. To open a directory that already holds data, use load instead.

The configuration is persisted to edge_config.json inside the shard directory, so a later load can recover it without you passing it again. Write-ahead log behavior follows config.wal_options, which defaults to 32 MiB segments when unset. Refer to Custom WAL Size.

Load an Existing Edge Shard

Opens an Edge Shard from existing files at path.

@staticmethod
def load(path: str, config: Optional[EdgeConfig] = None) -> EdgeShard

pub fn load(path: &Path, config: Option<EdgeConfig>) -> OperationResult<EdgeShard>

Returns the loaded EdgeShard instance.

Loading fails if the directory contains no segments and no configuration can be loaded or inferred.

Parameters that you change and that affect stored segments do not take effect immediately. Existing segments converge to the new value as the optimizers run.

Inspect the Path and Configuration

Rust only

Return the shard’s directory and its currently resolved configuration.

pub fn path(&self) -> &Path
pub fn config(&self) -> parking_lot::RwLockReadGuard<'_, EdgeConfig>

config returns a read guard rather than a copy, so the configuration cannot be mutated through it and the guard should be dropped promptly. To change configuration on a live shard, use set_hnsw_config, set_vector_hnsw_config, or set_optimizers_config.

Get Shard Information

Returns metadata about the shard’s contents.

def info(self) -> ShardInfo

pub fn info(&self) -> OperationResult<ShardInfo>

ShardInfo carries the following fields. The counts are summed across segments, and a point can be present in more than one segment before it is optimized, so points_count and indexed_vectors_count are approximate and can read higher than the number of distinct points:

An indexed_vectors_count well below points_count means segments are still waiting to be optimized. Refer to optimize.

Flush Pending Changes

Persists the write-ahead log and all segments to disk.

def flush(self) -> None

pub fn flush(&self) -> OperationResult<()>

Returns nothing in Python. In Rust, returns Ok(()) on success, or an error if the WAL or a segment could not be flushed.

flush blocks until the WAL and segment locks are free. A flush issued while an update or optimize is in flight waits for that operation to finish and then persists, rather than failing with a lock contention error. A genuine I/O error during the flush is still surfaced to the caller.

Close an Edge Shard

Closes the shard and releases its resources.

def close(self) -> None

Rust has no close method. EdgeShard implements Drop, so the shard is closed when it goes out of scope:

{
    let shard = EdgeShard::new(path, config)?;
    // ... use the shard ...
} // `shard` is dropped here, flushing to disk

In both languages, closing flushes pending data to disk. The data remains on disk and the directory can be reopened with load.