Data storage#

The policyengine_core.data_storage module contains the classes that are used to handle the storage of data in simulations.

InMemoryStorage#

class policyengine_core.data_storage.in_memory_storage.InMemoryStorage(is_eternal: bool)#

Bases: object

Low-level class responsible for storing and retrieving calculated vectors in memory

clone(share_arrays: bool = False) → InMemoryStorage#

Copy this storage.

By default every stored array is copied straight away.

With share_arrays, the clone starts with views of this storage’s arrays and copies an array only when it is first read through get, so an array the clone never reads is never copied. What get returns is the clone’s own array either way, so writing into it in place does not change this storage. (Masked arrays are copied straight away: a view would share their mask.)

The clone has its own index in both cases: put and delete on either storage replace or drop index entries without touching the arrays, so neither storage sees what the other stores, replaces or deletes after cloning.

The one difference from copying straight away: writing in place into one of this storage’s arrays, after cloning and before the clone first reads it, changes the value the clone reads.

A storage is not safe to read from several threads at once: two first reads of the same key can each make a copy.

delete(period: Period = None, branch_name: str = 'default') → None#

drop_computed(*, since: Optional[int] = None) → int#

Delete stored values that are not inputs, and return how many.

With since, only values stored with that sequence number or a later one are deleted (a value with no recorded number counts as later). Inputs, which put received without derived, are kept whatever their number.

get(period: Period, branch_name: str = 'default') → ArrayLike#

get_known_branch_periods() → list#

get_known_periods() → list#

get_memory_usage() → dict#

has(period: Period, branch_name: str = 'default') → bool#

Whether a value is stored for period under branch_name.

Unlike get, this never copies an array shared by clone.

has_unnumbered_values() → bool#: Whether a value was stored without put (so without a number).

inputs_since(since: Optional[int] = None) → List[Tuple[Period, int]]#: The period and number of each input stored at since or later (or ever).

is_derived(period: Period, branch_name: str = 'default') → bool#: Whether the value stored for period under branch_name was stored with derived=True; False if none is stored.

is_eternal: bool#

mark_derived_except(keys: Set[str]) → None#: Mark every stored value derived except those stored for keys.

put(value: ArrayLike, period: Period, branch_name: str = 'default', derived: bool = False, sequence_number: Optional[int] = None) → None#

Store value for period on branch_name.

derived marks a value the simulation calculated rather than took as input (see _inputs), and sequence_number records when the value was stored (a new number by default); see drop_computed().

OnDiskStorage#

Each independently created temporary holder storage gets its own subdirectory, so replacing a holder leaves files read by existing clones intact. Cloned storages retain their source’s directory and use new files for later writes. Explicit Holder.create_disk_storage(directory=...) calls and preserved storages keep the directory/variable layout used by simulation dumps.

OnDiskStorage.restore() reads direct key files in its directory. It does not read files in the replaced subdirectory or select the latest write across independently constructed storages. Clone or copy a restored storage before writing when its existing values must remain isolated.

class policyengine_core.data_storage.on_disk_storage.OnDiskStorage(storage_dir: str, is_eternal: bool = False, preserve_storage_dir: bool = False)#

Bases: object

Low-level class responsible for storing and retrieving calculated vectors on disk

Compound periods contain colons, which Windows forbids in filenames. Their physical filenames use semicolons in the period suffix instead; logical storage keys keep the original period strings. restore reads both these portable filenames and legacy filenames containing colons.

clone() → OnDiskStorage#

Create a private metadata view over this storage directory.

The file and enum mappings are copied so deleting or rewiring entries through the clone does not mutate the source storage. The underlying .npy files are shared, so neither storage writes over them: a later put of one of their keys, through either storage, writes a new file (see _path_to_write). Clones retain the original cleanup owner so the shared directory stays alive, but never own cleanup themselves, nor preserve the directory (see preserve_storage_dir).

delete(period: Period = None, branch_name: str = 'default') → None#

drop_computed(*, since: Optional[int] = None) → int#

Forget stored values that are not inputs, and return how many.

As InMemoryStorage.drop_computed(). The files stay on disk (clones may still read them) until the storage directory is removed; writing a dropped key again reuses this storage’s own file for it, as after delete.

get(period: Period, branch_name: str = 'default') → ArrayLike#

get_known_branch_periods() → list#

get_known_periods() → list#

has(period: Period, branch_name: str = 'default') → bool#

Whether a value is stored for period under branch_name.

Unlike get, this reads no file.

has_unnumbered_values() → bool#: As InMemoryStorage.has_unnumbered_values().

inputs_since(since: Optional[int] = None) → List[Tuple[Period, int]]#: As InMemoryStorage.inputs_since().

is_derived(period: Period, branch_name: str = 'default') → bool#: Whether the value stored for period under branch_name was stored with derived=True; False if none is stored.

mark_derived_except(keys: Set[str]) → None#: As InMemoryStorage.mark_derived_except().

property preserve_storage_dir: bool#

Whether this storage leaves its directory when it is collected.

A storage made with False removes its directory when collected, in the process that made it. One made with True, or set to True later, preserves the directory: neither it nor the removal of a simulation’s temporary directory containing it removes it (see TemporaryStorageDirectory). Two kinds of storage read another storage’s directory instead, and preserve nothing: a clone or a copied storage, for which this is True; and a storage made with True for a directory that a live storage made with False removes. Each keeps that storage alive, so the files last while it reads them.

put(value: ArrayLike, period: Period, branch_name: str = 'default', derived: bool = False, sequence_number: Optional[int] = None) → None#

Store value for period on branch_name.

derived and sequence_number are as in InMemoryStorage.put().

restore() → None#

Read back the values stored in this storage’s directory: for each key, the file named for it (not those in REPLACEMENTS_DIR).

This storage writes over portable files, as over those it wrote, until it is cloned or copied, or this process forks: from then on it writes a new file instead, since the clone, copy or forked process reads them. A storage made separately for the directory, the one that wrote them say, still writes over them, as before. So may another storage in this one’s family, when restore runs after it was cloned and this process has not forked since.

The period suffix of new filenames uses semicolons for the colons in compound period strings. Legacy filenames containing colons are still accepted. If both versions of a key exist after updating a legacy value, the portable filename holds the current value and wins regardless of directory listing order. Readers still indexed to the legacy file keep its old value until they restore again. Branch names are unchanged.

TemporaryStorageDirectory#

A caller-supplied simulation._data_storage_dir is created on first use if it does not exist. It keeps the caller-owned lifetime described below. Exits that skip finalizers (such as os._exit in multiprocessing workers) can leave uniquely named holder subdirectories in a caller-owned directory to accumulate across runs; the caller must remove them.

The temporary folder a simulation stores values on disk in.

A simulation with a MemoryConfig stores values on disk, each independently created holder’s in its own subfolder (an OnDiskStorage) of the simulation’s folder, Simulation.data_storage_dir. Unless the caller chose that folder, the simulation makes it, as a TemporaryStorageDirectory. That object removes the folder, with everything in it but the subfolders of preserved storages (below), when it is garbage-collected, or at interpreter exit, and only in the process that made it. What reads files in the folder, or stores values in a folder inside it, keeps the object alive. What each way of sharing the files guarantees:

Within the process that made the folder. The simulation keeps the folder, and so does every disk storage anywhere under it, whether a holder made it or the caller constructed it there. Simulation.clone, get_branch and derivative copy the holders: each copied disk storage is an OnDiskStorage.clone that reads the source’s files and keeps alive the folder and the storage that removes their subfolder. A pickled or copied disk storage does the same. The disk storages a clone makes itself go in a folder of its own. Replacing a holder creates a new subfolder, so its writes and cleanup leave the old holder’s files alone.

Folders nest: a simulation given the folder, or a folder in it, as its own (_data_storage_dir) keeps the folder alive from then on, and so does a clone of that simulation, which makes its own folder there. A folder made inside another live one keeps that one alive, and has it leave the subfolders it preserves, however deep the nesting.

So nothing removes a file any of these reads while it is alive, whatever is collected first, with one exception, as before: a disk storage removes everything in its subfolder when collected, including files there that another storage reads without keeping it alive. Those are the files of a second storage explicitly made to remove the same subfolder, and anything made inside the subfolder. Nor does a storage write over a file another reads: storages cloned or copied from one another write a new file instead (see OnDiskStorage._path_to_write), for the files any of them read back before (OnDiskStorage.restore) too. Storages each made for one subfolder still write the same file for a key, as before, and restore reads whatever files are in the subfolder when it runs. A restore run after a storage was cloned, with no fork since, is not covered either: another storage in its family may still write over a file it read back.
Explicitly preserved storages. A disk storage made with preserve_storage_dir=True (Holder.create_disk_storage(preserve=True), say), or whose preserve_storage_dir is later set to True, keeps its subfolder: removing the folder, or a folder containing it that a simulation made, leaves that subfolder, every file in it and the folders leading to it, for the caller to remove. Exceptions: a subfolder inside another disk storage’s goes when that storage removes its own, as above; and two kinds of storage read another storage’s subfolder rather than keep one. Those are a clone or copy, whose flag is True only because it never removes its source’s subfolder, and a storage made with the flag for a subfolder that a live storage made without it removes. Each keeps that storage alive while it reads the files, which that storage still removes once collected, as before.
Another process. A process forked from the one that made the folder has copies of all these objects; so does a process a disk storage is unpickled in. Each process removes only the folders and disk storage subfolders it made, but removing a folder removes everything in it, the folder a forked process made inside it included. Disk storages copied into another process, by forking or unpickling, or cloned from one there, write only new files of their own, and after forking, the disk storages of the process that made the folder write over no file they wrote or read back before. So neither process writes over a file the other reads through these storages. A forked process stores what new holders put on disk in a folder of its own, made inside the folder it inherited. But nothing in another process keeps the folder alive: once the process that made it collects the simulation and its storages, or exits, the folder is gone, with everything in it, the forked process’s own folder included, even while a forked or unpickling process still reads them. Explicit preservation is likewise kept only by the process that made the folder.
A folder the caller chose (_data_storage_dir). Nothing removes it, only the subfolders disk storages made in it, as before, unless it is in a folder a simulation made: that one removes it with everything else in it, once nothing keeps that one alive (above). A clone or forked process makes its own folder inside it. If it does not exist, it is created on first use.
A process that ends without running the exit handlers that remove its folders (killed, or ended with os._exit, as a process multiprocessing forks is) leaves the folders it made that were still alive, except those a forked process made inside a folder it inherited, which go with that folder.

class policyengine_core.data_storage.storage_directory.TemporaryStorageDirectory(parent: Optional[str] = None)#

Bases: object

A temporary directory for values a simulation stores on disk.

The directory is removed once this object is garbage-collected, or at interpreter exit, whichever comes first, and only by the process that created it. The simulation that created it keeps a reference to this object, and so does every disk storage made in it in this process (see OnDiskStorage), which the storages cloned or copied from those keep alive in turn. So do a simulation given this directory or a folder in it (Simulation._data_storage_dir), and a directory of this class made in it. Subfolders of disk storages that preserve theirs are left, with the folders leading to them, here or in a directory made in this one. What this guarantees for each way of sharing the files is in the docstring of this module.

Copying this object gives the same object. Unpickling it in the process that created it, while the directory exists, also gives the same object; anywhere else it gives one that never removes the directory, which belongs to the process that created it.

keep(path: str) → None#: Leave path, a folder in the directory, when removing it, or any live directory this one was made in.

release(path: str) → None#: Remove path with the directory again (see keep).

property removes_directory: bool#: Whether this object removes the directory when it is collected.