Data#

The policyengine_core.data module is a new feature from the original OpenFisca Core fork. It provides a standardised class definition of a Dataset, which contains all the data needed to instantiate an OpenFisca simulation with nationally (or lower) representative data.

Dataset#

class policyengine_core.data.Dataset(require: bool = False)#

Bases: object

The Dataset class is a base class for datasets used directly or indirectly for microsimulation models. A dataset defines a generation function to create it from other data, and this class provides common features like storage, metadata and loading.

ARRAYS = 'arrays'#

FLAT_FILE = 'flat_file'#

TABLES = 'tables'#

TIME_PERIOD_ARRAYS = 'time_period_arrays'#

data_format: str = None#

download(url: str = None, version: str = None) → None#

Downloads a file to the dataset’s file path.

Parameters:: url (str) – The url to download.

download_from_huggingface(owner_name: str, model_name: str, file_name: str, version: str = None)#

Downloads the dataset from HuggingFace.

Parameters:

owner_name (str) – The owner name.
model_name (str) – The model name.

property exists: bool#

Checks whether the dataset exists.

Returns:: Whether the dataset exists.
Return type:: bool

file_path: Path = None#

static from_dataframe(dataframe: DataFrame, time_period: str = None)#

Creates a dataset from a DataFrame.

Returns:: The dataset.
Return type:: Dataset

static from_file(file_path: str, time_period: str = None)#

Creates a dataset from a file.

Parameters:: file_path (str) – The file path to create the dataset from.
Returns:: The dataset.
Return type:: Dataset

generate()#

Generates the dataset for a given year (all datasets should implement this method).

Raises:: NotImplementedError – If the function has not been overriden.

label: str = None#

load(key: str = None, mode: str = 'r') → Union[File, array, DataFrame, HDFStore]#

Loads the dataset for a given year, returning a H5 file reader. You can then access the dataset like a dictionary (e.g.e Dataset.load(2022)[“variable”]).

Parameters:

key (str, optional) – The key to load. Defaults to None.
mode (str, optional) – The mode to open the file with. Defaults to “r”.

Returns:

The dataset.

Return type:

Union[h5py.File, np.array, pd.DataFrame, pd.HDFStore]

load_dataset()#

Loads a complete dataset from disk.

Returns:: The dataset.
Return type:: Dict[str, Dict[str, Sequence]]

name: str = None#

remove()#: Removes the dataset from disk.

save(key: str, values: Union[array, DataFrame])#

Overwrites the values for key with values.

Parameters:

key (str) – The key to save.
values (Union[np.array, pd.DataFrame]) – The values to save.

save_dataset(data, file_path: str = None) → None#

Writes a complete dataset to disk.

Parameters:: data – The data to save.

>>> example_data: Dict[str, Dict[str, Sequence]] = {
...     "employment_income": {
...         "2022": np.array([25000, 25000, 30000, 30000]),
...     },
... }
>>> example_data["employment_income"]["2022"] = [25000, 25000, 30000, 30000]

store_file(file_path: str)#

Moves a file to the dataset’s file path.

Parameters:: file_path (str) – The file path to move.

time_period: str = None#

upload(url: str = None)#

Uploads the dataset to a URL.

Parameters:: url (str) – The url to upload.

upload_to_huggingface(owner_name: str, model_name: str, file_name: str)#

Uploads the dataset to HuggingFace.

Parameters:

owner_name (str) – The owner name.
model_name (str) – The model name.

url: str = None#

property variables: List[str]#

Returns the variables in the dataset.

Returns:: The variables in the dataset.
Return type:: List[str]

Data#

Dataset#

PublicDataset#

PrivateDataset#