Simulations#
The policyengine_core.simulations module contains the definition of Simulation, the singular most important class in the repo. Simulations combine the logic of a country package with data, and can use the country logic and parameters to calculate the values of unknown variables. The class SimulationBuilder can create Simulations from a variety of inputs: JSON descriptions, or dataset arrays.
Simulation#
- class policyengine_core.simulations.simulation.Simulation(tax_benefit_system: TaxBenefitSystem = None, populations: Dict[str, Population] = None, situation: dict = None, dataset: Union[str, Type[Dataset]] = None, reform: Reform = None, trace: bool = False, default_input_period: str = None, default_calculation_period: str = None)#
Bases:
objectRepresents a simulation, and handles the calculation logic
- baseline: Simulation = None#
- build_from_dataset() None#
Build a simulation from a dataset.
Entity-specific role columns take precedence over a generic
rolecolumn, including period-suffixed entity role columns in flat files. Without either, each person receives the literaldefault_rolekey. Only flattened-role keys match: unmatched and compound keys remain unrecognized rather than selecting a first role or subrole.A recognized implicit default with a finite role maximum must fit every group’s membership count. Otherwise an explicit role column is required. Explicit dataset role columns do not validate role capacity.
default_role=Nonerequires a role column.
- build_from_populations(populations: Dict[str, Population]) None#
This method of initialisation requires the populations to be pre-initialised.
- Parameters:
populations (Dict[str, Population]) – A dictionary of populations, indexed by entity key.
- calculate(variable_name: str, period: Period = None, map_to: str = None, decode_enums: bool = False) ArrayLike#
Calculate
variable_nameforperiod.- Parameters:
variable_name (str) – The name of the variable to calculate.
period (Period) – The period to calculate the variable for.
map_to (str) – The name of the variable to map the result to. If None, the result is returned as is.
decode_enums (bool) – If True, the result is decoded from an array of integers to an array of strings.
- Returns:
The calculated variable.
- Return type:
ArrayLike
- calculate_dataframe(variable_names: List[str], period: Period = None, map_to: str = None) DataFrame#
Calculate
variable_namesforperiod.- Parameters:
variable_names (List[str]) – A list of variable names to calculate.
period (Period) – The period to calculate for.
- Returns:
A dataframe containing the calculated variables.
- Return type:
pd.DataFrame
- calculate_output(variable_name: str, period: Period = None) ArrayLike#
Calculate the value of a variable using the
calculate_outputattribute of the variable.
- check_macro_cache(variable_name: str, period: str) bool#
Check if the variable is able to have cached value
- clone(debug: bool = False, trace: bool = False, clone_tax_benefit_system: bool = True) Simulation#
Copy the simulation just enough to be able to run the copy without modifying the original simulation.
The copy records its own cache invalidations (
invalidated_caches, starting from this simulation’s pending ones). Ifbaselineis a branch of this simulation, as a reform simulation’s is, the copy gets a copy of that branch as its own: a branch of the copy, under the same name, with this baseline’s tax-benefit system, traced in the copy if this baseline is traced in this simulation. Any otherbaselineis shared: a branch’s is its parent’s, and a country package that builds a separate baseline simulation copies it, if it needs to, in its ownclone.Every cached array is copied, except in the first
cloneof this simulation made whileget_branchis creating a branch of it: that copy shares the arrays until it reads them (seeget_branch()). A subclass’sclonethat calls this one throughsuper().clonetakes part in that the same way. A subclassclonethat first clones the same simulation directly gets the sharing in that direct clone instead, and its branch is a full copy.The copy’s method aliases (
calc,df, and any other bound method of this simulation kept on the instance) are bound to the copy. Withclone_tax_benefit_system, the copy of the system names the copy as its simulation and the copy’s populations use its entities, so nothing in the copy refers back to this simulation through them. Without it, the copy shares this simulation’s system, which still names this simulation.
- create_shortcuts() None#
- property data_storage_dir: str#
Folder in which this simulation stores values on disk when memory is short (see
MemoryConfig).Set
_data_storage_dirto choose the folder: it is created on first use if it does not exist. Nothing removes it, unless it is in a temporary folder a simulation made, which this simulation then keeps alive, and which removes it with everything else in it once nothing keeps it. Otherwise this is a new temporary folder, removed once this simulation, every disk storage in the folder (those its clones and branches copied included), every simulation given a folder in it and every temporary folder made in it are garbage-collected in this process, or at interpreter exit, except the subfolders of disk storages that preserve theirs. In a process forked from the one the folder is for, this is a new folder for that process, made inside that one. See thestorage_directorymodule for what this guarantees.
- default_calculation_period: str = None#
- default_input_period: str = None#
- default_role: str = 'member'#
- default_tax_benefit_system: Type[ForwardRef('TaxBenefitSystem')] = None#
- default_tax_benefit_system_instance: TaxBenefitSystem = None#
- delete_arrays(variable: str, period: Period = None) None#
Delete a variable’s values visible to this simulation branch.
The calling branch, each ancestor branch, and the default branch are purged from this simulation’s private holder storage. Other branch names and the parent simulation’s holder storage remain unchanged. Deleted inputs stop counting as inputs: a value calculated later for the same period is a formula result, which
apply_reformdiscards andto_input_dataframedoes not export.- Parameters:
variable – the variable whose cached values should be deleted
period – the period to delete, or all periods when omitted
Example:
>>> from policyengine_core.country_template import CountryTaxBenefitSystem >>> simulation = Simulation(CountryTaxBenefitSystem()) >>> simulation.set_input('age', '2018-04', [12, 14]) >>> simulation.set_input('age', '2018-05', [13, 14]) >>> simulation.get_array('age', '2018-05') array([13, 14], dtype=int32) >>> simulation.delete_arrays('age', '2018-05') >>> simulation.get_array('age', '2018-04') array([12, 14], dtype=int32) >>> simulation.get_array('age', '2018-05') is None True >>> simulation.set_input('age', '2018-05', [13, 14]) >>> simulation.delete_arrays('age') >>> simulation.get_array('age', '2018-04') is None True >>> simulation.get_array('age', '2018-05') is None True
- derivative(variable: str, wrt: str, period: Period = None, delta: float = 1) ArrayLike#
Compute the derivative of a variable w.r.t another variable.
- Parameters:
variable (str) – The variable to differentiate.
wrt (str) – The variable to differentiate with respect to.
period (Period) – The period for which to compute the derivative.
delta (float) – The infinitesimal to use for the derivative.
- Returns:
The derivative.
- Return type:
ArrayLike
- describe_entities() dict#
- drop_computed_arrays() int#
Delete every value this simulation holds except inputs.
Inputs are the values stored through
set_input: the dataset or situation the simulation was built from, inputs set on it and, for a branch, inputs set on the simulations it was created from before it was created. Values a customset_inputhandler calculates are not inputs. Every other value is calculated again when next requested, and the simulation stops reading macro-cache files.Use this on a branch whose tax-benefit system or parameters differ from its parent’s, whose values the branch would otherwise inherit;
set_inputon a branch drops what depends on the input by itself. Branches already created from this simulation keep their values.- Returns:
The number of arrays deleted.
- Return type:
int
- extract_person(index: int = 0, exclude_entities: tuple = ('state',)) dict#
Extract a person from the simulation. Returns a situation JSON with their inputs (including their containing entities).
- Parameters:
index (int) – The index of the person to extract.
- Returns:
A dictionary containing the person’s values.
- Return type:
dict
- get_array(variable_name: str, period: Period) ArrayLike#
Return the value of
variable_nameforperiod, if this value is alreay in the cache (if it has been set as an input or previously calculated).Unlike
calculate(), this method does not trigger calculations and does not use any formula.
- get_branch(name: str = 'branch', clone_system: bool = False) Simulation#
Create a clone of this simulation, whose calculations are traced in the original.
The branch starts from the values this simulation has cached when the branch is created. It does not copy them up front: each of the branch’s holders gets its own index of this simulation’s arrays, and copies an array the first time the branch reads it. A numpy array the branch never reads is never copied (masked arrays, and values that are not numpy arrays, are copied when the branch is created).
What the branch stores (
set_input, calculations, deletions) goes into its own index only, and what it reads is its own copy, so nothing done through the branch changes this simulation’s values. What this simulation stores after branching stays out of the branch.The one difference from copying every array up front: code that writes in place into one of this simulation’s cached arrays (
array[mask] = 0orarray += 1, instead of storing a new array withset_input) after branching also changes the value the branch reads, if the branch has not read that array yet.As with the rest of a simulation, a branch is not safe to read from several threads at once: two first reads of the same array can each make a copy.
A new branch starts with the values this simulation holds; asked for a name it already has, this returns that branch as it is. An input set on the branch drops those that may have been calculated from the value it replaces (see
set_input()). A branch whose tax-benefit system or parameters are changed should calldrop_computed_arrays()before calculating.- Parameters:
name (str, optional) – Name of the branch. Defaults to “branch”.
clone_system (bool, optional) – Whether to clone the tax-benefit system. Use this if you’re changing policy parameters. Defaults to False.
- Returns:
The cloned simulation.
- Return type:
- get_holder(variable_name: str) Holder#
Get the
Holderassociated with the variablevariable_namefor the simulation
- get_input_variables(include_computed_variables: bool = True) List[str]#
Return variable names stored as inputs on this simulation.
- Parameters:
include_computed_variables – When
True, return the legacy runtime list of variables with stored values. WhenFalse, return only structurally input variables that were populated throughset_inputon the current branch.- Returns:
Stored input variable names.
- Return type:
List[str]
- get_known_periods(variable: str) List[Period]#
Get a list variable’s known period, i.e. the periods where a value has been initialized and
- Parameters:
variable – the variable to be set
Example:
>>> from policyengine_core.country_template import CountryTaxBenefitSystem >>> simulation = Simulation(CountryTaxBenefitSystem()) >>> simulation.set_input('age', '2018-04', [12, 14]) >>> simulation.set_input('age', '2018-05', [13, 14]) >>> simulation.get_known_periods('age') [Period((u'month', Instant((2018, 5, 1)), 1)), Period((u'month', Instant((2018, 4, 1)), 1))]
- get_memory_usage(variables: List[str] = None) dict#
Get data about the virtual memory usage of the simulation
- get_population(plural: str = None) Population#
- get_variable_population(variable_name: str) Population#
- invalidate_spiral_variables(variable: str) None#
- is_over_dataset: bool = False#
- link_to_entities_instances() None#
- macro_cache_read: bool = False#
- macro_cache_write: bool = False#
- map_result(values: ArrayLike, source_entity: str, target_entity: str, how: str = None)#
Maps values from one entity to another.
- Parameters:
arr (np.array) – The values in their original position.
source_entity (str) – The source entity key.
target_entity (str) – The target entity key.
how (str, optional) – A function to use when mapping. Defaults to None.
- Raises:
ValueError – If an invalid (dis)aggregation function is passed.
- Returns:
The mapped values.
- Return type:
np.array
- purge_cache_of_invalid_values() None#
- sample_person() dict#
Sample a person from the simulation. Returns a situation JSON with their inputs (including their containing entities).
- Returns:
A dictionary containing the person’s values.
- Return type:
dict
- set_input(variable_name: str, period: Period, value: ArrayLike) None#
Set a variable’s value for a given period
- Parameters:
variable – the variable to be set
value – the input value for the variable
period – the period for which the value is setted
Example: >>> from policyengine_core.country_template import CountryTaxBenefitSystem >>> simulation = Simulation(CountryTaxBenefitSystem()) >>> simulation.set_input(‘age’, ‘2018-04’, [12, 14]) >>> simulation.get_array(‘age’, ‘2018-04’) array([12, 14], dtype=int32)
If a
set_inputproperty has been set for the variable, this method may accept inputs for periods not matching thedefinition_periodof the variable. To read more about this, check the documentation.On a branch (see
get_branch()), the input also drops what the branch holds that may have been calculated from the value it replaces, so what the branch calculates next uses the input, as a simulation given the input before calculating anything would. Every value is stored after everything it was calculated from, and each simulation records the first stores its values may have been calculated from (seepolicyengine_core.data_storage.store_history): its own, its parent’s when it was created, and those of simulations its formulas calculated in. The branch drops each value it holds, other than an input, stored at or after the earliest recorded store ofvariable_namefor a period that shares a day withperiod, or the earliest recorded value ofvariable_nameuprated or carried over from another period (or given the default for want of one), or the first value that may depend on anything (restored from a dump, or calculated from a macro-cache read). If there is none of these, nothing it holds depends on the value and nothing is dropped. That is the case when a formula creates the branch while still calculating the variable it overrides, unless another branch it created for the same comparison already returned a value calculated from the variable; then only what came back from there, and what was calculated after, is dropped.Once an input is set on it, the branch stops reading macro-cache files, which are keyed by branch name and period but not by inputs. Each drop counts as an input change of the branch. A calculation that may have read a value from before the change, running in the branch or in any simulation that got a value from it, is not kept; the outermost such calculation in each simulation runs again until it reads nothing changed since (at most ten times; after that its result is returned but not kept), and an input stored for the very period it calculates after it began is its result. Setting an input to the value the branch already reads for that period, as an input, drops nothing.
What this does not track: a branch given a different tax-benefit system or parameters (call
drop_computed_arrays()on it); formulas that write into an array they read instead of returning a new one; formulas that test whether a value is stored (get_known_periods,get_array) or read another simulation’s storage directly, rather than calculating; a simulation other than the formula’s own branches calculated from a thread the formula starts without copying its context; and branches a formula keeps between calls, which hold what their parent held when they were created. On any simulation, input helpers remove overlapping calculated values of the variable they write. Dependency invalidation runs only on branches; branches already created from the branch keep their values; and a value another simulation calculated from the branch and kept stays when the branch’s input changes after that calculation ended.
- start_instant: str = None#
- subsample(n=None, frac=None, seed=None, time_period=None, quantize_weights: bool = True) Simulation#
Quantize the simulation to a smaller size by sampling households.
Rebuild from the holder’s recorded input tiers, including inputs of formula-backed variables. Calculate household weights for
time_periodto choose the sample, falling back to dataset-period input weights when that period has no input, formula, uprating or carry-over source. Rescale each input weight column to preserve its own entity’s total using the memberships exported for the rebuild. Calculated weights and other formula results are recomputed on the sample. Calculation branches are recreated on its populations; baseline policy is kept. If structural inputs are absent, loaded population IDs, memberships and roles supply the leaves needed to rebuild the same partition. This also supports structural variables whose IDs have formulas.- Parameters:
n (int, optional) – The number of households to sample. Defaults to 10_000.
frac (float, optional) – The fraction of households to sample. Defaults to None.
seed (int, optional) – The key used to seed the random number generator. Defaults to the dataset name.
time_period (str, optional) – Sample households based on their weight in this time period. Defaults to the default calculation period.
- Returns:
The quantized simulation.
- Return type:
- to_input_dataframe(include_computed_variables: bool = False) DataFrame#
Exports a DataFrame that can be loaded back into a new Simulation.
By default, only structurally input variables populated through
set_inputare exported from their recorded storage tiers. This avoids serializing pseudo-inputs and stale calculated values that would override formulas when reloaded.- Parameters:
include_computed_variables – If
True, export every variable with a known period, matching the historical unsafe behavior.- Returns:
The DataFrame containing the input values.
- Return type:
pd.DataFrame
- to_input_dict(include_computed_variables: bool = False) dict#
Exports a dictionary that can be loaded back into a new Simulation.
By default, only structurally input variables populated through
set_inputare exported from their recorded storage tiers. This avoids serializing pseudo-inputs and stale calculated values that would override formulas when reloaded.- Parameters:
include_computed_variables – If
True, export every variable with a known period, matching the historical unsafe behavior.- Returns:
The dictionary containing the input values.
- Return type:
dict
- property trace: bool#
- property true_input_variables: List[str]#
Stored variables that are safe to reload as source inputs.
Microsimulation#
- class policyengine_core.simulations.microsimulation.Microsimulation(tax_benefit_system: TaxBenefitSystem = None, populations: Dict[str, Population] = None, situation: dict = None, dataset: Union[str, Type[Dataset]] = None, reform: Reform = None, trace: bool = False, default_input_period: str = None, default_calculation_period: str = None)[source]#
Bases:
SimulationA Simulation whose entities use weights to represent larger populations.
- baseline = None#
- build_from_dataset() None#
Build a simulation from a dataset.
Entity-specific role columns take precedence over a generic
rolecolumn, including period-suffixed entity role columns in flat files. Without either, each person receives the literaldefault_rolekey. Only flattened-role keys match: unmatched and compound keys remain unrecognized rather than selecting a first role or subrole.A recognized implicit default with a finite role maximum must fit every group’s membership count. Otherwise an explicit role column is required. Explicit dataset role columns do not validate role capacity.
default_role=Nonerequires a role column.
- build_from_populations(populations: Dict[str, Population]) None#
This method of initialisation requires the populations to be pre-initialised.
- Parameters:
populations (Dict[str, Population]) – A dictionary of populations, indexed by entity key.
- calculate(variable_name: str, period: Period = None, map_to: str = None, use_weights: bool = True, decode_enums: bool = True) MicroSeries[source]#
Calculate
variable_nameforperiod.- Parameters:
variable_name (str) – The name of the variable to calculate.
period (Period) – The period to calculate the variable for.
map_to (str) – The name of the variable to map the result to. If None, the result is returned as is.
decode_enums (bool) – If True, the result is decoded from an array of integers to an array of strings.
- Returns:
The calculated variable.
- Return type:
ArrayLike
- calculate_add(variable_name: str, period: Period = None, map_to: str = None, use_weights: bool = True) MicroSeries[source]#
- calculate_dataframe(variable_names: list, period: Period = None, map_to: str = None, use_weights: bool = True) MicroDataFrame[source]#
Calculate
variable_namesforperiod.- Parameters:
variable_names (List[str]) – A list of variable names to calculate.
period (Period) – The period to calculate for.
- Returns:
A dataframe containing the calculated variables.
- Return type:
pd.DataFrame
- calculate_divide(variable_name: str, period: Period = None, map_to: str = None, use_weights: bool = True) MicroSeries[source]#
- calculate_output(variable_name: str, period: Period = None) ArrayLike#
Calculate the value of a variable using the
calculate_outputattribute of the variable.
- check_macro_cache(variable_name: str, period: str) bool#
Check if the variable is able to have cached value
- clone(debug: bool = False, trace: bool = False, clone_tax_benefit_system: bool = True) Simulation#
Copy the simulation just enough to be able to run the copy without modifying the original simulation.
The copy records its own cache invalidations (
invalidated_caches, starting from this simulation’s pending ones). Ifbaselineis a branch of this simulation, as a reform simulation’s is, the copy gets a copy of that branch as its own: a branch of the copy, under the same name, with this baseline’s tax-benefit system, traced in the copy if this baseline is traced in this simulation. Any otherbaselineis shared: a branch’s is its parent’s, and a country package that builds a separate baseline simulation copies it, if it needs to, in its ownclone.Every cached array is copied, except in the first
cloneof this simulation made whileget_branchis creating a branch of it: that copy shares the arrays until it reads them (seeget_branch()). A subclass’sclonethat calls this one throughsuper().clonetakes part in that the same way. A subclassclonethat first clones the same simulation directly gets the sharing in that direct clone instead, and its branch is a full copy.The copy’s method aliases (
calc,df, and any other bound method of this simulation kept on the instance) are bound to the copy. Withclone_tax_benefit_system, the copy of the system names the copy as its simulation and the copy’s populations use its entities, so nothing in the copy refers back to this simulation through them. Without it, the copy shares this simulation’s system, which still names this simulation.
- create_shortcuts() None#
- property data_storage_dir: str#
Folder in which this simulation stores values on disk when memory is short (see
MemoryConfig).Set
_data_storage_dirto choose the folder: it is created on first use if it does not exist. Nothing removes it, unless it is in a temporary folder a simulation made, which this simulation then keeps alive, and which removes it with everything else in it once nothing keeps it. Otherwise this is a new temporary folder, removed once this simulation, every disk storage in the folder (those its clones and branches copied included), every simulation given a folder in it and every temporary folder made in it are garbage-collected in this process, or at interpreter exit, except the subfolders of disk storages that preserve theirs. In a process forked from the one the folder is for, this is a new folder for that process, made inside that one. See thestorage_directorymodule for what this guarantees.
- datasets = []#
- default_calculation_period = None#
- default_dataset = None#
- default_input_period = None#
- default_role = 'member'#
- default_tax_benefit_system = None#
- default_tax_benefit_system_instance = None#
- delete_arrays(variable: str, period: Period = None) None#
Delete a variable’s values visible to this simulation branch.
The calling branch, each ancestor branch, and the default branch are purged from this simulation’s private holder storage. Other branch names and the parent simulation’s holder storage remain unchanged. Deleted inputs stop counting as inputs: a value calculated later for the same period is a formula result, which
apply_reformdiscards andto_input_dataframedoes not export.- Parameters:
variable – the variable whose cached values should be deleted
period – the period to delete, or all periods when omitted
Example:
>>> from policyengine_core.country_template import CountryTaxBenefitSystem >>> simulation = Simulation(CountryTaxBenefitSystem()) >>> simulation.set_input('age', '2018-04', [12, 14]) >>> simulation.set_input('age', '2018-05', [13, 14]) >>> simulation.get_array('age', '2018-05') array([13, 14], dtype=int32) >>> simulation.delete_arrays('age', '2018-05') >>> simulation.get_array('age', '2018-04') array([12, 14], dtype=int32) >>> simulation.get_array('age', '2018-05') is None True >>> simulation.set_input('age', '2018-05', [13, 14]) >>> simulation.delete_arrays('age') >>> simulation.get_array('age', '2018-04') is None True >>> simulation.get_array('age', '2018-05') is None True
- derivative(variable: str, wrt: str, period: Period = None, delta: float = 1) ArrayLike#
Compute the derivative of a variable w.r.t another variable.
- Parameters:
variable (str) – The variable to differentiate.
wrt (str) – The variable to differentiate with respect to.
period (Period) – The period for which to compute the derivative.
delta (float) – The infinitesimal to use for the derivative.
- Returns:
The derivative.
- Return type:
ArrayLike
- describe_entities() dict#
- drop_computed_arrays() int#
Delete every value this simulation holds except inputs.
Inputs are the values stored through
set_input: the dataset or situation the simulation was built from, inputs set on it and, for a branch, inputs set on the simulations it was created from before it was created. Values a customset_inputhandler calculates are not inputs. Every other value is calculated again when next requested, and the simulation stops reading macro-cache files.Use this on a branch whose tax-benefit system or parameters differ from its parent’s, whose values the branch would otherwise inherit;
set_inputon a branch drops what depends on the input by itself. Branches already created from this simulation keep their values.- Returns:
The number of arrays deleted.
- Return type:
int
- extract_person(index: int = 0, exclude_entities: tuple = ('state',)) dict#
Extract a person from the simulation. Returns a situation JSON with their inputs (including their containing entities).
- Parameters:
index (int) – The index of the person to extract.
- Returns:
A dictionary containing the person’s values.
- Return type:
dict
- get_array(variable_name: str, period: Period) ArrayLike#
Return the value of
variable_nameforperiod, if this value is alreay in the cache (if it has been set as an input or previously calculated).Unlike
calculate(), this method does not trigger calculations and does not use any formula.
- get_branch(name: str = 'branch', clone_system: bool = False) Simulation#
Create a clone of this simulation, whose calculations are traced in the original.
The branch starts from the values this simulation has cached when the branch is created. It does not copy them up front: each of the branch’s holders gets its own index of this simulation’s arrays, and copies an array the first time the branch reads it. A numpy array the branch never reads is never copied (masked arrays, and values that are not numpy arrays, are copied when the branch is created).
What the branch stores (
set_input, calculations, deletions) goes into its own index only, and what it reads is its own copy, so nothing done through the branch changes this simulation’s values. What this simulation stores after branching stays out of the branch.The one difference from copying every array up front: code that writes in place into one of this simulation’s cached arrays (
array[mask] = 0orarray += 1, instead of storing a new array withset_input) after branching also changes the value the branch reads, if the branch has not read that array yet.As with the rest of a simulation, a branch is not safe to read from several threads at once: two first reads of the same array can each make a copy.
A new branch starts with the values this simulation holds; asked for a name it already has, this returns that branch as it is. An input set on the branch drops those that may have been calculated from the value it replaces (see
set_input()). A branch whose tax-benefit system or parameters are changed should calldrop_computed_arrays()before calculating.- Parameters:
name (str, optional) – Name of the branch. Defaults to “branch”.
clone_system (bool, optional) – Whether to clone the tax-benefit system. Use this if you’re changing policy parameters. Defaults to False.
- Returns:
The cloned simulation.
- Return type:
- get_holder(variable_name: str) Holder#
Get the
Holderassociated with the variablevariable_namefor the simulation
- get_input_variables(include_computed_variables: bool = True) List[str]#
Return variable names stored as inputs on this simulation.
- Parameters:
include_computed_variables – When
True, return the legacy runtime list of variables with stored values. WhenFalse, return only structurally input variables that were populated throughset_inputon the current branch.- Returns:
Stored input variable names.
- Return type:
List[str]
- get_known_periods(variable: str) List[Period]#
Get a list variable’s known period, i.e. the periods where a value has been initialized and
- Parameters:
variable – the variable to be set
Example:
>>> from policyengine_core.country_template import CountryTaxBenefitSystem >>> simulation = Simulation(CountryTaxBenefitSystem()) >>> simulation.set_input('age', '2018-04', [12, 14]) >>> simulation.set_input('age', '2018-05', [13, 14]) >>> simulation.get_known_periods('age') [Period((u'month', Instant((2018, 5, 1)), 1)), Period((u'month', Instant((2018, 4, 1)), 1))]
- get_memory_usage(variables: List[str] = None) dict#
Get data about the virtual memory usage of the simulation
- get_population(plural: str = None) Population#
- get_variable_population(variable_name: str) Population#
- get_weights(variable_name: str, period: Period, map_to: str = None) Sequence[T][source]#
Return weights for a variable’s entity from
household_weight.
- invalidate_spiral_variables(variable: str) None#
- is_over_dataset = False#
- link_to_entities_instances() None#
- macro_cache_read = False#
- macro_cache_write = False#
- map_result(values: ArrayLike, source_entity: str, target_entity: str, how: str = None)#
Maps values from one entity to another.
- Parameters:
arr (np.array) – The values in their original position.
source_entity (str) – The source entity key.
target_entity (str) – The target entity key.
how (str, optional) – A function to use when mapping. Defaults to None.
- Raises:
ValueError – If an invalid (dis)aggregation function is passed.
- Returns:
The mapped values.
- Return type:
np.array
- purge_cache_of_invalid_values() None#
- sample_person() dict#
Sample a person from the simulation. Returns a situation JSON with their inputs (including their containing entities).
- Returns:
A dictionary containing the person’s values.
- Return type:
dict
- set_input(variable_name: str, period: Period, value: ArrayLike) None#
Set a variable’s value for a given period
- Parameters:
variable – the variable to be set
value – the input value for the variable
period – the period for which the value is setted
Example: >>> from policyengine_core.country_template import CountryTaxBenefitSystem >>> simulation = Simulation(CountryTaxBenefitSystem()) >>> simulation.set_input(‘age’, ‘2018-04’, [12, 14]) >>> simulation.get_array(‘age’, ‘2018-04’) array([12, 14], dtype=int32)
If a
set_inputproperty has been set for the variable, this method may accept inputs for periods not matching thedefinition_periodof the variable. To read more about this, check the documentation.On a branch (see
get_branch()), the input also drops what the branch holds that may have been calculated from the value it replaces, so what the branch calculates next uses the input, as a simulation given the input before calculating anything would. Every value is stored after everything it was calculated from, and each simulation records the first stores its values may have been calculated from (seepolicyengine_core.data_storage.store_history): its own, its parent’s when it was created, and those of simulations its formulas calculated in. The branch drops each value it holds, other than an input, stored at or after the earliest recorded store ofvariable_namefor a period that shares a day withperiod, or the earliest recorded value ofvariable_nameuprated or carried over from another period (or given the default for want of one), or the first value that may depend on anything (restored from a dump, or calculated from a macro-cache read). If there is none of these, nothing it holds depends on the value and nothing is dropped. That is the case when a formula creates the branch while still calculating the variable it overrides, unless another branch it created for the same comparison already returned a value calculated from the variable; then only what came back from there, and what was calculated after, is dropped.Once an input is set on it, the branch stops reading macro-cache files, which are keyed by branch name and period but not by inputs. Each drop counts as an input change of the branch. A calculation that may have read a value from before the change, running in the branch or in any simulation that got a value from it, is not kept; the outermost such calculation in each simulation runs again until it reads nothing changed since (at most ten times; after that its result is returned but not kept), and an input stored for the very period it calculates after it began is its result. Setting an input to the value the branch already reads for that period, as an input, drops nothing.
What this does not track: a branch given a different tax-benefit system or parameters (call
drop_computed_arrays()on it); formulas that write into an array they read instead of returning a new one; formulas that test whether a value is stored (get_known_periods,get_array) or read another simulation’s storage directly, rather than calculating; a simulation other than the formula’s own branches calculated from a thread the formula starts without copying its context; and branches a formula keeps between calls, which hold what their parent held when they were created. On any simulation, input helpers remove overlapping calculated values of the variable they write. Dependency invalidation runs only on branches; branches already created from the branch keep their values; and a value another simulation calculated from the branch and kept stays when the branch’s input changes after that calculation ended.
- start_instant = None#
- subsample(n=None, frac=None, seed=None, time_period=None, quantize_weights: bool = True) Simulation#
Quantize the simulation to a smaller size by sampling households.
Rebuild from the holder’s recorded input tiers, including inputs of formula-backed variables. Calculate household weights for
time_periodto choose the sample, falling back to dataset-period input weights when that period has no input, formula, uprating or carry-over source. Rescale each input weight column to preserve its own entity’s total using the memberships exported for the rebuild. Calculated weights and other formula results are recomputed on the sample. Calculation branches are recreated on its populations; baseline policy is kept. If structural inputs are absent, loaded population IDs, memberships and roles supply the leaves needed to rebuild the same partition. This also supports structural variables whose IDs have formulas.- Parameters:
n (int, optional) – The number of households to sample. Defaults to 10_000.
frac (float, optional) – The fraction of households to sample. Defaults to None.
seed (int, optional) – The key used to seed the random number generator. Defaults to the dataset name.
time_period (str, optional) – Sample households based on their weight in this time period. Defaults to the default calculation period.
- Returns:
The quantized simulation.
- Return type:
- to_input_dataframe(include_computed_variables: bool = False) DataFrame#
Exports a DataFrame that can be loaded back into a new Simulation.
By default, only structurally input variables populated through
set_inputare exported from their recorded storage tiers. This avoids serializing pseudo-inputs and stale calculated values that would override formulas when reloaded.- Parameters:
include_computed_variables – If
True, export every variable with a known period, matching the historical unsafe behavior.- Returns:
The DataFrame containing the input values.
- Return type:
pd.DataFrame
- to_input_dict(include_computed_variables: bool = False) dict#
Exports a dictionary that can be loaded back into a new Simulation.
By default, only structurally input variables populated through
set_inputare exported from their recorded storage tiers. This avoids serializing pseudo-inputs and stale calculated values that would override formulas when reloaded.- Parameters:
include_computed_variables – If
True, export every variable with a known period, matching the historical unsafe behavior.- Returns:
The dictionary containing the input values.
- Return type:
dict
- property trace: bool#
- property true_input_variables: List[str]#
Stored variables that are safe to reload as source inputs.
SimulationBuilder#
- class policyengine_core.simulations.simulation_builder.SimulationBuilder#
Bases:
object- add_group_entity(persons_plural: str, persons_ids: ArrayLike, entity: Entity, instances_json: dict) None#
Add all instances of one of the model’s entities as described in
instances_json.
- add_parallel_axis(axis)#
- add_perpendicular_axis(axis)#
- add_person_entity(entity: Entity, instances_json: dict) List[int]#
Add the simulation’s instances of the persons entity as described in
instances_json.
- add_variable_value(entity, variable, instance_index, instance_id, period_str, value)#
- build(tax_benefit_system: TaxBenefitSystem) Simulation#
- build_default_simulation(tax_benefit_system: TaxBenefitSystem, count: int = 1, simulation: Simulation = None) Simulation#
- Build a simulation where:
There are
countpersonsThere are
countinstances of each group entity, containing one personEvery person has, in each entity, the first role
- build_from_dict(tax_benefit_system: TaxBenefitSystem, input_dict: dict, simulation: Simulation = None) Simulation#
Build a simulation from
input_dict. Optionally overwrites an existing simulation.This method uses
build_from_entitiesif entities are fully specified, orbuild_from_variablesif not.
- build_from_entities(tax_benefit_system: TaxBenefitSystem, input_dict: dict, simulation: Simulation = None) Simulation#
Build a simulation from a Python dict
input_dictfully specifying entities.Examples:
>>> simulation_builder.build_from_entities({ 'persons': {'Javier': { 'salary': {'2018-11': 2000}}}, 'households': {'household': {'parents': ['Javier']}} })
- build_from_variables(tax_benefit_system: TaxBenefitSystem, input_dict: dict, simulation: Simulation = None) Simulation#
Build a simulation from a Python dict
input_dictdescribing variables values without expliciting entities.This method uses
build_default_simulationto infer an entity structureExample:
>>> simulation_builder.build_from_variables( {'salary': {'2016-10': 12000}} )
- check_persons_to_allocate(persons_plural, entity_plural, persons_ids, person_id, entity_id, role_id, persons_to_allocate, index)#
- create_entities(tax_benefit_system: TaxBenefitSystem) None#
- declare_entity(entity_singular: str, entity_ids: Iterable) Population#
- declare_person_entity(person_singular: str, persons_ids: Iterable) None#
- expand_axes()#
- explicit_singular_entities(tax_benefit_system: TaxBenefitSystem, input_dict: dict) dict#
Preprocess
input_dictto explicit entities defined using the single-entity shortcutExample:
>>> simulation_builder.explicit_singular_entities( {'persons': {'Javier': {}, }, 'household': {'parents': ['Javier']}} ) >>> {'persons': {'Javier': {}}, 'households': {'household': {'parents': ['Javier']}}
- finalize_variables_init(population)#
- get_count(entity_name)#
- get_ids(entity_name)#
- get_input(variable: str, period_str: str) Any#
- get_memberships(entity_name)#
- get_roles(entity_name)#
- get_variable_entity(variable_name)#
- init_variable_values(entity, instance_object, instance_id)#
- join_with_persons(group_population: GroupPopulation, persons_group_assignment: ArrayLike, roles: Iterable[str]) None#
Join persons to the groups named by their membership IDs.
Memberships and roles are supplied in person order. Each membership must match a unique declared group ID by value; equal numeric IDs may have different dtypes.
members_entity_idindexes the declared group order, including groups with no members. An empty assignment is supported.rolesmust be one dimensional and match the membership array’s length. Integers index the entity’s flattened roles; strings match flattened-role keys literally, with unmatched keys represented as unrecognized roles. This method does not validate role capacity.- Raises:
ValueError – if roles are not one dimensional with one entry per membership, declared group IDs repeat, or a membership cannot be matched by value to a declared group.
- raise_period_mismatch(entity, json, e)#
- register_variable(variable_name, entity)#
- set_default_period(period_str: str) None#