How to evaluate an intervention's impact

Paid Feature - Only available for some customers

This feature is only available to certain customers. If you are interested in using this feature, please contact Zepben for more information.

This guide covers how to assess whether an intervention had the intended effect, by comparing an intervention work package against its parent work package. It is not a fixed procedure: the right zones, metrics and checks depend on your study, so use the parts that fit. It assumes you have already run both work packages - see How to run an Intervention Work Package.

The comparison mostly uses network_performance_metrics_enhanced, and for the asset upgrade interventions also the asset level summary tables.

Checking the work packages are comparable

Check these before drawing any conclusions. If either of the first two fails, differences in the results may not come from the intervention at all.

• Compare applied interventions against candidates. A candidate that cannot be allocated is skipped without an error. See Check how many candidates were actually applied.
• Confirm the two work packages differ only in the intervention you want to measure. Same load time period, feeders, years, scenarios, meter placement and model configuration. Also check the intervention block itself: a candidate intervention can be stacked with loadReshaping, dvms or phaseRebalanceProportions, and those change results across the whole feeder. If they are stacked, the comparison measures all of them together.
• Check that zones without the intervention did not change. For a local intervention (anything except DVMS, LOAD_RESHAPING and PHASE_REBALANCING), the Local metrics load_kwh and generation_kwh in zones that received nothing should be essentially identical between the two work packages. If many zones differ, something other than the intervention differs, such as the network model or load.
• Start with the whole-year rows (season = 'yearly', time_of_day = 'all'). Candidate generation uses these rows, so they line up with how the intervention was targeted. Season and time-of-day rows can then show when the change happens.

Finding what each intervention targeted

Each applied intervention table (applied_community_bess, applied_lv_statcom, applied_oltc, applied_tap_optimization, applied_line_upgrade, applied_transformer_upgrade) shares work_package_id, scenario and rank with intervention_candidates, which is in the input schema (input.intervention_candidates). Both use the intervention work package's ID. Join on those (plus intervention_class) to find the candidate behind each applied intervention.

What the candidate's conducting_equipment_mrid points to depends on the intervention type:

Intervention	Candidate conducting_equipment_mrid is	Where to measure
COMMUNITY_BESS, LV_STATCOMS, DISTRIBUTION_TX_OLTC, DISTRIBUTION_TAP_OPTIMIZATION	The head of the targeted transformer zone	That zone in network_performance_metrics_enhanced
THERMAL_TRANSFORMER_UPGRADE	The upgraded transformer, which is also the head of its transformer zone	asset_level_thermal_loading_summary for the transformer, and its zone in network_performance_metrics_enhanced
THERMAL_LINE_UPGRADE	The upgraded line, as it appears in the asset level summaries	asset_level_thermal_loading_summary for that line
VOLTAGE_LINE_UPGRADE	The upgraded line	The transformer zones listed under measurement_zone_breakdown[].transformer_mrid in the candidate's info, and asset_level_voltage_summary for the line

For COMMUNITY_BESS and LV_STATCOMS, original_equipment_mrid in the applied table is the equipment next to the connection point, not the zone head. Use the candidate join to find the zone.

DVMS, LOAD_RESHAPING and PHASE_REBALANCING are systemic: they have no candidates or applied tables and affect every zone on the feeder.

What to compare

Zones

Intervention work packages always use transformer measurement zones, with an optional feeder head zone. Which zones you look at depends on the question:

Zones	Question it answers
The targeted zones	Did the intervention fix what it was chosen for?
All transformer zones on the affected feeders	What is the total benefit, and did anything get worse elsewhere? Interventions shift flows and voltages beyond the zone they sit in.
Feeder head zone	What changed on the MV network? Its Local metrics barely move for an LV intervention, because it covers only the MV network and MV-connected customers. Its Through-flow metrics (peak_import, peak_export, MV thermal) show the upstream effect.
Asset level summaries	Did the specific asset stop overloading? Best for line and transformer upgrades, because a zone's thermal metrics follow only the single most overloaded asset in each interval.

Metrics

A suggested starting point for each intervention type. The primary metric is the one the intervention was ranked or selected on, so a reduction there indicates it addressed what it was picked for. See the Scope column in network_performance_metrics_enhanced for what each metric counts.

Intervention	Suggested primary metric	Also worth checking
COMMUNITY_BESS	load_exceeding_normal_thermal_voltage_vcr + gen_exceeding_normal_thermal_voltage_cecv	Thermal kWh, hours and *_thermal_max_kw; peak_import and peak_export; voltage hours
LV_STATCOMS	load_exceeding_normal_voltage_vcr + gen_exceeding_normal_voltage_cecv	Voltage hours, v1_avg_section_voltage, v99_avg_section_voltage
DISTRIBUTION_TX_OLTC	load_exceeding_normal_voltage_vcr + gen_exceeding_normal_voltage_cecv	Voltage hours, v1 and v99
DISTRIBUTION_TAP_OPTIMIZATION	v1_avg_section_voltage (boost) or v99_avg_section_voltage (buck) moving toward nominal	Voltage over and under limit hours
THERMAL_TRANSFORMER_UPGRADE	asset_level_thermal_loading_summary for the transformer: overload_kwh_import, overload_kwh_export, max_loading_pct	hours_over_normal; the zone's thermal metrics
THERMAL_LINE_UPGRADE	asset_level_thermal_loading_summary for the line: overload_kwh_import, overload_kwh_export, max_loading_pct	hours_over_normal; max_loading_pct against targetLoadingPct
VOLTAGE_LINE_UPGRADE	Voltage swing (maximum_section_voltage - minimum_section_voltage) and voltage_delta_max in the listed transformer zones	load_exceeding_normal_thermal_voltage_vcr + gen_exceeding_normal_thermal_voltage_cecv (the value apportioned to candidates)
DVMS	Voltage hours and load_ / gen_exceeding_normal_voltage_* across all transformer zones	Feeder head zone voltages
LOAD_RESHAPING	Thermal and voltage metrics in the targeted time_of_day rows	The same metrics in the other time_of_day rows, and load_max_timestamp
PHASE_REBALANCING	voltage_delta_avg, voltage_delta_max	Thermal metrics. There is no phase imbalance metric in the NPME table; use per-phase values in energy_meter_voltages_raw

COMMUNITY_BESS

The battery's own energy is not in load_kwh, generation_kwh, load_max_kw or gen_max_kw, so its effect shows up in peak_import, peak_export, thermal overload and voltages instead. See COMMUNITY_BESS for how it is modelled. Compare rated_capacity in applied_community_bess against min_bess_maxP_kw x capacity_scaling_factor in the candidate's info: an undersized battery is applied with a warning and can only remove part of the overload. Compare the time_of_day rows too, as a battery charging at the wrong time can make one period worse while improving another.

LV_STATCOMS

STATCOMs only exchange reactive power, so expect no thermal benefit. Thermal metrics can rise slightly, because reactive current adds to line current.

DISTRIBUTION_TX_OLTC and DISTRIBUTION_TAP_OPTIMIZATION

Both change tap ratio only, so there is no thermal effect to look for. Tap optimisation applies one static change per year, so a zone whose voltage issue varies within the year may improve in one season and worsen in another. Check the season rows.

THERMAL_TRANSFORMER_UPGRADE and THERMAL_LINE_UPGRADE

Both need assetLevelSummaries with thermalLoading=True on the intervention work package as well as the parent. Use the asset level summary as the main evidence: if another asset in the zone is also overloaded, the zone metric may barely move after the upgrade.

VOLTAGE_LINE_UPGRADE

Candidates are ranked by value recovered per metre, not voltage severity, so check the voltage figures zone by zone rather than relying on rank.

DVMS, LOAD_RESHAPING and PHASE_REBALANCING

These act on the whole feeder, so comparing all zones is more informative than a targeted subset. LOAD_RESHAPING moves energy in time rather than removing it, so compare the time_of_day rows outside the targeted period as well.

Comparing the two work packages

There are two ways to get a row-by-row comparison.

Generate a diff in the UI. This writes the differences to network_performance_metrics_enhanced_diff. Every metric is work package 1 minus work package 2, in the order you pick them, so pick the parent first. Then a positive value means the intervention reduced the metric. peak_export is negative, so a smaller export peak shows as a negative diff. The examples below assume the parent was picked first.

Join the two work packages yourself. Pair rows on scenario, feeder, conducting_equipment_mrid, terminal_sequence_number, season, time_of_day and timestamp. Do not pair on measurement_zone_name: it is generated fresh for each run, so the same zone has a different name in each work package. For the forecast year, use link_to_input rather than timestamp, which is in UTC.

-- Example: Compare a metric between the parent and intervention work packages, whole year
SELECT feeder,
       conducting_equipment_mrid,
       SUM(CASE WHEN work_package_id = 'yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy'
                THEN load_exceeding_normal_thermal_voltage_vcr END) AS parent_vcr,
       SUM(CASE WHEN work_package_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
                THEN load_exceeding_normal_thermal_voltage_vcr END) AS intervention_vcr
FROM network_performance_metrics_enhanced
WHERE work_package_id IN ('yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy', 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')
  AND season = 'yearly'
  AND time_of_day = 'all'
GROUP BY scenario, feeder, conducting_equipment_mrid, terminal_sequence_number, season, time_of_day, timestamp;

Rolling up results

The rules for adding metrics across zones apply equally to their differences:

• Local and Mixed metrics (energy and $): can be summed across transformer zones.
• Through-flow kWh, $, hours and events: can be summed across transformer zones. Do not add the feeder head to the zones beneath it, as that mixes measurements of the same power flow.
• Peaks (*_max_kw, peak_import, peak_export): do not sum. Zones peak at different times.
• Voltages: take the max, min or a percentile rather than a sum.

The full table is under Can I add it up across zones? in the network_performance_metrics_enhanced row by row breakdown. Reporting the targeted zones and the all-zone total separately can help: a large improvement in the targeted zones alongside a smaller all-zone total means part of the problem moved elsewhere.

Common pitfalls

• Candidates skipped silently. Always check applied against candidates before reading any result.
• Config drift or stacked interventions. Any difference besides the intervention you are measuring, including a stacked load reshape, contaminates the comparison. Check zones that received nothing are unchanged.
• Undersized interventions. A battery smaller than its candidate's requirement is applied anyway, with a warning in logs.
• Chained work packages. A work package chained from another intervention work package contains only its own interventions. Compare it against the non-intervention work package the chain started from, not the intervention work package it was chained from.
• Pairing on measurement_zone_name. It changes every run; pair on the head asset and terminal instead.
• Year from timestamp. Use link_to_input.year.
• Diff sign. The diff table is work package 1 minus work package 2, in the order picked when generating it. peak_export is negative, so a reduction shows as a negative diff.
• load_ and gen_ thermal swap. If an intervention flips the net flow direction at a zone head, overload moves between the load_ and gen_ columns. Compare their sum as well as each one.
• Zone thermal metrics follow one asset. In each interval only the single most overloaded asset in the zone is recorded. Use asset level summaries for asset upgrades.
• Adding feeder head and transformer zones together, or summing peaks. See Rolling up results.
• Community battery energy. The battery's own energy is not counted in load_kwh, generation_kwh, load_max_kw or gen_max_kw, so do not use changes in those as evidence of what the battery did.
• Feeder head Local metrics. With transformer zones on, these cover only MV-connected customers.
• Thermal hours and events depend on config. They are only populated when the matching calculate...Thermal options are enabled in Enhanced Metrics config. Zero hours with the option off does not mean there was no overload; check the thermal kWh metrics too.
• Nulls. Voltage-driven metrics are null where a zone had no voltage data, and a diff is null if either side is null.

Example: COMMUNITY_BESS

One possible approach for a COMMUNITY_BESS work package, using the queries below as a starting point.

1. Confirm what was applied.

-- Example: Candidates generated
SELECT COUNT(*)
FROM input.intervention_candidates
WHERE work_package_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
  AND intervention_class = 'COMMUNITY_BESS';

-- Example: Batteries actually applied
SELECT COUNT(*)
FROM applied_community_bess
WHERE work_package_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';

2. Compare the targeted zones, whole year, using the diff table generated in the UI.

-- Example: Change in value at risk for the targeted zones, whole year
SELECT feeder,
       conducting_equipment_mrid,
       load_exceeding_normal_thermal_voltage_vcr,
       gen_exceeding_normal_thermal_voltage_cecv
FROM network_performance_metrics_enhanced_diff
WHERE diff_id = 'zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz'
  AND season = 'yearly'
  AND time_of_day = 'all'
  AND conducting_equipment_mrid IN (
    SELECT conducting_equipment_mrid
    FROM input.intervention_candidates
    WHERE work_package_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
      AND intervention_class = 'COMMUNITY_BESS'
  );

The diff has one row per year, so a battery applied in a later year shows no change in the years before year_applied.

3. Total across all transformer zones on the same feeders, to capture side effects.

-- Example: Change in value at risk per feeder, all transformer zones
SELECT feeder,
       SUM(load_exceeding_normal_thermal_voltage_vcr + gen_exceeding_normal_thermal_voltage_cecv) AS value_at_risk_reduction
FROM network_performance_metrics_enhanced_diff
WHERE diff_id = 'zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz'
  AND season = 'yearly'
  AND time_of_day = 'all'
  AND conducting_equipment_mrid IN (
    SELECT conducting_equipment_mrid
    FROM network_performance_metrics_enhanced
    WHERE work_package_id = 'yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy'
      AND mz_type = 'TRANSFORMER'
  )
GROUP BY feeder;

4. Check the time-of-day rows (repeat query 2 without the time_of_day = 'all' filter) to see whether the battery's charging made any period worse.