Input Files
Required Input Files
The following files are required for the GIS Network Extractor to run:
File(s) | Data type | Default location |
---|---|---|
Processor Config | Single .json file | ./processor-config.json |
Cable Types | Single .json file | ./cable-types.json |
Network Hierarchy | Single .json file | ./network-hierarchy.json |
Nodes Config | Single .json file | ./nodes-config.json |
Poles | Single .json file | ./poles.json |
GIS Network files | Single or multiple .json/.geojson files | ./gis |
Optional Files
The following files may be provided as input for the GIS Network Extractor, however they are not required:
File(s) | Data type | Default location |
---|---|---|
Meter Mapping | Single or multiple CSV files | ./meter_mappings |
Substation Transformer Tap Settings | Single .csv file | none |
Line Regulator Settings | Single .csv file | none |
Voltage Regulator Settings | Single .csv file | none |
Source Impedance | Single .csv file | none |
Tap Settings | Single CSV file | none |
Meter Mapping File
CPPAL
The GIS Network Extractor takes meter mapping files to provide a mapping of meters and supply points to transformers. These files use CSV format and contain the following columns: company, nmi,meter_id, supply_point_id, meter_no, transformer_id, transformer_desc, feeder, service_addr, cust_class, cust_tariff. Each file is required to have its date in the file name so the appropriate mapping file can be used.
Sample content:
company,nmi,meter_id,supply_point_id,meter_no,transformer_id,transformer_desc,feeder,service_addr,cust_class,cust_tariff
"ACME","FOOH1234567",123456789,11929433,"1234",,,"LP001","140 THISSON STREET,SPRINGFIELD,SPF,12345A","C","T1"
"ACME","FOOH1234567",123456789,11929433,"1234",,,"LP002","140 OTHERSON STREET,SPRINGFIELD,SPF,12345A","R","T1"
"ACME","FOOH1234567",123456789,11929533,"1234",,,"LP003","LEVEL44,76 WOOP WOOP STREET,SPRINGFIELD,SPF,12345A","I","T2"
"ACME","FOOH1234567",123456789,11929538,"1234",,,"LP004","WEST TOWER,76 WOOP WOOP STREET,SPRINGFIELD,SPF,12345A","I","T1"
"ACME","FOOH1234567",123456789,11929538,"1234",,,"LP006","WEST TOWER,76 WOOP WOOP STREET,SPRINGFIELD,SPF,12345A","C","T1"
"ACME","FOOH1234567",123456789,11929538,"1234",,,"LP007","TENANCY2,OPP 76 WOOP WOOP STREET,SPRINGFIELD,SPF,12345A","I","T3"
"ACME","FOOH1234567",123456789,11929538,"1234",,,"LP008","TENANCY1,OPP 76 WOOP WOOP STREET,SPRINGFIELD,SPF,12345A","R","T3"
EE
The GIS Network Extractor takes transformer-premise-nmi mapping files to provide a mapping of meters and premises to transformers. These files use CSV format and contain the following columns: nmi,premnum,peace_substation,peace_zonesub,sad_feeder,sad_substation,sad_zonesub,dlf,dlf_class. Each file is required to have its date in the file name so the appropriate mapping file can be used.
Sample content:
nmi,premnum,peace_substation,peace_zonesub,sad_feeder,sad_substation,sad_zonesub,dlf,dlf_class
4123456781,1234561,PTX1,PZS1,SF1,STX1,ZS1,DLF1,DLFC1
4123456782,1234562,PTX2,ZS2,SF2,STX2,ZS2,DLF2,DLFC2
4123456783,1234563,PTX3,ZS3,SF3,STX3,ZS3,DLF3,DLFC3
Processor Config File
The processor configuration allows specifying whether some of the processing steps will be executed, as well as allows filtering of what data will be ingested and saved to the resulting database. If a config file is not specified or present, defaults will be as per below.
Structure:
{
# Whether meter data should be processed (if true a meter mappings file must be provided)
# Default: true
"processMeters": boolean,
# Whether to create meters and consumers from the meter mappings file for premises that don't have a corresponding transformer.
# Should only be used for limited scope network processing, otherwise it will suppress warnings about meter data quality.
# See processor configuration help for details.
# Default: true
"includeLonelyMeters": boolean,
# Whether the Sub-transmission network should be ingested (must be in a file name like SUB_TRANS_NETWORK.*.json
# Default: true
"includeSubtrans": boolean,
# HV filters (see below).
# Default: null
"hv": null | { # Exactly one of the following fields must be set
"include": ["<feeder_name_1>", "<feeder_name_2>", ...]
"exclude": ["<feeder_name_1>", "<feeder_name_2>", ...]
},
# Whether LV should be saved to the database.
# Default: true
"includeLV": boolean,
# Whether default values will be assigned for transformers and lines.
# Default: true
"assignDefaults": boolean,
# Default WireInfo to use for lines missing one (see below)
# Default: null
"defaultWireInfo": null | <WireInfo spec>,
# Default PerLengthSequenceImpedance to use for lines missing one (see below)
# Default: null
"defaultPerLengthSequenceImpedance": null | <PerLengthSequenceImpedance spec>,
# Default PowerTransformerInfo to use for power transformers missing one (see below)
# Default: null
"defaultPowerTransformerInfo": null | <PowerTransformerInfo spec>,
# Whether to run use case validation after generating output database
# Default: true
"validation": boolean,
# Whether to process PoF aliases as names. Requires PoF connection parameters to be passed in.
# Default: false
"processPofAliases": boolean,
# Path to a file containing the DER register.
# Default: null
"derRegisterPath": null | string,
# Path to a file containing zone substation transformer tap changer settings. CPPAL only.
# Default: null
"substationTransformerTapSettingsPath": null | string,
# Path to a file containing the zone substation regulator settings CSV. CPPAL and EE formats supported.
# Default: null
"regulatorSettingsPath": null | string,
# Path to a file containing the line regulator settings CSV. CPPAL only.
# Default: null
"lineRegulatorSettingsPath": null | string,
# Path to a file containing the source impedances CSV. CPPAL and EE formats supported.
# Default: null
"sourceImpedancesPath": null | string,
}
Filters and includeLV
Filters allow specifying which feeders get processed and stored in the result, however there are file name requirements that must be met for filters to work as intended. For the most efficient handling of data, separate files should exist for HV and LV network, and they must be named accordingly in the form below:
HV Files: <feeder_name>_mv.[geo]json
LV Files: <feeder_name>_lv.[geo]json
includeLV: false
will only work efficiently if LV files are named in this form, as they will be excluded from the ingest process.
You can filter only at the HV level, and you can specify only one of the include/exclude filters. For HV filters the filter is based on the feeder name, and the feeder name must be at the start of the filename.
An include filter will include only feeders with filenames that start with any of the given strings, and conversely an exclude filter will exclude any feeders with filenames that start with any of the given strings in the same way.
Note the strings are not pattern matches and no regex is supported.
Sample content:
Include 2 HV feeders and their LV but without meter processing or subtrans network.
{
"processMeters": false,
"includeSubtrans": false,
"hv": {
"include": [
"FNA001",
"FNA002"
]
},
"includeLV": true
}
Include 2 HV feeders, plus sub-trans with meter processing, but no LV.
{
"hv": {
"include": [
"FNA001",
"FNA002"
]
},
"includeLV": false
}
Two HV feeders and their LV, plus sub-trans with meter processing.
{
"hv": {
"include": [
"FNA001",
"FNA002"
]
}
}
Default Wire Info
A default WireInfo
may be specified in the processor config as follows:
{
"defaultWireInfo": {
# Set to true to specify an overhead line, and false for an underground cable
"isOverhead": boolean,
# The rated current of the wire.
# Default: null
"ratedCurrent": null | int,
# The material of the wire. This should be one of WireMaterialKind, e.g. "copperCadmium".
# Default: "UNKNOWN"
"material": string
}
}
Sample:
{
"defaultWireInfo": {
"isOverhead": true,
"ratedCurrent": 3,
"material": "copperCadmium"
}
}
Default Per-Length Sequence Impedance
A default PerLengthSequenceImpedance
may be specified in the processor config as follows (see CIM documentation for field definitions):
{
"defaultPerLengthSequenceImpedance": {
"r": null | double,
"x": null | double,
"bch": null | double,
"gch": null | double,
"r0": null | double,
"x0": null | double,
"b0ch": null | double,
"g0ch": null | double
}
}
All fields default to null
. Sample:
{
"defaultPerLengthSequenceImpedance": {
"r": 0.0025,
"x": 0.003,
"r0": 0.0025,
"x0": 0.003
}
}
Default Power Transformer Info
Due to the hierarchical nature of PowerTransformerInfo
, the structure of its configuration will be denoted using multiple production rules.
A full example is given at the end of this section. Refer to the CIM documentation for field definitions.
<PowerTransformerInfo spec> = {
"transformerTankInfos": [<TransformerTankInfo spec>*]
}
<TransformerTankInfo spec> = {
"transformerEndInfos": [<TransformerEndInfo spec>*]
}
<TransformerEndInfo spec> = {
"connectionKind": "<WindingConnection value name>", # defaults to "UNKNOWN_WINDING"
"emergencyS": null | int, # defaults to null
"endNumber": int, # defaults to 0
"insulationU": null | int, # defaults to null
"phaseAngleClock": null | int, # defaults to null
"r": null | double, # defaults to null
"ratedS": null | int, # defaults to null
"ratedU": null | int, # defaults to null
"shortTermS": null | int, # defaults to null
"transformerStarImpedance": null | <TransformerStarImpedanceDto spec>, # defaults to null
"energisedEndNoLoadTests": null | <NoLoadTest spec>, # defaults to null
"energisedEndShortCircuitTests": null | <ShortCircuitTest spec>, # defaults to null
"groundedEndShortCircuitTests": null | <ShortCircuitTest spec>, # defaults to null
"openEndOpenCircuitTests": null | <OpenCircuitTest spec>, # defaults to null
"energisedEndOpenCircuitTests": null | <OpenCircuitTest spec> # defaults to null
}
<TransformerStarImpedance spec> = {
"r": null | double, # defaults to null
"r0": null | double, # defaults to null
"x": null | double, # defaults to null
"x0": null | double # defaults to null
}
<NoLoadTest spec> = {
"energisedEndVoltage": null | int, # defaults to null
"excitingCurrent": null | double, # defaults to null
"excitingCurrentZero": null | double, # defaults to null
"loss": null | int, # defaults to null
"lossZero": null | int # defaults to null
}
<ShortCircuitTest spec> = {
"current": null | double, # defaults to null
"energisedEndStep": null | int, # defaults to null
"groundedEndStep": null | int, # defaults to null
"leakageImpedance": null | double, # defaults to null
"leakageImpedanceZero": null | double, # defaults to null
"loss": null | int, # defaults to null
"lossZero": null | int, # defaults to null
"power": null | int, # defaults to null
"voltage": null | double, # defaults to null
"voltageOhmicPart": null | double # defaults to null
}
<OpenCircuitTest spec> = {
"energisedEndStep": null | int, # defaults to null
"energisedEndVoltage": null | int, # defaults to null
"openEndStep": null | int, # defaults to null
"openEndVoltage": null | int, # defaults to null
"phaseShift": null | double # defaults to null
}
Full example:
{
"defaultPowerTransformerInfo": {
"transformerTankInfos": [
{
"transformerEndInfos": [
{
"connectionKind": "D",
"emergencyS": 1,
"endNumber": 1,
"insulationU": 3,
"r": 4.4,
"ratedS": 5,
"ratedU": 6,
"shortTermS": 7,
"transformerStarImpedance": {
"r": 1.1,
"r0": 2.2,
"x": 3.3,
"x0": 4.4
},
"energisedEndNoLoadTests": {
"energisedEndVoltage": 1,
"excitingCurrent": 2.2,
"excitingCurrentZero": 3.3,
"loss": 4,
"lossZero": 5
},
"energisedEndShortCircuitTests": {
"current": 1.1,
"energisedEndStep": 2,
"groundedEndStep": 3,
"leakageImpedance": 4.4,
"leakageImpedanceZero": 5.5,
"loss": 6,
"lossZero": 7,
"power": 8,
"voltage": 9.9,
"voltageOhmicPart": 10.10
}
},
{
"connectionKind": "Y",
"emergencyS": 1,
"endNumber": 2,
"insulationU": 3,
"phaseAngleClock": 30,
"r": 4.4,
"ratedS": 5,
"ratedU": 6,
"shortTermS": 7
}
]
}
]
}
}
Cable Types File
The GIS Network Extractor derives cable type information from a json file structured as follows:
{
"cableTypes": [{
"id": string,
"resistance": double,
"reactance": double,
"zeroSeqResistance": double,
"zeroSeqReactance": double,
"nominalVoltage": double,
"rating": int,
"maxFaultCurrent": double,
"normalRating": double,
"springDay": double,
"springNight": double,
"summerDay": double,
"summerNight": double,
"autumnDay": double,
"autumnNight": double,
"winterDay": double,
"winterNight": double,
"neutralRating": double
}
]
}
If you have a running copy of the Energy Workbench Server
you can download the existing cable types.
Sample content:
{
"cableTypes": [
{
"id": "Cu_0.1i_SWAS_SD_11_6",
"resistance": 0.342,
"reactance": 0.115,
"zeroSeqResistance": 0.795,
"zeroSeqReactance": 0.066,
"nominalVoltage": 11,
"rating": 215,
"maxFaultCurrent": 10.1,
"normalRating": 1,
"springDay": 1,
"springNight": 1,
"summerDay": 1,
"summerNight": 1,
"autumnDay": 1,
"autumnNight": 1,
"winterDay": 1.14,
"winterNight": 1.14,
"neutralRating": 215
},
{
"id": "Cu_0.25i_pswsb_SD_6.6_3",
"resistance": 0.133,
"reactance": 0.074,
"zeroSeqResistance": 0.95,
"zeroSeqReactance": 0.541,
"nominalVoltage": 6.6,
"rating": 345,
"maxFaultCurrent": 10.1,
"normalRating": 1,
"springDay": 1,
"springNight": 1,
"summerDay": 1,
"summerNight": 1,
"autumnDay": 1,
"autumnNight": 1,
"winterDay": 1.117,
"winterNight": 1.117,
"neutralRating": 345
}
]
}
Network Hierarchy File
The GIS Network Extractor derives network hierarchy information from a json file structured as follows:
{
"businesses": [{
"id": string,
"name": string,
"regions": [{
"id": string,
"name": string,
"zones": [{
"id": string,
"name": string,
"feeders": [{
"id": string,
"name": string
}]
}]
}]
}]
}
If you have a running copy of the Energy Workbench Server
you can download
the existing network hierarchy.
Sample content:
{
"businesses": [
{
"id": "business1",
"name": "BUSINESS1",
"regions": [
{
"id": "region1",
"name": "REGION1",
"zones": [
{
"id": "zone1",
"name": "ZONE1",
"feeders": [
{
"id": "feeder1",
"name": "FEEDER1"
}
]
}
]
}
]
}
]
}
Nodes Config File
The GIS Network Extractor derives node configuration information from a json file structured as follows:
Structure:
{
"mappings": [
{
"gisClass": "<gis-node-class>",
"cimClass": "<conducting-equipment-class>",
"style": "<diagram-object-style>"
}
],
"switchStyles": [
{
"value": "<attribute-value>",
"style": "<diagram-object-style>"
}
],
// optional
"reactorClasses": [
"<gis-node-class>"
],
// optional
"defaultReactorSizeKVAR": 1.0
}
reactorClasses
defaults to ["reactor_site"]
, and defaultReactorSizeKVAR
defaults to 25.0
when they are not provided.
Sample content:
{
"mappings": [
{
"gisClass": "acr",
"cimClass": "Recloser",
"style": "RECLOSER"
},
{
"gisClass": "hvSwitch",
"cimClass": "Disconnector",
"style": "SWITCH"
}
],
"switchStyles": [
{
"value": "bridge",
"style": "BRIDGE"
},
{
"value": "air-break",
"style": "SWITCH"
},
{
"value": "live line clamp",
"style": "LIVE_LINE_CLAMP"
}
],
"reactorClasses": [
"reactor_site"
],
"defaultReactorSizeKVAR": 25.0
}
Substation Transformer Tap Settings file
Structure:
ID,SUBSTATION_NAME,NAME_PLATE_RATING,VECTOR_GROUP
<Transformer mRID>,<Zone Substation mRID>,<TX rating in MVA>,<Vector group of TX>
Tap settings are applied based on the rating or vector group depending on what part of the network the Transformer is located. For Citipower substations the NAME_PLATE_RATING dictates the tap settings that are applied, whereas for Powercor substations the vector group dictates the tap settings.
See for how tap settings get applied based on vector group/rating.
Sample content:
ID,SUBSTATION_NAME,NAME_PLATE_RATING,VECTOR_GROUP
10297282,SK,27,Unknown
10297293,BAN,30,DY1
10297304,BAS,30,DY1
10297310,SK,30,DYN1
Line Regulator Settings file
Structure:
Regulators,CT_Ratio,VT_Ratio,Set_Point,Bandwidth,Resistance,Reactance,CT_Primary,Time_Delay,Reverse_Resistance,Reverse_Reactance
<Transformer Name>,<float>,<float>,<float>,<float>,<float>,<float>,<float>,<float>,<float>,<float>
Matching is performed on transformer name via the Regulators column.
Sample content:
Regulators,CT_Ratio,VT_Ratio,Set_Point,Bandwidth,Resistance,Reactance,CT_Primary,Time_Delay,Reverse_Resistance,Reverse_Reactance
9723 ALLANSFORD REG,80,183.3,117,3,3,4,400,50,-3,-4
9724 ALLANSFORD REG,80,183.3,117,3,3,4,400,50,-3,-4
9725 ALLANSFORD REG,80,183.3,117,3,3,4,400,50,-3,-4
ANNUELLO ROAD P47 REG,40,183.3,118.2,3,5.4,4.3,200,80,0,0
ANNUELLO ROAD P47 REG 1,40,183.3,118.2,3,5.4,4.3,200,80,0,0
Voltage Regulator Settings file
Structure CPPAL
Same structure as above except with 2 less columns and Substation mRIDs rather than transformer names as a key.
Station,CT_Ratio,VT_Ratio,Set_Point,Bandwidth,Resistance,Reactance,CT_Primary,Time_Delay
<SUBSTATION_MRID>,<float>,<float>,<float>,<float>,<float>,<float>,<float>,<float>
Sample content CPPAL:
Station,CT_Ratio,VT_Ratio,Set_Point,Bandwidth,Resistance,Reactance,CT_Primary,Time_Delay
AC,300,100,112.2,10,0,0,1500,100
AL,300,100,111.1,1.2,4.4,4.4,1500,100
AP,300,100,113.9,1.1,,,1500,100
AR,300,100,112.2,1.1,2.2,3.1,1500,100
ART,100,200,110,2,4.8,3.6,600,100
Structure EE
Same structure as above except with 2 less columns and Substation mRIDs rather than transformer names as a key.
PREFIX,VO,LDC_PERCENT,IREF,LDC_BLOCKING,PF
<Substation mRID>,<float>,<float>,<float>,<N | Y>,<float>
Sample content EE:
PREFIX,VO,LDC_PERCENT,IREF,LDC_BLOCKING,PF
ARD,11.5,,,,0.9
ASM,11,0.02,1574,N,0.9
ATA,11.2,,,,0.9
AVE,11.033,0.0604,296,N,0.9
BAL,11.033,0.0423,778,N,0.9
Source Impedance file
Structure CPPAL
Bus,TS,Name,kV,3PH.kA,LG.kA,LL.kA,thev.r1,thev.x1,thev.r0,thev.x0,Limit,Column1
<Bus ID (unused): string>,<Substation names (unused): string>,<Substation mRID (whitespace) Voltage>,<integer>,<float>,<float>,<float>,<float>,<float>,<float>,<float>,<float (unused)>,<string (unused)>
Sample Content CPPAL:
Bus,TS,Name,kV,3PH.kA,LG.kA,LL.kA,thev.r1,thev.x1,thev.r0,thev.x0,Limit,Column1
38015,ATS-BLTS,AC 11,11,19,20.4,16.4,0.02,0.342,0.008,0.269,18.4,VEDC
36015,ATS-BLTS,AC 66,66,15.3,12.5,13.3,0.286,2.641,0.436,4.282,21.9,VEDC
38017,ATS-BLTS,AL 11,11,7.6,9.1,6.5,0.021,0.908,0.012,0.416,18.4,VEDC
36017,ATS-BLTS,AL 66,66,14.8,11.4,12.8,0.322,2.732,0.559,4.95,21.9,VEDC
38221,FBTS,AP 11,11,15.7,15.7,13.6,0.018,0.451,0.014,0.423,18.4,VEDC
36221,FBTS,AP 66,66,15.9,11.2,13.8,0.237,2.539,0.483,5.415,21.9,VEDC
Structure EE
Substation,FaultedBus,Bus,BasekV,SLG_A,TPh_A,Rpos,Xpos,Rneg,Xneg,Rzero,Xzero,RevDate,EditedBy,Comment
<Substation mRID (whitespace) Substation name: string>,<(unused) string>,<(unused) string>,<(unused) string>,<(unused) float>,<(unused) float>,<float>,<float>,<float>,<float>,<float>,<float>,<float>,<(unused) string>,<(unused) string>
Sample Content EE:
Substation,FaultedBus,Bus,BasekV,SLG_A,TPh_A,Rpos,Xpos,Rneg,Xneg,Rzero,Xzero,RevDate,EditedBy,Comment
ADA ADAMINABY,6746,ADA033A,33.0,1040.0,885.7,5.34531,20.8363,5.34467,20.8384,1.10689,12.0062,6/06/2018,,some comment
ADA ADAMINABY,6732,ADA011A,11.0,3524.3,3014.4,0.49526,2.04778,0.49522,2.04786,0.05294,1.20884,4/10/2016,,another comment
ADE ADELONG,11465,ADE011A,11.0,2253.0,2153.6,0.36771,2.92594,0.36761,2.9272,0.15179,2.55666,7/04/2011,,S.East Rpt: TFR 1 upgraded to 3MVA
AFD ASHFORD,11784,AFD022A,22.0,923.7,878.1,1.849585,14.346209,1.858122,14.357422,0.858186,12.295956,21/09/2010,,Original Issue
Tap Settings file
Structure CPPAL
ID,Tx Nominal Tap No.,Tap Position,Highest Tap Step (Max Boost),Lowest Tap Step (Max Buck),Control Enabled,RatioTapChanger.stepVoltageIncrement
<integer>,<integer>,<integer>,<integer>,<integer>,<boolean>,<double -1..1>
Sample Content CPPAL:
ID,Tx Nominal Tap No.,Tap Position,Highest Tap Step (Max Boost),Lowest Tap Step (Max Buck),Control Enabled,RatioTapChanger.stepVoltageIncrement
19981169,4,1,8,1,FALSE,0.025
#N/A,4,2,8,1,FALSE,0.025
20320900,4,2,8,1,FALSE,0.025
94723101,4,3,8,1,FALSE,0.025
Structure EE
tx_asset_id,tx_label,sub_asset_id,sub_label,tx_kva,tx_phases,tx_primary_voltage,sub_tap_voltage,opendss_hv_tap,tx_impedance,tap_increment_0225,tap_setting_delta_from_nominal,tx_max_buck_tap_num,tx_max_boost_tap_num,tx_nominal_tap_num,calculated_tap_position,tap_position,tap_range,highest_tap_step_max_boost,lowest_tap_step_max_buck,control_enabled
<(unused) integer>,<(unused) string>,<(unused) integer>,<string>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<(unused) integer>,<integer>,<(unused) integer>,<integer>,<(unused) integer>,<integer>,<integer>,<boolean>>
Sample Content EE:
tx_asset_id,tx_label,sub_asset_id,sub_label,tx_kva,tx_phases,tx_primary_voltage,sub_tap_voltage,opendss_hv_tap,tx_impedance,tap_increment_0225,tap_setting_delta_from_nominal,tx_max_buck_tap_num,tx_max_boost_tap_num,tx_nominal_tap_num,calculated_tap_position,tap_position,tap_range,highest_tap_step_max_boost,lowest_tap_step_max_buck,control_enabled
4038170,Generic TX 32895,96767,30-61401,,,,,1,,,0,,,,,,0,,,FALSE
107909,63M41022,96768,44956,50,HV3,11kV,11000,1,5.18,,0,1,5,3,3,3,4,5,1,FALSE
107911,T11304,96771,45015,200,HV3,11kV,11000,1,3.5,,0,,,3,3,3,0,,1,FALSE
7177368,TE44841,96772,45046,100,HV3,11kV,1100,1,4,,0,1,7,5,5,5,6,7,1,FALSE