Config File

The configuration allows specifying whether some of the processing steps will be executed, as well as allows filtering of what data will be ingested and saved to the resulting databases. If a config file is not specified or present, defaults will be as per below.

Attribute	Description	Data type
company	Specifies company specific options.	Company Specific Config
common	Specifies common options.	Common Config

Sample Config File

You can download the sample default configuration file and modify it to suit your installation. There is also a sample non-default configuration file which may provide additional examples if you require them (e.g. PoF TCS, metrics etc.).

tip

The default sample has the options defined with default values, so simply remove the ones you don't want to change to reduce the size of your config.

You will need to remove the unwanted company sections to produce a valid JSON file.

Company Specific Config

This section doubles as the definition of the company specific processing to run, and provides customisations options for these processors.

Attribute	Description	Data type	Default
id	The ID of the company configuration to use.	String - CPPAL, EE or UE	CPPAL
conductorTypes	The company specific conductor type configuration.	CPPAL Conductor Types EE Conductor Types UE Conductor Types

CPPAL Conductor Types

Specifies CPPAL (Citipower/Powercor) specific conductor type paths.

Attribute	Description	Data type	Default
path	A file containing the cable types JSON or a JDBC connection string to the 'SINCAL standard types' database.	String	./cable-types.json
hvDefaultConductorTypes	Path to a CSV containing additional HV conductor types.	String	None.

EE Conductor Types

Specifies EE (Essential Energy) specific conductor type paths.

Attribute	Description	Data type	Default
path	Path to a 'SINCAL standard types' database.	String	./cable-types.db

UE Conductor Types

Specifies UE (United Energy) specific conductor type paths.

Attribute	Description	Data type	Default
hvOverheadPath	Path to a CSV containing HV conductor types.	String	./hv-overhead-types.csv
hvUndergroundPath	Path to a CSV containing HV cable types.	String	./hv-underground-types.csv
lvPath	Path to a CSV containing additional LV conductor and cable types.	String	None.

Common Config

The configuration that is common between all companies.

Attribute	Description	Data type
input	Customisations to the inputs of the ingestor.	Input Config
processing	Configuration which controls how the ingestor processes the input data.	Processing Config
output	Customisations to the output from the ingestor.	Output Config

Input Config

Customisations to the inputs of the ingestor.

Attribute	Description	Data type	Default
gisData	Defines what GIS data should be ingested.	GIS Data Config
nodesConfigPath	The file containing the nodes JSON config.	String	nodes-config.json
conductorExclusionConfigPath	Optional path to a file containing the conductor exclusion JSON config.	String	None.
networkHierarchyPath	The file containing the network hierarchy JSON.	String	network-hierarchy.json
derRegisterPath	Optional path to a file containing the DER register.	String	None.
substationTransformerTapSettingsPath	Optional path to a file containing zone substation transformer tap changer settings.	String	None.
regulatorSettingsPath	Optional path to a CSV file containing the zone substation regulator settings.	String	None.
lineRegulatorSettingsPath	Optional path to a CSV file containing the line regulator settings.	String	None.
sourceImpedancesPath	Optional path to a CSV file containing the source impedances.	String	None.
spanRatingsPath	Optional path to a CSV file containing the span level ratings for AC line segments.	String	None.
txTapsPath	Optional path to a CSV file containing the transformer tap settings.	String	None.
poleDataLocation	Optional path to a file or database containing pole data, or a full JDBC connection string.	String	None.
meters	Defines what meters should be ingested - MAPPING_FILE or TCS.	Meter File Config TCS Meter Config	MAPPING_FILE
powerOnFusion	Optional configuration for pulling data from PowerOn Fusion/Advantage.	PowerOn Fusion Config	None.
onMissingNodeMappingHandler	Optional handler for missing node configuration mappings.	String - REPLACE_WITH_JUNCTION or EXCEPTION	EXCEPTION

GIS Data Config

Defines the location and inclusion rules for the GIS data to process.

Attribute	Description	Data type	Default
path	The path where the input GIS JSON files are located.	String	gis
warnOnMissingFeeder	Whether to log a message if the head of a feeder can not be found. When the metadata is missing from geojson most (or all) of these messages are false positives when LV is processed, so suppressing them is preferable.	Boolean	true
filters	Customisations to the output from the ingestor.	Filter Config	None.

Filter Config

Filters allow specifying which feeders get processed and stored in the result, however there are file name requirements that must be met for filters to work as intended. For the most efficient handling of data, separate files should exist for HV and LV network, and they must be named accordingly in the form below:

HV Files: <feeder_name>_mv.[geo]json or <feeder_name>.[geo]json
LV Files: <feeder_name>_lv.[geo]json or <feeder_name>_<dist_sub_name>.[geo]json

For HV filters, the filter is based on the feeder name, and the feeder name must be at the start of the filename.

An include filter will include only feeders with filenames that start with any of the given strings, and conversely an exclude filter will exclude any feeders with filenames that start with any of the given strings in the same way.

Attribute	Description	Data type	Default
hv	The filters to apply to the HV files.	Filter Definition	None.
lv	The filters to apply to the LV files.	Filter Definition	None.

Filter Definition

Attribute	Description	Data type	Default
type	The type of filter.	String - include or exclude	None.
ids	The ids to filter based in the filter type.	List[String]	None.

note

The ids strings are not pattern matches, and no regex is supported.

Meter File Config

A meter input type of MAPPING_FILE indicates the meters will be read from files in a specified directory.

Attribute	Description	Data type	Default
type	Indicates the meters come from a mapping file.	String	MAPPING_FILE
path	The path where the input meter mappings csv files are located.	String	meter_mappings
filePattern	Specifies the file name pattern for meter mapping files .	String	NMI_TX_MAPPING_(\d{8}).*[.]csv
datePattern	Specifies the pattern used to represent dates for the records in the meter mappings csv files.	String	yyyyMMdd

TCS Meter Config

A meter input type of TCS indicates the meters will be read from the PowerOn Fusion TCS schema.

info

You must specify the PowerOn Fusion Config in order to use TCS mappings.

Attribute	Description	Data type	Default
type	Indicates the meters come the PowerOn Fusion TCS schema.	String	TCS
customerTypeMappings	A mapping of TCS customer type codes (keys) to CIM CustomerKind (values).	JSON Object	None.

PowerOn Fusion Config

The details required to connect to a PowerOn Fusion database. When provided, additional information can be extracted including aliases, operational restrictions and TCS meter mappings etc.

Attribute	Description	Data type	Default
host	The hostname of the PoF database server.	String
port	The port number of the PoF database server.	Integer - 1..65535	1521
instance	The database instance to use on the server.	String	nms
databaseType	The type of database being used.	String - ORACLE or SQL_SERVER	ORACLE
sqlServerIntegratedSecurity	Indicates if integrated security should be used when connecting to an SQL Server instance.	Boolean	false
enmacSchema	An override for the 'enmac' schema.	String	None.
tcsSchema	An override for the 'tcs' schema.	String	None.
processAliases	Indicates if PoF aliases should be read for each GIS ID.	Boolean	true

Processing Config

Configuration which controls how the ingestor processes the input data.

Attribute	Description	Data type	Default
lvThreshold	Defines the LV voltage upper limit in volts (inclusive).	Integer	1000
includeSubtrans	Whether to include the Sub-transmission network in the resulting database.	Boolean	true
includeLV	Whether to include LV in the processed result. Currently only works for processing when HV and LV are in separate files.	Boolean	true
validation	Whether to run and log use case validation.	Boolean	true
stitchConductors	Controls if the conductor stitching processor runs.	Boolean	true
createEnergyConsumerLocations	Controls if locations are created for energy consumers.	Boolean	false
createEquivalentBranchLocations	Controls if locations are created for equivalent branches.	Boolean	false
meters	Defines meter processing options.	Meter Config
defaults	Defines default value creation options.	Default Config
phases	Defines phase processing options.	Phase Config

Meter Config

Configuration which controls how the ingestor processes the meter input data.

Attribute	Description	Data type	Default
enabled	Whether to process meters or not.	Boolean	true
includeLonelyMeters	Whether to create meters and consumers from the meter mappings for premises that don't have a corresponding transformer.	Boolean	true
connectionPointIdentifier	Allows renaming the connection point identifier. e.g. use ICP instead of NMI.	String	NMI

Default Config

Configuration which controls how the ingestor processes assigns defaults to missing input data.

Attribute	Description	Data type	Default
assignDefaults	Whether to assign default values for missing transformer and line data if missing.	Boolean	true
defaultWireInfo	Optional WireInfo to assign to lines missing one. Ignored if assignDefaults is false.	WireInfo	None.
defaultPerLengthSequenceImpedance	Optional PerLengthSequenceImpedance to assign to lines missing one. Ignored if assignDefaults is false.	PerLengthSequenceImpedance	None.
defaultPowerTransformerInfo	Optional PowerTransformerInfo to assign to transformers missing one. Ignored if assignDefaults is false.	PowerTransformerInfo	None.

Phase Config

Configuration which controls how the ingestor processes phases in the input data.

Attribute	Description	Data type	Default
extractPhases	Controls if the phase information will be extracted from the source data. If disabled everything will be extracted as ABC.	Boolean	true
trustLv1Phases	Indicates that the LV1 phases (AN, BN and CN) can be trusted. If not trusted, they will be replaced with XN.	Boolean	false
forceThreePhaseLv	Force the LV network below a 3-phase transformer to ignore the phasing of the backbone and replace it with 3-phase.	Boolean	false
applyPhaseConsistency	Indicates if phase consistency processing should be run.	Boolean	false
terminalStationPattern	The regex pattern used to find terminal stations .	String	.*TS
phaseDebugNames	Indicates if they phase information should be added to the conductor names before and after the phase consistency processing is run. This information can be very useful for debugging phase issues.	Boolean	false
phaseConsistencyDebugIds	List of ID's to log the phases of during phase consistency processing for debug purposes.	List[String]	[]
defaultPhaseOption	Controls how phases are added to equipment with unknown phases.	String - THREE_PHASE or SPREAD	THREE_PHASE

Output Config

Customisations to the output from the ingestor.

Attribute	Description	Data type	Default
ewbDataPath	The root path for the EWB data.	String	ewb
detailedLog	Controls if a detailed set of errors are logged, or just the summary.	Boolean	false
saveOnException	Indicates that unexpected exceptions should be trapped and the database should still be saved, even though it may be in a broken state. If this is enabled, the migrator will exit with a FAILED_TO_PROCESS code on exceptions.	Boolean	false
metrics	Optional configuration to generate metrics about ingestion runs.	Metrics Config	None.
completionCallback	Optional completion callback configuration.	Completion Callback Config	None.
cloudUploader	Optional cloud upload configuration. If provided, the ingestor will upload the generated EWB database to the cloud location.	AWS S3 Cloud Storage Config	None.
quality	Configuration to control what, if any, quality checks are run.	Quality Config

Metrics Config

Optional configuration for writing metrics to a PostgresSQL database.

Attribute	Description	Data type	Default
host	The hostname of the metrics database.	String
port	The port number of the metrics database.	Integer - 1..65535	5432
database	The name of the database on the Postgres host containing the metrics schema.	String	metrics
urlOptions	Additional options to attach to the Postgres JDBC URL, including the ? separator. This may include things such as the user	String	None.

Completion Callback Config

Configuration for an optional callback to make with the execution status when the ingestion is complete.

Attribute	Description	Data type	Default
url	The full URL to send the callback response to, including any parameters e.g. callback tokens etc.	String
verifyCertificates	Indicates if the certificate of the callback URL should be validated.	String	true

AWS S3 Cloud Storage Config

To configure the ingestor to upload the generated EWB database to AWS S3 storage, set the cloudUploader provider to S3

Attribute	Description	Data type	Default
provider	The provider of the cloud storage to upload into.	String	S3
region	Optional override of the default region. If not provided, the region configured by the AWS SDK will be used.	String	None.
bucketName	The name of the S3 bucket.	String

Quality Config

Configuration for controlling the quality checks and outputs of the ingestor.

Attribute	Description	Data type	Default
enabled	Whether to process quality or not.	Boolean	true
path	Controls the output directory of the quality checks.	String	quality
checks	Controls which quality checks are run. Comma separated list of valid quality checks.	List[Available Quality Checks]	["ALL"]

Available Quality Checks

The different checks that are available. If you include ALL, everything else will be ignored and all checks will be run. Similarly, if you include NONE with any other check it will be ignored.

Attribute	Description
NONE	Don't run any checks.
ALL	Run all checks.
PHASES	Run phase checks.
CUSTOMERS	Run customer validation checks.
LOOPS	Run loop detection checks.

Default Objects

The values for the default instances are object representations of the objects defined in the Zepben CIM profile (see CIM documentation for field definitions).

Default Wire Info

An instance of a WireInfo. The value of isOverhead will be used to create the appropriate subtype (OverheadWireInfo or CableInfo). All missing fields default to null.

Sample:

{
  "defaultWireInfo": {
    "isOverhead": true,
    "ratedCurrent": 3,
    "material": "copperCadmium"
  }
}

Default Per-Length Sequence Impedance

An instance of a PerLengthSequenceImpedance. All missing fields default to null.

Sample:

{
  "defaultPerLengthSequenceImpedance": {
    "r": 0.0025,
    "x": 0.003,
    "r0": 0.0025,
    "x0": 0.003
  }
}

Default Power Transformer Info

An instance of a PowerTransformerInfo. This includes nested copies of any associations. All missing fields default to null.

Sample:

{
  "defaultPowerTransformerInfo": {
    "transformerTankInfos": [
      {
        "transformerEndInfos": [
          {
            "connectionKind": "D",
            "emergencyS": 1,
            "endNumber": 1,
            "insulationU": 3,
            "r": 4.4,
            "ratedS": 5,
            "ratedU": 6,
            "shortTermS": 7,
            "transformerStarImpedance": {
              "r": 1.1,
              "r0": 2.2,
              "x": 3.3,
              "x0": 4.4
            },
            "energisedEndNoLoadTests": {
              "energisedEndVoltage": 1,
              "excitingCurrent": 2.2,
              "excitingCurrentZero": 3.3,
              "loss": 4,
              "lossZero": 5
            },
            "energisedEndShortCircuitTests": {
              "current": 1.1,
              "energisedEndStep": 2,
              "groundedEndStep": 3,
              "leakageImpedance": 4.4,
              "leakageImpedanceZero": 5.5,
              "loss": 6,
              "lossZero": 7,
              "power": 8,
              "voltage": 9.9,
              "voltageOhmicPart": 10.10
            }
          },
          {
            "connectionKind": "Y",
            "emergencyS": 1,
            "endNumber": 2,
            "insulationU": 3,
            "phaseAngleClock": 30,
            "r": 4.4,
            "ratedS": 5,
            "ratedU": 6,
            "shortTermS": 7
          }
        ]
      }
    ]
  }
}