Running the Application

Command Line Arguments

Flag	Description	Required	Default
--backfill-aggregation-index <MAPPINGS_DIR>	Backfill the aggregation index from legacy NMI -> TX mapping files instead of processing the consumption data. See Backfilling the Aggregation Index.	No	Disabled
--backfill-end-date <DATE>	Close all backfilled mappings at the end of the specified ISO date.	No	None
--backfill-start-date <DATE>	Treat the first backfilled snapshot as valid from the specified ISO date.	No	First snapshot date
-c,--conf <CONFIGURATION>	Specifies the path to the config file.	Yes	load-processor.conf
--detailed-stats-log	Show detailed messages for each warning collected in the stats output, instead of just the summary.	No	Disabled
-h,--help	Shows this help message.	No	No
--ignore-after <DATE>	Ignore all meter reading after the specified ISO date.	No	None
--ignore-before <DATE>	Ignore all meter reading before the specified ISO date.	No	None
--parse-format <CPPAL>	Specifies the format file to parse and ingest.	No	CPPAL
--rebuild-index	Rebuild the load readings index instead of processing the consumption data. Use if your index is gets deleted or corrupted.	No	Disabled

To run the tool, execute the following command at a terminal:

java -Xmx<heap size> -Xms<heap size> -jar <path to load-processor-jar> [--backfill-aggregation-index <MAPPINGS_DIR>] [--backfill-end-date <DATE>] [--backfill-start-date <DATE>] [-c <CONFIGURATION>] [--detailed-stats-log] [-h] [--ignore-after <DATE>] [--ignore-before <DATE>] [--parse-format <CPPAL>] [--rebuild-index]

where:

• heap size is the desired size of the heap. E.g. 12g

e.g:

java -Xmx12g -Xms12g -jar load-processor.jar -c config.json --parse-format CPPAL --ignore-before 2020-01-01

Backfilling the Aggregation Index

The load processor and load aggregator can only process readings on or after the earliest date in the aggregation index, which is written by the GIS network extractor each time it runs. To process older readings, the history of the index can be built from the legacy NMI -> TX mapping files (NMI_TX_MAPPING_yyyyMMdd.csv or NMI_TX_MAPPING_yyyyMMdd.csv.gz) with:

java -Xmx16g -jar load-processor.jar -c config.json --backfill-aggregation-index <MAPPINGS_DIR> [--backfill-start-date <DATE>] [--backfill-end-date <DATE>] [--ignore-before <DATE>] [--ignore-after <DATE>]

• The index is written to the ewbDataPath from the configuration. It is created if it doesn't exist.
• Each mapping file is treated as a snapshot of the mappings from its date, until the next snapshot. Mappings are converted the same way the GIS network extractor would: meters are mapped to NMIs, NMIs to <transformer_id>-agg (or <feeder>-agg for HV customers without a transformer), and transformers to <feeder>-agg.
• --ignore-before and --ignore-after limit which snapshots are used.
• Company names in the mapping files are converted to the network company names using the companyMapping from the configuration (e.g. PCOR to Powercor). Names that are already network names are kept as is.
• Snapshots that can't be read (e.g. corrupt or truncated gzip files) are skipped with a warning, and the previous snapshot continues to be used.
• If the index is new, the mappings from the last snapshot are left open so the next GIS network extractor run updates them as normal. If the index already has mappings, the backfill must end before them, and any backfilled mappings still valid at the end are closed the day before the earliest existing mapping.

The full CPPAL network requires around 16GB, and each snapshot takes around 10-15 seconds to process.

Heap Size Estimation

The LV Mapping data requires around 1.5GB/day

The meter data requires around 1GB/day

A 3 day interval input data would require around 7.5GB

Together with the process overhead, the recommended heap size is 12GB