Data Ingestion
The Energy Workbench deals with a lot of time series data. Each day, the previous days data becomes available for consumption and the EWB ingests this to have the latest data ready to be accessed from the EWB APIs.
The daily data that gets crunched includes the following:
- A network model from PowerOn Fusion.
- Energy consumption data from smart meters.
- Mappings of current meter, supply point, NMI, transformers and weather stations.
- Weather data such as temperature, rainfall, wind speeds etc.
This data is processed by various tools released as part of the Energy Workbench:
| Tool | Description |
|---|---|
PoF Network Extractor | Extracts a network model from PowerOn Fusion and processes LV meter mappings. |
Load Processor | Processes meter energy usage data CSV files and stores it in the EWB time series database. |
Load Aggregator | Aggregates meter energy usage data to their corresponding distribution transformer. |
Weather Processor | Processes weather data CSV files and stores it in the database. |
For more information on each on these tools, see their corresponding documentation.
ewb_daily.sh
A script has been provided to invoke all the above tools in the appropriate order to perform the daily data crunching.
It can be found at EWB_RELEASE_DIR/bin/ewb_daily.sh.
The script relies on the ewb.conf script found in EWB_RELEASE_DIR/conf/ewb.conf. If you have moved this conf file
the conf include path in the script needs to updated to its location. See the deployment doco for more information
about ewb.conf.
It is recommended to set this script to run via a cron job after all appropriate data files have been copied to the input directories defined in each tools conf file. See the corresponding tool documentation for information on their config and the types of files they ingest.
The script performs the following tasks:
- Stops the EWB server.
- Run the PoF Network Extractor.
- Run the Load Processor.
- Run the Load Aggregator.
- Run the Weather Processor.
- Run the Weather Station Mapper.
- Start the EWB Server.
!!! note The PoF Network Extractor will use the ENMAC_USER and ENMAC_PASSWORD environment variables. Ensure the daily script is run in a way that these environment variables are available.
If any of the data processing tools fail, the script will still restart the EWB server and then exit with a non 0 code.
Any errors that occur during the processing will need to be fixed and then the failed processes will need to be manually
invoked to process the data.
Check the syslog if it has been enabled via ewb.conf for errors and also in the logs for each tool in the logs/
directory under the EWB release directory (or at the otherwise configured location).