Timeseries Data Ingestion

This section describes the file formats and requirements for ingesting customer meter reading data (AMI/NMI data) into the Zepben timeseries database for use in Hosting Capacity Module work packages.

Sign Convention Reference

For information about how these channel types (B, E, K, Q) relate to power sign conventions in HCM, see Power Sign Conventions.

---

Reading Types and Channel Codes

Zepben supports ingestion of meter reading data from multiple Australian Distribution Network Service Providers (DNSPs), each with their own file format specifications. All formats represent interval meter data (typically 30-minute intervals) for active and reactive power consumption and generation at customer connection points (This will be a NMI in Australia and ICP in New Zealand).

All formats use variations of the AEMO standard channel codes to indicate the direction and type of power flow:

Channel Code	Meaning	Direction
E	Active energy import	Customer consumes power (grid -> customer)
B	Active energy export	Customer generates power (grid <- customer)
Q	Reactive energy import	Customer consumes reactive power (grid -> customer)
K	Reactive energy export	Customer generates reactive power (grid <- customer)

Customer Perspective

The Import/Export terminology used in general HCM documentation is from the customer's perspective. Import means the customer draws power from the grid; Export means the customer delivers power to the grid.

Note that AEMO's NMI Procedure uses market-perspective terminology where the meaning of "import" and "export" is inverted (e.g. the E suffix originates from AEMO's "Export from the market to the customer"). See Power Sign Conventions -- AEMO Terminology for a full comparison table.

Ask Zepben if you need assistance.

---

File Format Specifications

Essential Energy (EE)

Essential Energy File Format Details ▼

---

Endeavour Energy

Endeavour Energy File Format Details ▼

---

Citipower/Powercor

Citipower/Powercor File Format Details ▼

---

United Energy (UE)

United Energy File Format Details ▼

---

General Requirements and Rules

CSV Format Types

Narrow form (Essential Energy, Endeavour): One row = one reading

Wide form (Citipower/Powercor, United Energy): One row = 24-hour period (48 intervals)

CSV Formatting Rules

• Unknown columns are ignored by the ingestor
• No spaces around commas: use a,b,c not a, b, c
• Quote handling: Varies by DNSP (Citipower/Powercor and UE treat quotes as literal; Essential Energy and Endeavour parse quotes). Best practice is to avoid commas and quotes in string values.

Identifier Uniqueness

• Meter identifiers: Must be unique within a company
• Channel identifiers: Must be unique only within the same meter

Duplicate Reading Handling

Duplicate Readings Must Be Avoided

Two meter readings are considered duplicates if they share the same:

• Meter ID
• Channel ID
• Reading type (e.g., reactive generation)
• Timestamp

When duplicate readings are provided, the system does not guarantee which reading will be retained (it may be the earliest in the file, the latest, or determined by alphabetical file order).

Action required: Ensure duplicate meter readings are removed before ingestion.

---

Missing Intervals and Gap Filling

When the HCM builds a customer's annual load profile, any interval that has no reading in the timeseries database is gap-filled using a simple clock based approach so that every customer has a complete, fixed-length year. The fill is based on time of day: a missing interval takes the most recent reading seen for that same clock time.

This means:

• A single missing half-hour (say, 3pm the 5th of May) is filled with data from that time of day from the previous day (3pm, 4th of May).
• A whole missing day reproduces the previous day's shape.
• A longer outage repeats the last real day until readings resume.

If a customer's data only starts partway through the load window, the period before their first reading is treated as zero load and generation.

The four reading directions (real import, real export, reactive import, reactive export) are filled independently - each direction is only ever updated by readings of that direction, meaning partial data is handled per-channel.

note

Gap filling is deliberately simple and not intended to be fully realistic. It guarantees a consistent profile size and approximates the daily loadshape, but it will not perfectly match reality: carried-forward values are not adjusted for weekday vs weekend, temperature, or seasonal effects, and the results do not distinguish gap-filled values from measured ones.

Short gaps should have only a minor impact. However, long or seasonal outages, or meters installed partway through the window, can shift the annual load significantly.

Supplying as complete and continuous a dataset as possible is the user's responsibility. This may include externally generated synthetic data from a more sophisticated algorithm (e.g. customer groupings, controlling for additional variables). Talk to Zepben if you have any further questions.

Data Quality Recommendations

Timestamp Consistency

• Use consistent timezone representation (UTC recommended)
• Ensure timestamps align to expected interval boundaries:
  • 30-minute intervals: :00:00 or :30:00
  • 15-minute intervals: :00:00, :15:00, :30:00, or :45:00

Data Validation Checklist

Before providing data for ingestion:

1. Check for and remove duplicate readings
2. Validate timestamp formats match the specification
3. Ensure required columns are present and non-blank where specified
4. Verify unit codes and channel types match the allowed values
5. Check that reading values are appropriate for the unit (e.g., kWh vs Wh)