Timelines and delta migration

Sist oppdatert 2018-06-29 10:47

Migration to DAM happens in several stages and with full migrations and delta migrations a few weeks apart. This page describes how DAM creates timelines for each entity and how new data transferred on delta migrations replace or cancel out previously migrated data.

Keys used in establishing timelines

For each of the migration file types, different keys will be used to identify a unique entity timeline. This will be a combination of the GLN in the file name and one or two fields in the actual files.

Dimensions of time and overwriting data in migration

The final timeline migrated into Elhub is established by working with two dimensions of time. DAM will will let new data overwrite old data using predefined rules.

Validity period

As established by ValidFrom and ValidTo in the migration files.

It is important to emphasize that each change of one of the represented values in the files should result in one new record. For example if the meter constant on the metering point has been changed it should result in two separate rows. One row representing the old state, with an end date, and another row with an empty end date representing the current new state.

Migration sequence

Where in the sequence of migration is the data transmitted. Based on when the file was transmitted and where in the file the data is located.

  • Oldest: First row in the first transmitted file
  • Newest: Last row in the last transmitted file

Overwriting data in delta migration

For each timeline, new data will overwrite old data (migration sequence) for the validity period of the new data. This is the key feature of delta migration. This can be either updates, corrections, cancellations or any other changes to the data.

For metering points, the new data will only overwrite for the transmitted validity period, allowing partly overwritten data to still be seen as OK for validity periods not covered by the new data. Gaps or partly overwritten data will be reported as errors to grid owners, but not to balance suppliers.

For contracts, the new data will overwrite all older migrated data that is partly overlapping the validity period of the new data row.

Resubmitting whole metering points on delta migration

The overwriting rules defined above gives possibilities, but also limitations, in sending in full metering point history on delta migration for single metering points. This means that metering points can be flagged for delta in the source system and the whole metering point, with contracts and customers, are transmitted to DAM on delta migration. But there is also a need to consider the need to explicit cancellation of data. 

Timeline requirements

FileMarket partyGaps allowedPartly overlapping data allowedMust have current data - i.e. row with empty ValidTo
MP - Metering Point – Målepunkt Grid Owner

No

The exception to the rule is where a historic customer address has been classified as "secret". This will however result in gap errors reported since these metering points should never be flagged as special in any way.

NoYes. Elhub must allways know the current state of the metering point. Else it will not be transmitted into Elhub.
MP - Metering Point – Målepunkt Balance SupplierYesYesNo
CUS - Customer– Kunde Grid OwnerNot relevant - Only the newest data is usedNot relevant - Only the newest data is usedShould. Only the newest data about a customer is used by DAM. If only older validity period is transmitted, then this data will be used.
CUS - Customer– Kunde Balance Supplier
CO - Contract - Kontrakt Grid OwnerYesYes, DAM will cancel the partly overlapping row.No
CO - Contract - KontraktBalance Supplier

Cancellation of migrated data

As of  delta migration has been removed from the GoLive plan for Elhub. This functionality will remain in DAM, but should not be used.

Cancellation of migrated data is used when previously migrated data has to be removed from the timeline of a metering point or a contract. Cancellation of customer data is not relevant because only the newest data is used and only active customers for non-cancelled contracts on valid metering points will be transferred into Elhub.

Cancellation of a full timeline

Relevant for metering points and contracts:

  • cancelling the metering point and contract history completely ( it should never have been migrated if the first place )
  • the amount or nature of changes done on the metering point or contract timeline requires a "reset" of the timeline before transmitting new data

To cancel a full timeline, send a row with empty ValidFrom and ValidTo in the future. All other data in the row should be correct to eliminate the possibility that any format validation errors are triggered.

If this solution is used to "reset" timeline before transmitting new data, the order of which the data is transmitted is very important. The cancellation row must be earlier in the file than the new data.

Cancellation of contracts

If a migrated contract is cancelled between full and delta migration, it needs to be explicitly cancelled in the migration files. This is done by using special values in CO.ContractCode (GC, BC, LC)

The cancellation contract will cancel all previously migrated data that is partly (or fully) overlapping the validity period of the cancellation contract, according to the general rules for overwriting stated above.

Examples

This PDF contains drawn examples of how the timeline is checked and established in DAM (Norwegian only).

2018-06-08 - Eksempler tidslinje.pdf