02 - Initial Migration and Structure Data Establishment

Sist oppdatert 2019-02-25 10:50

Edielportalen:

  • Go Live 02 <Rolle> Initiell migrering og etablering av strukturdata

Execution period

Elhub Go Live: The activities in this section are executed in the period 7/1 to 25/1.

Summary of activities

The activities described in this chapter includes receiving structure data and migration data from the market, to process this data, and to eventually load it into Elhub. This chapter also includes checklists for migrated data to be performed by market parties.

Content overview

Initial migrations contains the following main parts:

  • 2.1 Receive structure data
  • 2.2 Receive migration files from market parties
  • 2.3 Stage structure and master data
  • 2.4 Market party feedback and quality evaluation
  • 2.5 Load structure data
  • 2.6 Structure data validation
  • 2.7 Loading and validating Master Data
  • 2.8 Put Elhub in business as usual mode
  • 2.9 Market parties QA in Elhub

Commissioning steps

Task ID Commissioning step Start time Deadline Execution Organization Step description Done
2.0.1 Sjekkpunkt

Produksjonsmiljøet for Elhub er klart

8/1

9:00

N/A Elhub Kapittel 1 i CRB er gjennomført. Ingen blokkerende feil i Elhub-systemet
  •  
2.1 Receive structure data

Most of the steps in this section can be done in parallell. Loading and verification needs to be done at the end.
2.1.1 Reminder to market parties about migration 7/1 9/1 Statnett

Send reminder on email to market parties about upcoming delta migration.

Send notification to market parties that does migration

  •  
2.1.2 Receive structure Data from Ediel 7/1 9/1 Statnett

Receive Market Parties from Ediel Portal into DAM landing, including message exchange authorizations.

Retrieve files, then commit to stash-repo for tracking.

Get structure data from external parties


  •  
2.1.3 Create old market parties 7/1 9/1 Statnett

In the files from EDIEL, only the market parties after Go Live will be included. Since we migrate the history of the master data, it is possible we need to migrate contracts with market parties that don't exist anymore. Those market parties need to be created as well.

Create old marked parties structure data

  •  
2.1.4 Receive grid structure data from NOA and create supporting files. 7/1 9/1 Statnett

Receive MGAs, MBAs and their relations as files from NOA. Then create additionalWebportalMGA data to augment the production data (e.g. loss parameters and BSL).

Retrieve files, then commit to stash-repo for tracking.

Get and create structure data for grid

  •  
2.1.5 Receive structure data from IFS 7/1 9/1 Statnett

Retrieve account number per organization.

Retrieve files, then commit to stash-repo for tracking.

Get structure data from external parties

  •  
2.1.6 Receive structure data from BRREG 7/1 9/1 Statnett

Retrieve organization name per organization number

Retrieve files, then commit to stash-repo for tracking.

Get structure data from external parties

  •  
2.1.7 Update postal codes data if needed 7/1 9/1 Statnett

If changes has been made by Posten, then update the POSTAL_CODES table (refdata).

Update Postal Codes in DAM

  •  
2.1.8 Create structure data for foreign MGAs 7/1 9/1 Statnett

Foreign MGAs (FMGA) must be created so they can be received and loaded in DAM via ODI. The dataisallreadycreated,butneeds to be verified.

Create structure data files for foreign grid

  •  
2.1.1 Get and create structure data for SGAs 7/1 9/1 Statnett

The data is gathered from LIO andsub gridowners and transformed to be on the correct format for EDM.

Get and create structure data for SGAs

  •  
2.1.9 Receive Balance Responsibility contracts per MGA and production units from NBS 7/1 9/1 Statnett

Receive BRP for BS pr MGA from NBS into DAM landing. Upload into DAM loading must be done after cleanup and upgrade of DAM.

Get structure data from NBS

  •  
2.1.10 Receive file with certificate entitled production points from NECS 7/1 9/1 Statnett 

Wewillrecievealist of component codes from NECS. These will have to be mapped to metering point IDsbymigrationteam.

Get structure data from NECS

  •  
2.1.11 Upload structure data to DAM 7/1 9/1 Statnett 

Once we have structure data from all external parties, it can be uploaded using ODI.

How to load structure data in EDM

  •  
2.1.12 Ensure internal market parties are added in EDM 7/1 9/1 Statnett

Internal market parties are used in EDM to upload connection points.

Ensure internal market parties in EDM

  •  
2.1.13 Verify structure data in DAM landing 7/1 9/1 Statnett 

Once all structure data has been loaded into DAM landing, verify that all is present.

If there are errors, delete data in tables, fix and upload again.

Validating landing structure data

  •  
2.2 Receive migration files from market parties
2.2.1 VerifyfunctionalconfigrationinEDM 9/1 13/1 Statnett

Before we start uploading files, we need to ensure that all functional parameters are set correctly.

Verify functional configuration in EDM

  •  
2.2.2 Ensure indexes for format and integrity check 9/1 13/1 Statnett

Ensure indexes for phase UPLOAD

Create and drop indexes on EDM tables

  •  
2.2.3 Verify database health for EDM database 9/1 13/1 Statnett

We need to ensure that we have no known issues in the database.

EDM database health check

  •  
2.2.4 Open for upload of files to EDM 9/1 13/1 Statnett

Configurationparametersneedstobe set

  •  
2.2.5 Upload sanity files 9/1 13/1 Statnett

Sanity files for DAM to ensure that data reads into DAM ok.

  •  
2.2.6 Notify market that EDM is ready for upload 9/1 13/1 Statnett

Final notification

Send notification to market parties that does migration

  •  
2.2.7 Laste opp migreringsfiler til DAM

9/1

00:00

9/1

22:00

Nettselskaper og kraftleverandører

Data om målepunkt, kunder og kontrakter, samt 1 uke måleverdier for timesavregnede (24.12. – 30.12.). I tillegg minimum siste periodevolum med tilhørende start- og sluttstand, på relevante (profilavregnede) målepunkter

Gjennomføring av dette steget skal rapporteres i Edielportalen.

NA
2.2.8 Receive data through migration files

9/1

00:00

9/1

22:00

Statnett 

Includes format check and integrity check.

Both master data and metering values

Format and integrity check

  •  
2.2.9

Verify all grid owners uploaded data

9/1 13/1 Statnett

We need data from all grid owners.

See Format and integrity check postconditions

If grid owner has not uploaded, call them.

  •  
2.2.11 Ensure indexes for timeline check 9/1 13/1 Statnett

Ensure indexes for phase TIMELINE

Create and drop indexes on EDM tables

  •  
2.2.9.1 Create and upload connection points file 9/1 13/1 Statnett

Connection points arenotrecievedfrommarketparties,andneeds to be uploaded by DAM Operators. It is also used as a sanity test of file upload.

Create and upload connection points

  •  
2.2.10 Block upload of files to EDM 9/1 13/1 Statnett

Configurationparametersneedstobe set

  •  
2.2.12

Gather locked stats on entity tables after loading

9/1 13/1 Statnett

Make sure stats are gathered

Gather stats on EDM tables

  •  
2.2.13 Run timeline check 9/1 13/1 Statnett

Timeline check replaces old data and updates data to set them as current (eligible for staging and consistency)

How to run timeline check

  •  
2.2.14 Ensure indexes for phase CONSISTENCY 9/1 13/1 Statnett

Ensure indexes for phase CONSISTENCY

Create and drop indexes on EDM tables

  •  
2.2.15 Run consistency check 9/1 13/1 Statnett

Run consistency on the data, including meta list and core process.

This step can be skipped and re-run in a later stage in case of time constraint.No consiequence for data loaded into Elhub, but is required for feedback to market parties.

How to run consistency check

  •  
2.2.16 Read Grid Owner reports and upload in DAM 9/1 13/1 Statnett

Read grid owner reportsandcreateoutput.

Process EDM grid owner reports

  •  

2.3 Stage structure and master data

2.3.1 Set internal DB IDs in EDM 9/1 13/1 Statnett

Need to sync

  • CountryID
  • Maxidson direct DB insert tables

How to sync internal DB ids from EIP to EDM

  •  
2.3.2 Stage Structure data 9/1


13/1
Statnett

Including

  • Market parties
  • Grid structure
  • Balance responsibility contracts
  • Message exchange auths

How to stage structure data

  •  
2.3.3 Run staging integrity 9/1 13/1 Statnett

Staging integrity is usedasalaststepto exclude (remove) data from staging.

How to run staging integrity check

  •  
2.3.4 Stage Master data 9/1 13/1 Statnett

Stage master data including indexes and volumes

How to stage master data

  •  
2.3.5 Check entity counts between staging and landing 9/1 13/1 Statnett

Check counts

  • landing vs staged
  • structure data

Check counts between EDM and EIP

  •  
2.3.8 Logical Backup for DAM DB 9/1 13/1 Accenture

Restore point if something goes wrong in loading. Backup all tables in DAM

Take the logical database backup for DAM database.

DAM DB Back

  •  
2.4 Market party feedback and quality evaluation
2.2.16 Run Data Quality reports in EDM 9/1 13/1 Statnett

Create and publish DQ reports in EDM to market parties.

Create EDM data-quality reports

  •  
2.2.17 Create downloadable reports 9/1 13/1 Statnett Create EDM downloadable reports
  •  
2.3.6 Publish custom EDM report for grid owners and balance suppliers 9/1 13/1 Statnett

Gridownershasaspecial report in DAM, giving them completeness info. This includes:

  • Metering points stopped by DAM
  • Missing exchanges
  • Counts of different types of metering points
  • Contract deviations

Balance suppliers will also get the contract deviations.

Publish grid owner and balance supplier report i EDM

  •  
2.2.20 Notify market parties that feedback is available 9/1 13/1 Statnett

Both email and elhub.no status page

  • NCF
  • Downloadable reports
  •  
2.3.7 Checkpoint Completeness and quality of data in EDM is sufficient for initial migration 9/1 13/1 Statnett

We need to evaluate the quality before going ahead.

Check completeness and quality of data in EDM

  •  
2.3.8 Report migration statistics internally 9/1 13/1 Statnett Report migration statistics internally
  •  
2.5 Load structure data
2.4.1 Ensure indexes for loading 9/1 13/1 Statnett

Ensure indexes for phase LOADING

Create and drop indexes on EDM tables

  •  
2.4.2 Load Market parties 9/1 13/1 Statnett

The following entities must be migrated: Type - Migration Sequence

  • Organization - 1
  • Party- 2
  • Party Function - 3
  • Business Role Relations - 4
  • Message Exchange Authorization - 5

These data are transferred using DB Insert method.

Transfer data from DAM to EIP

  •  
2.4.5 Load Grid Structure 9/1 13/1 Statnett

The following entities must be migrated: Type - Migration Sequence

  • MBA - 10
  • MGA / SGA - 11
  • MGA / SGA Relations - 12

Transferred using FlexSync XML Files:

Transfer data from DAM to EIP

  •  
2.4.6 Load Balance Responsibility Contracts 9/1 13/1 Statnett

Contracts for MGA and SGA to be loaded in one go. Data comes from the same tableinDAM.

  • Primary sequence: 30
  • Relevance inclusion group: 470

Transfer method DB Insert

Transfer data from DAM to EIP

  •  
2.4.RMM Refresh MTM MVs 9/1 13/1 Accenture

This is a dependency for PopulateGridAreaStatusTable, and has to be done after grid area data has been populated.

Refresh MTM MVs

  •  
2.4.9 Ensure DB health of EIP 9/1 13/1 Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
2.4.10 Logical Backup EnergyIP DB 9/1 13/1 Accenture

Take the full EnergyIP database logical backup.

EnergyIp DB Backup

At this moment, nothing is loaded yet to HBASE. Therefore it is sufficient to only back-up the Oracle DB ofEnergyiP.

  •  
2.6 Structure data validation
2.4.7 Run completness reports 9/1 13/1 Statnett 

Reports to run:

  • MarketPartyDataIntegrityCheck
  • MarketPartyDataCount
  • PartyFunctionBusinessRoleDataIntegrityCheck
  • PartyFunctionBusinessRoleDataCount
  • WebPortalUserDataIntegrityCheck
  • WebPortalUserDataCount
  • GridAreaDataIntegrityCheck
  • GridAreaDataCount
  • GridLossParameterIntegrityCheck
  • GridLossParameterDataCount
  • BRCDataIntegrityCheck
  • BRCUserDataCount
  • EBOKeyCheck

creport - How to run a completeness report or a load script

  •  
2.4.8 Check entity counts between EDM and EIP 9/1 13/1 Statnett

Checklandingvs staging vs EIP for structure data

Check counts between EDM and EIP

  •  
2.7 Loading and validating Master Data
2.6.1 Load MasterData 9/1 13/1 Statnett

Master data is loaded from the DAM staging tables to EnergyIP.

  • Primary sequence: 100
  • Relevanceincusiongroup: 500

Data is transferred using XML File transfer method.

Transfer data from DAM to EIP

  •  
2.6.2 Load birth numbers andorganizationnumbertstoEIP 9/1 13/1 Statnett

Load data into Identity and User_Mapping using ODI.

Data is transferred using DB Insert method.

How to load masked identification numbers from EDM to EIP


  •  
2.6.3 Load ADU 9/1 13/1 Statnett

Average Daily Usage is loaded from the DAM staging table to EnergyIP.

Firstpartof this step is to retrieve the internal SP IDs of Metering Points from EIP to DAM

Load ADU (Average Daily Usage)

  •  
2.6.6 Load Aggregated values

9/1

13/1

Statnett

Aggregated values are used in grid loss calculation screens in Elhub Web Portal when changing/tuning grid loss params.

Report to run:

  • PopulateGridAreaStatusTable
  • PopulatePrelimGridLossTable

creport - How to run a completeness report or a load script

  •  
2.6.7 Create virtual channels on connection points in EIP 9/1 13/1 Statnett

Connection points are migrated by EDM, but the virtual channels (with contributors) are created in this step.

Create virtual channels on connection points in EIP

  •  
2.6.8 Ensure DB health of EIP 9/1 13/1 Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
2.6.8.gg Synchronize tables after source systems Truncate 9/1 13/1 Accenture

Synchronize tables after source systems Truncate (GoldenGate)

GoldenGate Configuration Steps (initial load)

  •  
2.6.9 Run completeness reports for exchange, accounting and connection points 9/1 13/1 Statnett

Reports to run:

  • ExchangePointIntegrityCheck
  • ExchangePointCount
  • AccountingPointDataIntegrityCheck
  • AccountingPointDataCount
  • ContractIntegrityCheck
  • ContractCount
  • ConnectionPointIntegrityCheck
  • ConnectionPointCount

creport - How to run a completeness report or a load script

  •  
2.6.9.1 Check entity counts between EDM and EIP 9/1 13/1 Statnett

Checklandingvs staging vs EIP for master data

Check counts between EDM and EIP

  •  
2.6.10 Evaluate completeness and quality of migration 9/1 13/1 Statnett

We need to evaluate the quality before going ahead.

Check completeness and quality of data in EDM

  •  
2.6.11 Logical Backup EnergyIP DB 9/1 13/1 Accenture

Take Logical database backup for EIP database.

EnergyIp DB Backup

At this moment, nothing is loaded yet to HBASE. Therefore it is sufficient to only back-up the Oracle DB ofEnergyiP.

  •  
2.6.12 Logical Backup DAM DB 9/1 13/1 Accenture

Take Logical database backup for DAM database.

DAM DB Backup

  •  

2.8 Put Elhub in business as usual mode

2.7.1 Set x_process_milestone 12/1 13/1 Accenture

Set x_process_milestone

Set X_PROCESS_MILESTONE

  •  
2.7.2 Put EIP in business as usual mode 12/1
13/1 Accenture

Put EIP in business as usual mode

How to switch modes in EIP using ConfigurationManagementTool.sh - Prepare Operations Mode

  •  
2.7.3 Ensure DB health of EIP 12/1
13/1 Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
2.7.4

Synchronize master data to OBIEE (ODI data load)

12/1
13/1 Accenture

Synchronize master data to OBIEE (ODI data load)

Use snapshot date interval from 08.01.2019 to sysdate

Run full load described in section "Steps to run FULL Load" in How to set up ODI Scheduling

  •  
2.7.5 Execute EIDG load 12/1 13/1 Accenture

Execute EIDG load (cache metering points) toensureperformanceofMTM jobs and other jobs operating with large sets of data.

Load EIDG

  •  
2.7.6 Decide to turn on ReadOnly BRSs

16/1


16/1
Statnett
  •  
2.7.7 Turn onReadOnly BRSs

16/1


16/1


Accenture

Turn on access to BRSs thatdonot change the datainElhub. This allows the market parties to QA their datathroughBRSs,andQA their CIS systems at the same time.

List of BRSs to turn on:

  • BRS-NO-303 Request Master Data
  • BRS-NO-315 Request Metering Values
  • BRS-NO-324 Request Imbalance Settlement

Turn on Read Only BRSs

Open/Close BRS for traffic


  •  
2.7.8 Open Elhub Web Portal for all users

16/1

16/1 Accenture

Open Elhub Web Portal for all users

Turn on OHS to start web traffic to Portal

  •  
N/A End migration mode

Accenture

End migration mode

Lift softblackoutwhenportalisavailable

  •  
2.7.8.1 Open access to Elhub Web plugin 16/1 16/1 Accenture

Openupinfrastructuresoexternal parties can access theElhubweb plugin.

Open access to Elhub Web plugin

  •  


2.7.8.2 Open Access to Web Plugin for market parties 16/1 16/1 Accenture

Turn ON WebPlugin access for the market parties

Turn ON/OFF WebPlugin access for the market parties

  •  


2.5.2 Load NOA file 9/1 13/1 Accenture

See guideinnextcolumn. Use manual approach, don't schedule.

Load NOA file

  •  
2.6.8.1

Load Channel Milestones

9/1

13/1

Statnett

Report to run:

  • ChannelMilestoneLoad

Parameters:

  • ChannelType - Interval
  • OverwriteExisting - true

Channel milestone for non profiled metering points is set to Sunday January 20 00:00 so that validation for metering value continuency and reminders will work in the reverification week.

creport - How to run a completeness report or a load script

  •  


2.7.8.3 Start doing daily verification in Elhub Web Portal 14/1
Statnett

Should run daily in the morning until final migration phase.

Sanitysjekk av Elhub Aktørportal

  •  
2.7.9 AddElhubOperators 14/1

Statnett

AddElhuboperators

Log in to the portal using the super user from step 1.8.2 and create the Elhub Operators (and also new super users).

Refer to user documentation: 2.1 Administrate Users and User Profiles

  •  
2.7.10 Create super users for Market Parties in Web Portal 14/1

Statnett

The list is maintained in the Edielportal.

The following needs to be done:

This must be done by an Elhub Operator in the web portal.

The process has been automated by Elsuper - Automagisk brukerlasting

  •  
2.7.11 Handle exceptions with market parties - webportalusers 16/1 18/1 Statnett

Create missing super users manually.

Legge inn superbrukere i Elhub Aktørportal manuelt


  •  
2.9 Market parties QA in Elhub
!!Elhub Web Portal, EMIF and Web Plugin made available to market!!
2.8.0

Sjekkpunkt

Datakvalitet ved initiell migrering ok

16/1

9:00

N/A Elhub Datakvalitet nivå 1 >99,9% totalt for hele markedet > 99% av alle opplastede målepunkter korrekt migrert inn i Elhub
  •  
2.8.11 Copy migration files to EDM operators 16/1 16/1 Accenture

A copy of migration files from initialmigrationarekepton the servers in case there is a need for them in the second migration.

Backup of Old EDM migration files from elhwls servers


  •  
2.8.1

Opprett brukere for ansatte i Elhub Aktørportal

16/1


16/1

Alle markedsaktører

Den enkelte aktørs superbruker oppretter vanlige brukere for sine egne ansatte i Elhub Web Portal.

Gjennomføring av dette steget skal rapporteres i Edielportalen for nett og kraft.

NA

2.8.2 Sjekke tilganger til produksjonsmiljøet for Elhubansatte 14/1 16/1 Statnett

Sjekkliste som brukes av alle Elhubansatte for å sjekke at de har riktige tilganger til alle verktøy.

Sjekke tilganger til produksjonsmiljøet for Elhubansatte

  •  
2.8.3
Sjekkliste for kvalitetssikring av migrerte data

16/1

18/1 Statnett

Sjekkliste som brukes internt av Elhub operatører for å kvalitetssikre data gjennom Elhub Web Portal etter endt initiell migrering og frem til Go Live.

Sjekkliste for kvalitetssikring av Elhub-operatører

  •  
2.8.4 Sjekkliste for nettselskaper 16/1
22/1

Netteiere

Sjekkliste som brukes internt av nettselskaper for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for nettselskaper

Gjennomføring av dette steget skal rapporteres i Edielportalen for nettselskaper.

NA
2.8.5 Sjekkliste for tjenestetilbydere 16/1 22/1

Tjenestetilbyder

Sjekkliste som brukes internt av tjenestetilbyder for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for tjenestetilbyder

NA
2.8.6 Sjekkliste for kraftleverandører og regulert kraftleverandør 16/1 22/1

Kraftleverandører, regulerte kraftleverandører

Sjekkliste som brukes internt av kraftleverandører for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for kraftleverandører

Gjennomføring av dette steget skal rapporteres i Edielportalen for kraftleverandører, ikke regulerte.

NA
2.8.7 Sjekkliste for balanseansvarlige 16/1 22/1

Balanseansvarlige

Sjekkliste som brukes internt av balanseansvarlige for å kvalitetssikre data gjennom Elhub Web Portal og BRSeretter endt initiell migrering og frem til Go Live.

Sjekkliste for balanseansvarlig

NA
2.8.8 Sjekkliste for tredjepart 16/1 22/1

Tredjeparter

Sjekkliste som brukes internt av tredjepart for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for tredjepart

NA
2.8.10 Logical Back up EnergyIP DB 18/1 18/1 Accenture

Take Logical database backup after all corrections to the structure data are done.

TA - ExArch - Infra - Exadata

EnergyIp DB Backup

17.1: Changed date and removed HBASE from step as agreed in morning meeting

  •  
2.8.1.1

Sjekkpunkt

Markedsaktører har kontroll på manglende målepunkter i Elhub

25/1

9:00


Elhub

Alle nettselskaper rapporterer at de har gjennomført oppstartssjekklisten og har kontroll på manglende målepunkter (dvs. at de blir riktig ved neste migrering)


  •  
2.8.9 Verify and send reminders to market parties of grid loss parameters configuration 25/1 25/1 Statnett

Migration is setting default parameters. This step will check which grids (and grid owners) has not changed the grid loss param away from the default values.

Check for grid loss parameters

  •