02 - Initial Migration and Structure Data Establishment

Sist oppdatert 2019-02-25 10:50

Edielportalen:

  • Go Live 02 <Rolle> Initiell migrering og etablering av strukturdata

Execution period

Elhub Go Live: The activities in this section are executed in the period 7/1 to 25/1.

Summary of activities

The activities described in this chapter includes receiving structure data and migration data from the market, to process this data, and to eventually load it into Elhub. This chapter also includes checklists for migrated data to be performed by market parties.

Content overview

Initial migrations contains the following main parts:

  • 2.1 Receive structure data
  • 2.2 Receive migration files from market parties
  • 2.3 Stage structure and master data
  • 2.4 Market party feedback and quality evaluation
  • 2.5 Load structure data
  • 2.6 Structure data validation
  • 2.7 Loading and validating Master Data
  • 2.8 Put Elhub in business as usual mode
  • 2.9 Market parties QA in Elhub

Commissioning steps

Task IDCommissioning stepStart timeDeadlineExecution OrganizationStep descriptionDone
2.0.1Sjekkpunkt

Produksjonsmiljøet for Elhub er klart

8/1

9:00

N/AElhubKapittel 1 i CRB er gjennomført. Ingen blokkerende feil i Elhub-systemet
  •  
2.1 Receive structure data

Most of the steps in this section can be done in parallell. Loading and verification needs to be done at the end.
2.1.1Reminder to market parties about migration7/19/1Statnett

Send reminder on email to market parties about upcoming delta migration.

Send notification to market parties that does migration

  •  
2.1.2Receive structure Data from Ediel7/19/1Statnett

Receive Market Parties from Ediel Portal into DAM landing, including message exchange authorizations.

Retrieve files, then commit to stash-repo for tracking.

Get structure data from external parties


  •  
2.1.3Create old market parties7/19/1Statnett

In the files from EDIEL, only the market parties after Go Live will be included. Since we migrate the history of the master data, it is possible we need to migrate contracts with market parties that don't exist anymore. Those market parties need to be created as well.

Create old marked parties structure data

  •  
2.1.4Receive grid structure data from NOA and create supporting files.7/19/1Statnett

Receive MGAs, MBAs and their relations as files from NOA. Then create additionalWebportalMGA data to augment the production data (e.g. loss parameters and BSL).

Retrieve files, then commit to stash-repo for tracking.

Get and create structure data for grid

  •  
2.1.5Receive structure data from IFS7/19/1Statnett

Retrieve account number per organization.

Retrieve files, then commit to stash-repo for tracking.

Get structure data from external parties

  •  
2.1.6Receive structure data from BRREG7/19/1Statnett

Retrieve organization name per organization number

Retrieve files, then commit to stash-repo for tracking.

Get structure data from external parties

  •  
2.1.7Update postal codes data if needed7/19/1Statnett

If changes has been made by Posten, then update the POSTAL_CODES table (refdata).

Update Postal Codes in DAM

  •  
2.1.8Create structure data for foreign MGAs7/19/1Statnett

Foreign MGAs (FMGA) must be created so they can be received and loaded in DAM via ODI. The dataisallreadycreated,butneeds to be verified.

Create structure data files for foreign grid

  •  
2.1.1Get and create structure data for SGAs7/19/1Statnett

The data is gathered from LIO andsub gridowners and transformed to be on the correct format for EDM.

Get and create structure data for SGAs

  •  
2.1.9Receive Balance Responsibility contracts per MGA and production units from NBS7/19/1Statnett

Receive BRP for BS pr MGA from NBS into DAM landing. Upload into DAM loading must be done after cleanup and upgrade of DAM.

Get structure data from NBS

  •  
2.1.10Receive file with certificate entitled production points from NECS7/19/1Statnett 

Wewillrecievealist of component codes from NECS. These will have to be mapped to metering point IDsbymigrationteam.

Get structure data from NECS

  •  
2.1.11Upload structure data to DAM7/19/1Statnett 

Once we have structure data from all external parties, it can be uploaded using ODI.

How to load structure data in EDM

  •  
2.1.12Ensure internal market parties are added in EDM7/19/1Statnett

Internal market parties are used in EDM to upload connection points.

Ensure internal market parties in EDM

  •  
2.1.13Verify structure data in DAM landing7/19/1Statnett 

Once all structure data has been loaded into DAM landing, verify that all is present.

If there are errors, delete data in tables, fix and upload again.

Validating landing structure data

  •  
2.2 Receive migration files from market parties
2.2.1VerifyfunctionalconfigrationinEDM9/113/1Statnett

Before we start uploading files, we need to ensure that all functional parameters are set correctly.

Verify functional configuration in EDM

  •  
2.2.2Ensure indexes for format and integrity check9/113/1Statnett

Ensure indexes for phase UPLOAD

Create and drop indexes on EDM tables

  •  
2.2.3Verify database health for EDM database9/113/1Statnett

We need to ensure that we have no known issues in the database.

EDM database health check

  •  
2.2.4Open for upload of files to EDM9/113/1Statnett

Configurationparametersneedstobe set

  •  
2.2.5Upload sanity files9/113/1Statnett

Sanity files for DAM to ensure that data reads into DAM ok.

  •  
2.2.6Notify market that EDM is ready for upload9/113/1Statnett

Final notification

Send notification to market parties that does migration

  •  
2.2.7Laste opp migreringsfiler til DAM

9/1

00:00

9/1

22:00

Nettselskaper og kraftleverandører

Data om målepunkt, kunder og kontrakter, samt 1 uke måleverdier for timesavregnede (24.12. – 30.12.). I tillegg minimum siste periodevolum med tilhørende start- og sluttstand, på relevante (profilavregnede) målepunkter

Gjennomføring av dette steget skal rapporteres i Edielportalen.

NA
2.2.8Receive data through migration files

9/1

00:00

9/1

22:00

Statnett 

Includes format check and integrity check.

Both master data and metering values

Format and integrity check

  •  
2.2.9

Verify all grid owners uploaded data

9/113/1Statnett

We need data from all grid owners.

See Format and integrity check postconditions

If grid owner has not uploaded, call them.

  •  
2.2.11Ensure indexes for timeline check9/113/1Statnett

Ensure indexes for phase TIMELINE

Create and drop indexes on EDM tables

  •  
2.2.9.1Create and upload connection points file9/113/1Statnett

Connection points arenotrecievedfrommarketparties,andneeds to be uploaded by DAM Operators. It is also used as a sanity test of file upload.

Create and upload connection points

  •  
2.2.10Block upload of files to EDM9/113/1Statnett

Configurationparametersneedstobe set

  •  
2.2.12

Gather locked stats on entity tables after loading

9/113/1Statnett

Make sure stats are gathered

Gather stats on EDM tables

  •  
2.2.13Run timeline check9/113/1Statnett

Timeline check replaces old data and updates data to set them as current (eligible for staging and consistency)

How to run timeline check

  •  
2.2.14Ensure indexes for phase CONSISTENCY9/113/1Statnett

Ensure indexes for phase CONSISTENCY

Create and drop indexes on EDM tables

  •  
2.2.15Run consistency check9/113/1Statnett

Run consistency on the data, including meta list and core process.

This step can be skipped and re-run in a later stage in case of time constraint.No consiequence for data loaded into Elhub, but is required for feedback to market parties.

How to run consistency check

  •  
2.2.16Read Grid Owner reports and upload in DAM9/113/1Statnett

Read grid owner reportsandcreateoutput.

Process EDM grid owner reports

  •  

2.3 Stage structure and master data

2.3.1Set internal DB IDs in EDM9/113/1Statnett

Need to sync

  • CountryID
  • Maxidson direct DB insert tables

How to sync internal DB ids from EIP to EDM

  •  
2.3.2Stage Structure data9/1


13/1
Statnett

Including

  • Market parties
  • Grid structure
  • Balance responsibility contracts
  • Message exchange auths

How to stage structure data

  •  
2.3.3Run staging integrity9/113/1Statnett

Staging integrity is usedasalaststepto exclude (remove) data from staging.

How to run staging integrity check

  •  
2.3.4Stage Master data9/113/1Statnett

Stage master data including indexes and volumes

How to stage master data

  •  
2.3.5Check entity counts between staging and landing9/113/1Statnett

Check counts

  • landing vs staged
  • structure data

Check counts between EDM and EIP

  •  
2.3.8Logical Backup for DAM DB9/113/1Accenture

Restore point if something goes wrong in loading. Backup all tables in DAM

Take the logical database backup for DAM database.

DAM DB Back

  •  
2.4 Market party feedback and quality evaluation
2.2.16Run Data Quality reports in EDM9/113/1Statnett

Create and publish DQ reports in EDM to market parties.

Create EDM data-quality reports

  •  
2.2.17Create downloadable reports9/113/1StatnettCreate EDM downloadable reports
  •  
2.3.6Publish custom EDM report for grid owners and balance suppliers9/113/1Statnett

Gridownershasaspecial report in DAM, giving them completeness info. This includes:

  • Metering points stopped by DAM
  • Missing exchanges
  • Counts of different types of metering points
  • Contract deviations

Balance suppliers will also get the contract deviations.

Publish grid owner and balance supplier report i EDM

  •  
2.2.20Notify market parties that feedback is available9/113/1Statnett

Both email and elhub.no status page

  • NCF
  • Downloadable reports
  •  
2.3.7Checkpoint Completeness and quality of data in EDM is sufficient for initial migration9/113/1Statnett

We need to evaluate the quality before going ahead.

Check completeness and quality of data in EDM

  •  
2.3.8Report migration statistics internally9/113/1StatnettReport migration statistics internally
  •  
2.5 Load structure data
2.4.1Ensure indexes for loading9/113/1Statnett

Ensure indexes for phase LOADING

Create and drop indexes on EDM tables

  •  
2.4.2Load Market parties9/113/1Statnett

The following entities must be migrated: Type - Migration Sequence

  • Organization - 1
  • Party- 2
  • Party Function - 3
  • Business Role Relations - 4
  • Message Exchange Authorization - 5

These data are transferred using DB Insert method.

Transfer data from DAM to EIP

  •  
2.4.5Load Grid Structure9/113/1Statnett

The following entities must be migrated: Type - Migration Sequence

  • MBA - 10
  • MGA / SGA - 11
  • MGA / SGA Relations - 12

Transferred using FlexSync XML Files:

Transfer data from DAM to EIP

  •  
2.4.6Load Balance Responsibility Contracts9/113/1Statnett

Contracts for MGA and SGA to be loaded in one go. Data comes from the same tableinDAM.

  • Primary sequence: 30
  • Relevance inclusion group: 470

Transfer method DB Insert

Transfer data from DAM to EIP

  •  
2.4.RMMRefresh MTM MVs9/113/1Accenture

This is a dependency for PopulateGridAreaStatusTable, and has to be done after grid area data has been populated.

Refresh MTM MVs

  •  
2.4.9Ensure DB health of EIP9/113/1Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
2.4.10Logical Backup EnergyIP DB9/113/1Accenture

Take the full EnergyIP database logical backup.

EnergyIp DB Backup

At this moment, nothing is loaded yet to HBASE. Therefore it is sufficient to only back-up the Oracle DB ofEnergyiP.

  •  
2.6 Structure data validation
2.4.7Run completness reports9/113/1Statnett 

Reports to run:

  • MarketPartyDataIntegrityCheck
  • MarketPartyDataCount
  • PartyFunctionBusinessRoleDataIntegrityCheck
  • PartyFunctionBusinessRoleDataCount
  • WebPortalUserDataIntegrityCheck
  • WebPortalUserDataCount
  • GridAreaDataIntegrityCheck
  • GridAreaDataCount
  • GridLossParameterIntegrityCheck
  • GridLossParameterDataCount
  • BRCDataIntegrityCheck
  • BRCUserDataCount
  • EBOKeyCheck

creport - How to run a completeness report or a load script

  •  
2.4.8Check entity counts between EDM and EIP9/113/1Statnett

Checklandingvs staging vs EIP for structure data

Check counts between EDM and EIP

  •  
2.7 Loading and validating Master Data
2.6.1Load MasterData9/113/1Statnett

Master data is loaded from the DAM staging tables to EnergyIP.

  • Primary sequence: 100
  • Relevanceincusiongroup: 500

Data is transferred using XML File transfer method.

Transfer data from DAM to EIP

  •  
2.6.2Load birth numbers andorganizationnumbertstoEIP9/113/1Statnett

Load data into Identity and User_Mapping using ODI.

Data is transferred using DB Insert method.

How to load masked identification numbers from EDM to EIP


  •  
2.6.3Load ADU9/113/1Statnett

Average Daily Usage is loaded from the DAM staging table to EnergyIP.

Firstpartof this step is to retrieve the internal SP IDs of Metering Points from EIP to DAM

Load ADU (Average Daily Usage)

  •  
2.6.6Load Aggregated values

9/1

13/1

Statnett

Aggregated values are used in grid loss calculation screens in Elhub Web Portal when changing/tuning grid loss params.

Report to run:

  • PopulateGridAreaStatusTable
  • PopulatePrelimGridLossTable

creport - How to run a completeness report or a load script

  •  
2.6.7Create virtual channels on connection points in EIP9/113/1Statnett

Connection points are migrated by EDM, but the virtual channels (with contributors) are created in this step.

Create virtual channels on connection points in EIP

  •  
2.6.8Ensure DB health of EIP9/113/1Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
2.6.8.ggSynchronize tables after source systems Truncate9/113/1Accenture

Synchronize tables after source systems Truncate (GoldenGate)

GoldenGate Configuration Steps (initial load)

  •  
2.6.9Run completeness reports for exchange, accounting and connection points9/113/1Statnett

Reports to run:

  • ExchangePointIntegrityCheck
  • ExchangePointCount
  • AccountingPointDataIntegrityCheck
  • AccountingPointDataCount
  • ContractIntegrityCheck
  • ContractCount
  • ConnectionPointIntegrityCheck
  • ConnectionPointCount

creport - How to run a completeness report or a load script

  •  
2.6.9.1Check entity counts between EDM and EIP9/113/1Statnett

Checklandingvs staging vs EIP for master data

Check counts between EDM and EIP

  •  
2.6.10Evaluate completeness and quality of migration9/113/1Statnett

We need to evaluate the quality before going ahead.

Check completeness and quality of data in EDM

  •  
2.6.11Logical Backup EnergyIP DB9/113/1Accenture

Take Logical database backup for EIP database.

EnergyIp DB Backup

At this moment, nothing is loaded yet to HBASE. Therefore it is sufficient to only back-up the Oracle DB ofEnergyiP.

  •  
2.6.12Logical Backup DAM DB9/113/1Accenture

Take Logical database backup for DAM database.

DAM DB Backup

  •  

2.8 Put Elhub in business as usual mode

2.7.1Set x_process_milestone12/113/1Accenture

Set x_process_milestone

Set X_PROCESS_MILESTONE

  •  
2.7.2Put EIP in business as usual mode12/1
13/1Accenture

Put EIP in business as usual mode

How to switch modes in EIP using ConfigurationManagementTool.sh - Prepare Operations Mode

  •  
2.7.3Ensure DB health of EIP12/1
13/1Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
2.7.4

Synchronize master data to OBIEE (ODI data load)

12/1
13/1Accenture

Synchronize master data to OBIEE (ODI data load)

Use snapshot date interval from 08.01.2019 to sysdate

Run full load described in section "Steps to run FULL Load" in How to set up ODI Scheduling

  •  
2.7.5Execute EIDG load12/113/1Accenture

Execute EIDG load (cache metering points) toensureperformanceofMTM jobs and other jobs operating with large sets of data.

Load EIDG

  •  
2.7.6Decide to turn on ReadOnly BRSs

16/1


16/1
Statnett
  •  
2.7.7Turn onReadOnly BRSs

16/1


16/1


Accenture

Turn on access to BRSs thatdonot change the datainElhub. This allows the market parties to QA their datathroughBRSs,andQA their CIS systems at the same time.

List of BRSs to turn on:

  • BRS-NO-303 Request Master Data
  • BRS-NO-315 Request Metering Values
  • BRS-NO-324 Request Imbalance Settlement

Turn on Read Only BRSs

Open/Close BRS for traffic


  •  
2.7.8Open Elhub Web Portal for all users

16/1

16/1Accenture

Open Elhub Web Portal for all users

Turn on OHS to start web traffic to Portal

  •  
N/AEnd migration mode

Accenture

End migration mode

Lift softblackoutwhenportalisavailable

  •  
2.7.8.1Open access to Elhub Web plugin16/116/1Accenture

Openupinfrastructuresoexternal parties can access theElhubweb plugin.

Open access to Elhub Web plugin

  •  


2.7.8.2Open Access to Web Plugin for market parties16/116/1Accenture

Turn ON WebPlugin access for the market parties

Turn ON/OFF WebPlugin access for the market parties

  •  


2.5.2Load NOA file9/113/1Accenture

See guideinnextcolumn. Use manual approach, don't schedule.

Load NOA file

  •  
2.6.8.1

Load Channel Milestones

9/1

13/1

Statnett

Report to run:

  • ChannelMilestoneLoad

Parameters:

  • ChannelType - Interval
  • OverwriteExisting - true

Channel milestone for non profiled metering points is set to Sunday January 20 00:00 so that validation for metering value continuency and reminders will work in the reverification week.

creport - How to run a completeness report or a load script

  •  


2.7.8.3Start doing daily verification in Elhub Web Portal14/1
Statnett

Should run daily in the morning until final migration phase.

Sanitysjekk av Elhub Aktørportal

  •  
2.7.9AddElhubOperators14/1

Statnett

AddElhuboperators

Log in to the portal using the super user from step 1.8.2 and create the Elhub Operators (and also new super users).

Refer to user documentation: 2.1 Administrate Users and User Profiles

  •  
2.7.10Create super users for Market Parties in Web Portal14/1

Statnett

The list is maintained in the Edielportal.

The following needs to be done:

This must be done by an Elhub Operator in the web portal.

The process has been automated by Elsuper - Automagisk brukerlasting

  •  
2.7.11Handle exceptions with market parties - webportalusers16/118/1Statnett

Create missing super users manually.

Legge inn superbrukere i Elhub Aktørportal manuelt


  •  
2.9 Market parties QA in Elhub
!!Elhub Web Portal, EMIF and Web Plugin made available to market!!
2.8.0

Sjekkpunkt

Datakvalitet ved initiell migrering ok

16/1

9:00

N/AElhubDatakvalitet nivå 1 >99,9% totalt for hele markedet > 99% av alle opplastede målepunkter korrekt migrert inn i Elhub
  •  
2.8.11Copy migration files to EDM operators16/116/1Accenture

A copy of migration files from initialmigrationarekepton the servers in case there is a need for them in the second migration.

Backup of Old EDM migration files from elhwls servers


  •  
2.8.1

Opprett brukere for ansatte i Elhub Aktørportal

16/1


16/1

Alle markedsaktører

Den enkelte aktørs superbruker oppretter vanlige brukere for sine egne ansatte i Elhub Web Portal.

Gjennomføring av dette steget skal rapporteres i Edielportalen for nett og kraft.

NA

2.8.2Sjekke tilganger til produksjonsmiljøet for Elhubansatte14/116/1Statnett

Sjekkliste som brukes av alle Elhubansatte for å sjekke at de har riktige tilganger til alle verktøy.

Sjekke tilganger til produksjonsmiljøet for Elhubansatte

  •  
2.8.3
Sjekkliste for kvalitetssikring av migrerte data

16/1

18/1Statnett

Sjekkliste som brukes internt av Elhub operatører for å kvalitetssikre data gjennom Elhub Web Portal etter endt initiell migrering og frem til Go Live.

Sjekkliste for kvalitetssikring av Elhub-operatører

  •  
2.8.4Sjekkliste for nettselskaper16/1
22/1

Netteiere

Sjekkliste som brukes internt av nettselskaper for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for nettselskaper

Gjennomføring av dette steget skal rapporteres i Edielportalen for nettselskaper.

NA
2.8.5Sjekkliste for tjenestetilbydere16/122/1

Tjenestetilbyder

Sjekkliste som brukes internt av tjenestetilbyder for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for tjenestetilbyder

NA
2.8.6Sjekkliste for kraftleverandører og regulert kraftleverandør16/122/1

Kraftleverandører, regulerte kraftleverandører

Sjekkliste som brukes internt av kraftleverandører for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for kraftleverandører

Gjennomføring av dette steget skal rapporteres i Edielportalen for kraftleverandører, ikke regulerte.

NA
2.8.7Sjekkliste for balanseansvarlige16/122/1

Balanseansvarlige

Sjekkliste som brukes internt av balanseansvarlige for å kvalitetssikre data gjennom Elhub Web Portal og BRSeretter endt initiell migrering og frem til Go Live.

Sjekkliste for balanseansvarlig

NA
2.8.8Sjekkliste for tredjepart16/122/1

Tredjeparter

Sjekkliste som brukes internt av tredjepart for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering og frem til Go Live.

Sjekkliste for tredjepart

NA
2.8.10Logical Back up EnergyIP DB18/118/1Accenture

Take Logical database backup after all corrections to the structure data are done.

TA - ExArch - Infra - Exadata

EnergyIp DB Backup

17.1: Changed date and removed HBASE from step as agreed in morning meeting

  •  
2.8.1.1

Sjekkpunkt

Markedsaktører har kontroll på manglende målepunkter i Elhub

25/1

9:00


Elhub

Alle nettselskaper rapporterer at de har gjennomført oppstartssjekklisten og har kontroll på manglende målepunkter (dvs. at de blir riktig ved neste migrering)


  •  
2.8.9Verify and send reminders to market parties of grid loss parameters configuration25/125/1Statnett

Migration is setting default parameters. This step will check which grids (and grid owners) has not changed the grid loss param away from the default values.

Check for grid loss parameters

  •