05 - Final migration

Sist oppdatert 2019-02-25 10:50

Edielportalen:

  • Go Live 05 <Rolle> Endelig migrering                      

Execution period

Elhub Go Live: The activities in this section are executed in the period 1/2 to 9/2.

Summary of activities

The activities described in this chapter include deleting data in EIP and EDM, receiving final round of migration data from the market, to process this data, and to eventually load it into Elhub. This chapter also includes checklists for migrated data to be performed by market parties.


Content overview

Final migration contains the following main parts:

  • 5.0 Making EIP ready for final migration
  • 5.1 Making DAM ready for final migration 
  • 5.2 Receive and validate master data and metering values
  • 5.3 Stage and load master data and metering values
  • 5.4 Set metering point range
  • 5.5 Market parties QA in Elhub


Viktig informasjon til alle markedsaktører


Viktig informasjon 1: Hovedprinsippene for bufring av forretningsprosesser:


  • Leverandørbytteprosesser bufres fra og med 24. januar. Deretter starter man å bufre andre prosesser suksessivt i henhold til CRB.
  • Fra og med fredag klokka 00:00 skal alle nye prosesser som oppstår ikke lagres i KIS, men bufres utenfor KIS.
  • 1. februar skal man sikre at alle meldinger som er kommet på Edifact er behandlet.
  • Når punktet over er gjort kan man gjøre migreringsuttrekket fra databasen sin. Det skal gjøres en gang i løpet av fredagen, eller senest søndag 3/2 klokka 14:00 
  • Fra og med bufring starter og inntil man har gjort migreringsuttrekket på fredag kan prosesser ikke bufres i KIS. Da blir de i tilfellet både migrert og debufret, noe som vil følge til at meldingen vil avvises når den spilles av til Elhub og aldri sendt til mottakeren. Dette vil skape ekstraarbeid for alle parter. Viktig at alle markedsaktører bekrefter at de har en vanntett prosedyre som sikrer at ingen prosesser blir både migrert og bufret.
  • Opplasting av migreringsuttrekket til Elhub kan gjøres fra søndag 3/2 klokka 14:00, og alle aktørene må ha gjort det innen mandag klokka 14:00.


 Viktig informasjon 2: Hvordan gjøre endringer eller tillegg til de migrerte dataene


Dette kan gjøres på to måter. 

  1. Opprette manglende målepunkter/kontrakter eller målepunkter som har blitt stoppet i migrering
    1. Bruk prosedyren i 6.1.1 Last opp Excel-filer for manglende målepunkter
  2. Endring på målepunkter som har blitt migrert
    1. Dette skal i utgangspunktet gjøres via BRSer. Dersom man står fast, send henvendelse til post@elhub.no


Commissioning steps

Task IDCommissioning stepStart timeDeadlineExecution OrganizationStep descriptionDone
5.0 Making EIP ready for final migration
!!Elhub Web Portal, EMIF and Web Plugin made UNavailable to market!!
5.0.0Stop doing daily verifications in the Web Portal2/2
StatnettNot necessary and not possible as the Web Portal will be down.
  •  
N/AStart migration mode

1/2

16:00

1/2

17:00

Accenture

Start migration mode

Set soft blackout before starting any activities

  •  
5.0.1.ohsTurn off OHS to stop all web traffic toElhub1/21/2Accenture

Turn off OHS to stop all web traffic to Elhub

Turn off OHS to stop all web traffic to Elhub

  •  
5.0.1.fwTurn off WebPlugin access for the market parties1/21/2Accenture

Turn OFF WebPlugin access for the market parties

Turn ON/OFF WebPlugin access for the market parties

  •  
5.0.1.oag

Turn off OAG to stop all web service traffic toElhub

1/21/2Accenture

Turn off OAG to stop all web service traffic to Elhub

Turn off OAG (EMIF) to stop all web service traffic to Elhub


  •  
5.0.1

Put EIP in migration mode

1/2

1/2

Accenture

Put EIP in migration mode

How to switch modes in EIP using ConfigurationManagementTool.sh - Prepare Migration Mode

  •  
5.0.2Logical Backup EnergyIP DB

1/2

1/2

Accenture

Take a logical backup before wiping EIP

TA - ExArch - Infra - Exadata

CRB BACKUP USING LOGICAL BACKUP METHOD

  •  
5.0.3.ggPause GG synchronization1/21/2Accenture

Pause GG synchronization

Goldengate Start and Stop steps .

  •  
5.0.3.odiTurn off scheduled procedures ODI

1/2

1/2

 Accenture

Turn off scheduled procedures ODI

Turn off/on scheduled procedures ODI

  •  
5.0.3Delete master data from Energy IP

1/2

2/2


Accenture

Run the final wipe of EnergyIP. Note that this is not the same as the initial wipe and will leave some of the data in the system. Ensure backup step 5.0.2 is completed.

Secondary delete of data from EnergyIP

  •  
5.0.3.1Refresh MTM MVs2/22/2Accenture

This is a dependency for PopulateGridAreaStatusTable, and has to be done after grid area data has been populated.

Refresh MTM MVs

  •  
5.0.4Ensure DB health of EIP

2/2

2/2

Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.0.6Delete all messages on OSB queues. 2/2
2/2Accenture

Delete all messages on OSB queues.

Delete all messages on OSB/SOA/BPM queues.

  •  
5.0.7Delete all messages on SOA/BPM queues. 2/2
2/2Accenture

Delete all messages on SOA/BPM queues.

Delete all messages on OSB/SOA/BPM queues.

  •  
5.0.8Terminate any running processes in SOA/BPM.2/2

2/2

Accenture

Terminate any running processes in SOA/BPM.

Terminate any running processes in SOA/BPM

  •  
5.0.5Synchronize tables after source systems Truncate2/22/2Accenture

Synchronize tables after source systems Truncate (GoldenGate)

GoldenGate Configuration Steps (initial load)

  •  
5.0.9

Turn on OAG to start all web service traffic

(BRS's will be off so this is for migration only)

2/22/2Accenture

Turn on both OAG instances.

Turn on OAG (EMIF) to start all web service traffic to Elhub

  •  
5.0.10Open DAM for file upload2/2

2/2

23:59

Statnett

Configurationparametersneedstobe set

  •  
5.0.11Load sanity files2/2

2/2

23:59

StatnettLoad sanity files in EDM
  •  
5.0.12Notify certain market parties2/2

2/2

23:59

Statnett
  •  
5.1 Making DAM ready final migration
5.1.2Ensurepartitonson EDM tables

31/1

08:00 CET

31/1Accenture

Ensure that partitioned tables have the correct partitions.

Partitioning metering value tables

  •  
5.1.3Delete old migration files

31/1

31/1

Accenture

Remove old migration files

Delete EDM migration files from elhwls servers

  •  
5.1.3.aClean the quarantine folder in DMZ31/131/1Accenture

Manually clean the quarantine folder in DMZ

Clean the quarantine folder in DMZ

  •  
5.1.6Logical Backup DAM DB

31/1

31/1

12:00 CET

Accenture

Logical Backup before westartmigration. Will include updated config and/or packages/code.

DAM DB Backup  

  •  
5.1.4Reminder to market parties about migration

31/1

31/1

Statnett

Send reminder on email to market parties about upcoming migration.

Send notification to market parties that does migration

  •  
5.1.1Delete master data from EDM database

31/1

31/1


 Statnett

Remove master data from EDM database

Delete master data from EDM database


  •  
5.1.5Verify structure data in DAM landing

31/1

31/1

Statnett

We rerun the step from initial migration just to make sure that nothing has happened.

Validating landing structure data"

  •  
5.2 Receive and validate master data and metering values
5.2.1Ensure indexes for receiving files

31/1

31/1

Statnett

Ensure indexes for phase UPLOAD

Create and drop indexes on EDM tables

  •  
5.2.2Verify database health for EDM database

31/1

31/1

Statnett

We need to ensure that we have no known issues in the database.

EDM database health check

  •  
5.2.3Verify functional configuration in EDM

31/1

31/1

Statnett

Before we start uploading files, we need to ensure that all functional parameters are set correctly. Verify:

  • NBS_BRP_FROM_DATE
  • NOW_DATE
  • EDIEL_VAL_ST_DATE

Verify functional configuration in EDM

  •  
5.2.3.9Load sanity files31/131/1StatnettLoad the sanity point files and ensure all is ok.
  •  
5.2.4Open for upload of files to EDM

31/1

31/1

23:59


Statnett

Configurationparametersneedstobe set

  •  
5.2.4.1Notify market that DAM is ready31/1

31/1

23:59

Statnett

SendemailtoDAM users that DAM is open.

Send notification to market parties that does migration

  •  
5.2.5Laste opp migreringsfiler til DAM

1/2

00:00


4/2

14:00


Nettselskaper og kraftleverandører

Data om målepunkt, kunder og kontrakter, samt 1 uke måleverdier for timesavregnede (21/1 – 27/1). I tillegg minimum siste periodevolum med tilhørende start- og sluttstand, på relevante (profilavregnede) målepunkter.

DAM vil være åpent i periodene:

• Fredag 1/2 00:00 til 16:00
• Søndag 3/2 00:00 til mandag 4/2 14:00

Gjennomføring av dette steget skal rapporteres i Edielportalen.

NA
5.2.6

Receive master data and metering values


1/2

00:00


4/2

14:00


Statnett

Includes:

  • accounting points with contracts and customers
  • exchange point
  • metering values
    • special series
    • limitedamoutofprofiled reads (for EAC)

Format and integrity check

  •  
5.2.7

Sjekkpunkt

Migrering er komplett fra alle markedsaktører

5/2

12:00

N/AElhub

Alle nettselskaper skal ha lastet opp sine migreringsfiler.


  •  
5.2.8

Rapporter til NBS

4.2

00:00

6.2

23:59

Nettselskaper

Måleverdier for 28.1. - 3.2. blir rapportert til NBS.

Gjennomføring av dette steget skal rapporteres i Edielportalen.

N/A
5.2.8.5Create and upload connection points file9/113/1Statnett

Connection pointsarenotrecievedfrommarketparties,andneedsto be uploaded by DAM Operators. It is also used as a sanity test of file upload.

Upload connection points

  •  
5.2.9

Block upload of files to EDM

5/2

7/2

Statnett

Configurationparametersneedstobe set to -1

  •  
5.2.11Ensure indexes for timeline check

5/2

7/2

Statnett

Ensure indexes for phase TIMELINE

Create and drop indexes on EDM tables

  •  
5.2.13Run timeline check

5/2

7/2

Statnett

Timeline check replaces old data and updates data to set them as current (eligible for staging and consistency)

How to run timeline check

  •  
5.2.14Ensure indexes for next phase

5/2

7/2

Statnett

Ensure indexes for phase CONSISTENCY

Create and drop indexes on EDM tables

  •  
5.2.16Read Grid Owner reports and upload in DAM5/213/1Statnett

Read grid owner reports, create output and load into DAM.

Process EDM grid owner reports

  •  
5.3 Stage and load master data and metering values
5.3.1Run staging integrity

5/2

7/2

Statnett

Staging integrity is used as a last step to exclude (remove) data from staging.

How to run staging integrity check

  •  
5.3.2Stage master data, including post checks

5/2

7/2

Statnett

Stage master data.

How to stage master data

  •  
5.3.3Run go and bs reports5/27/2StatnettUsed in quality assurance
  •  
5.3.17Check completeness and quality of migration

5/2

7/2

Statnett

We need to check the completeness and quality before going ahead.

Check completeness and quality of data in EDM

Do in parallel with backup.

  •  
5.3.6Logical Backup DAM DB

5/2

5/2

Accenture

Backup staging result (*_RESULT) tables,stagingmetatables, and calculated_metering_values. Take logical backup if loading completely fails.

DAM DB Backup 

This step can be skipped in case of time-constraint, but gained time is limited by previous step being run in parallel.

  •  
5.3.7Ensure DB health of EIP

5/2

7/2

Statnett

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.8Update Migration_Sequence in DAM.

5/2

7/2

Statnett

Update Migration_Sequence table's Last_Executed_primary column to NULL in EDM where Relevance_Inclusion_Group_Id is equal or greater than 500


See exception handling chapter in


Transfer data from DAM to EIP


  •  
5.3.9.prepMonitor masterdata load5/27/2

Accenture

Statnett

Eyes on glass monitoring and take remediating actions to keep load speed above threshold (TBD).

Key resources:

  • EIP (Radomir)
  • DBA
  • ODI (Ashish?)

Monitor master data load

  •  
5.3.9LoadmasterdatatoElhub

5/2

7/2

Statnett

This follows the same process as for full load.

Transfer data from DAM to EIP

  •  
5.3.10Load birth numbers andorganizationnumbertsto EIP

5/2

7/2

Statnett

Load data into Identity and User_Mapping using ODI.

Data is transferred using DB Insert method.

How to load masked identification numbers from EDM to EIP

  •  
5.3.11Load ADU

5/2

7/2

Statnett

Average Daily Usage is loaded from the DAM staging table to EnergyIP.

Firstpartof this step is to retrieve the internal SP IDs of Metering Points from EIP to DAM

Load ADU (Average Daily Usage)

  •  
5.3.12Ensure DB health of EIP

5/2

7/2

Statnett

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.14

Create virtual channels on connection points in EIP

5/2

7/2

Statnett

Create virtual channels on connection points in EIP

Create virtual channels on connection points in EIP

  •  
5.3.15Ensure DB health of EIP

5/2

7/2

Statnett

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.15.ggSynchronize tables after source systems Truncate

Accenture

Synchronize tables after source systems Truncate (GoldenGate)

GoldenGate Configuration Steps (initial load)

DO NOT execute unless explicitly instructed.

5.3.16Run completeness reports for exchange and accounting points, and check results.

5/2

7/2

Statnett

Reports to run:

  • ExchangePointIntegrityCheck
  • ExchangePointCount
  • AccountingPointDataIntegrityCheck
  • AccountingPointDataCount
  • ContractIntegrityCheck
  • ContractCount
  • EBOKeyCheck

How to run a completeness report or a load script

  •  
5.3.17Check entity counts between EDM and EIP

5/2

7/2

Statnett

Checklandingvs staging vs EIP for master data

Check counts between EDM and EIP

  •  
5.3.18Logical Backup EnergyIP DB

6/2

7/2

Accenture

Take logical database backup after loading latest master data.

TA - ExArch - Infra - Exadata

CRB BACKUP USING LOGICAL BACKUP METHOD

This step can be skipped in case of time-constraint.

  •  
5.3.20Set x_process_milestone6/2


7/2

Accenture

Set x_process_milestone to avoid migrated data triggering creation of these messages.

Set X_PROCESS_MILESTONE

  •  
5.3.21Put EIP in business as usual mode6/2


7/2

Accenture

Switch to Operations Mode

How to switch modes in EIP using ConfigurationManagementTool.sh - Prepare Operations Mode

  •  
5.3.22Ensure DB health of EIP6/2


7/2

Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.22.bAllocate CPU for operations6/2


7/2

Accenture

Allocate CPU for operations

CPU allocation for operations

  •  
5.3.24Synchronize master data to OBIEE (ODI data load)6/2


7/2

Accenture

Synchronize master data to OBIEE (ODI data load)

Use snapshot date interval from 01.02.2019 to sysdate

Run full load described in section "Steps to run FULL Load" in How to set up ODI Scheduling

  •  
5.3.25Execute EIDG load6/2


7/2

Accenture

Execute EIDG load (cache metering points) toensureperformanceofMTM jobs and other jobs operating with large sets of data.

Load EIDG

  •  
5.3.22.c

Turn on OHS to start web traffic to Portal and Plugin


6/2
7/2
Accenture

Turn on the portal,pluginandloginOHS instances.

Turn on OHS to start web traffic to Portal and Plugin

Note: WebPlugin Access is turned off for Market Parties at this point.

  •  
N/AEnd migration mode

Accenture

End migration mode

Lift soft blackout whenportalis available

  •  
5.3.23Turn on ReadOnly BRSs

6/2

7/2

Accenture

Turn on access to BRSs thatdonot change the datainElhub. This allows the market parties to QA their datathroughBRSs,andQA their CIS systems at the same time.

List of BRSs to turn on:

  • BRS-NO-303 Request Master Data
  • BRS-NO-315 Request Metering Values
  • BRS-NO-324 Request Imbalance Settlement

Turn on Read Only BRSs

Open/Close BRS for traffic

  •  
5.3.26
Open Access to Web Plugin for market parties6/27/2Accenture

Turn ON WebPlugin access for the market parties

Turn ON/OFF WebPlugin access for the market parties

  •  


5.3.27Set metering point range for internal market parties6/2


7/2

Accenture

Set metering point range for internal market parties

Set metering point range for internal metering points in Elhub

  •  
5.4 Set metering point range and channel milestone
This section is done in parallel with Market Parties QA
5.3.13Load Channel Milestones

5/2

7/2

Statnett

Report to run:

  • ChannelMilestoneLoad

Parameters:

  • ChannelType - Interval
  • OverwriteExisting - true

Channel milestone for non profiled metering points is set to Sunday February 10 00:00 so that validation for metering value continuency and reminders will work in the start up week.

creport - How to run a completeness report or a load script

  •  
5.4.2Start doing daily verification in Elhub Web Portal8/2
Statnett

Should run daily in the morning until final migration phase.

Sanitysjekk av Elhub Aktørportal

  •  
5.X Consistency Check and Dataquality assessment in DAM
This section should run in parallel with Market Parties QA and can be started as soon asEIDGand OBIEE load is done
5.X.1Run consistency checks

7/2

7/2

Statnett

Run consistency on the data,includingmetalistandcore process.

This step can be skipped in case of time constraint.

How to run consistency check

  •  
5.X.2Rundataqualityreports and publish downloadable reports in DAM and elhub.no

7/2

7/2

Statnett

The downloadable reports include an overview over failed metering points, as well as metering points with insufficient metering values. Verify that all special series have been received from grid owners.

Create EDM data-quality reports

  •  
5.X.3Notify market parties that NCFs are ready for download

7/2

7/2

Statnett

Notify market parties by email when all checks have run, and reports have been published in DAM.

Send notification to market parties that does migration

  •  
5.X.4Check completeness and quality of migration

7/2

7/2

Statnett

We need to check the completeness and quality before going ahead.

Check completeness and quality of data in EDM

  •  
5.X.5Report migration statistics internally7/27/2StatnettReport migration statistics internally
  •  
5.X.6Logical Backup DAM DB

8/2

8/2

Accenture

Take Logical database backup. Backup-migration is done.

DAM DB Backup

This step can be skipped in case of time-constraint.

  •  
5.5 Market parties QA in Elhub
!!Elhub Web Portal, EMIF and Web Plugin made available to market!!
Steps 5.5.1 to 5.5.6 (check list activities) can be run in parallell.
5.5.1Sjekkliste for kvalitetssikring av migrerte data

8/2

8/2

16:00

Statnett

Sjekkliste som brukes internt av Elhub operatører for å kvalitetssikre data gjennom Elhub Web Portal etter endt initiell migrering og frem til Go Live.

Sjekkliste for kvalitetssikring av Elhub-operatører

  •  
5.5.2Sjekkliste for nettselskaper

8/2

8/2

16:00 for rapportering av avvik


Netteiere

Sjekkliste som brukes internt av nettselskaper for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for nettselskaper

Gjennomføring av dette steget skal rapporteres i Edielportalen for nettselskaper.

Når man selvregistrerer at dette steget er utført i Edielportalen, skal man i Notat-feltet notere hvor mange målepunkt man mangler i Elhub.

For eksempel dersom det mangler 10 målepunkter, skriver man følgende i notatet: 10

NA
5.5.3Sjekkliste for tjenestetilbyder

8/2

14/2 for utbedring av avvik

Tjenestetilbyder

Sjekkliste som brukes internt av tjenestetilbyder for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for tjenestetilbyder


NA
5.5.4Sjekkliste for kraftleverandører og regulert kraftleverandør

8/2

8/2

16:00 for rapportering av avvik


Kraftleverandører og regulerte kraftleverandører

Sjekkliste som brukes internt av kraftleverandører for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for kraftleverandører

Gjennomføring av dette steget skal rapporteres i Edielportalen for kraftleverandører, ikke regulerte.

Når man selvregistrerer at dette steget er utført i Edielportalen, skal man i Notat-feltet notere hvor mange kontrakter man mangler i Elhub.

For eksempel dersom det mangler 10 kontrakter, skriver man følgende i notatet: 10

NA
5.5.5Sjekkliste for balanseansvarlige

8/2

14/2 for utbedring av avvik

Balanseansvarlige

Sjekkliste som brukes internt av balanseansvarlige for å kvalitetssikre data gjennom Elhub Web Portal og BRSeretter endt initiell migrering.

Sjekkliste for balanseansvarlig


NA
5.5.6Sjekkliste for tredjepart

8/2

14/2 for utbedring av avvik

Tredjeparter

Sjekkliste som brukes internt av tredjepart for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for tredjepart


NA
5.5.7Logical Backup EnergyIP DB

8/2

8/2

Accenture

We make a logical DB backup after all corrections to the structure data are done.

TA - ExArch - Infra - Exadata

EnergyIp DB Backup

NA

Will be done 8.2.2019 19:00 CET as part of preparations for 9.2 deploy.

5.5.8

Sjekkpunkt

Datakvalitet i Elhub OK, avvik er håndterbart -> Point of no return

9/2

8:00

N/A
ElhubAktører har gått gjennom oppstartssjekklister og rapportert avvik. Antall feilede målepunkter eller manglende målepunkter skal være under et håndterbart antall.
  •  
5.5.9Sjekkliste for nettselskaper

8/2

14/2

Netteiere

Utbedring av avvik

Sjekkliste for nettselskaper

Gjennomføring av dette steget skal rapporteres i Edielportalen for nettselskaper.


NA
5.5.10Sjekkliste for kraftleverandører og regulert kraftleverandør

8/2

14/2

Kraftleverandører og regulerte kraftleverandører


Utbedring av avvik

Sjekkliste for kraftleverandører

Gjennomføring av dette steget skal rapporteres i Edielportalen for kraftleverandører, ikke regulerte.


NA