05 - Final migration

Sist oppdatert 2019-02-25 10:50

Edielportalen:

  • Go Live 05 <Rolle> Endelig migrering                      

Execution period

Elhub Go Live: The activities in this section are executed in the period 1/2 to 9/2.

Summary of activities

The activities described in this chapter include deleting data in EIP and EDM, receiving final round of migration data from the market, to process this data, and to eventually load it into Elhub. This chapter also includes checklists for migrated data to be performed by market parties.


Content overview

Final migration contains the following main parts:

  • 5.0 Making EIP ready for final migration
  • 5.1 Making DAM ready for final migration 
  • 5.2 Receive and validate master data and metering values
  • 5.3 Stage and load master data and metering values
  • 5.4 Set metering point range
  • 5.5 Market parties QA in Elhub


Viktig informasjon til alle markedsaktører


Viktig informasjon 1: Hovedprinsippene for bufring av forretningsprosesser:


  • Leverandørbytteprosesser bufres fra og med 24. januar. Deretter starter man å bufre andre prosesser suksessivt i henhold til CRB.
  • Fra og med fredag klokka 00:00 skal alle nye prosesser som oppstår ikke lagres i KIS, men bufres utenfor KIS.
  • 1. februar skal man sikre at alle meldinger som er kommet på Edifact er behandlet.
  • Når punktet over er gjort kan man gjøre migreringsuttrekket fra databasen sin. Det skal gjøres en gang i løpet av fredagen, eller senest søndag 3/2 klokka 14:00 
  • Fra og med bufring starter og inntil man har gjort migreringsuttrekket på fredag kan prosesser ikke bufres i KIS. Da blir de i tilfellet både migrert og debufret, noe som vil følge til at meldingen vil avvises når den spilles av til Elhub og aldri sendt til mottakeren. Dette vil skape ekstraarbeid for alle parter. Viktig at alle markedsaktører bekrefter at de har en vanntett prosedyre som sikrer at ingen prosesser blir både migrert og bufret.
  • Opplasting av migreringsuttrekket til Elhub kan gjøres fra søndag 3/2 klokka 14:00, og alle aktørene må ha gjort det innen mandag klokka 14:00.


 Viktig informasjon 2: Hvordan gjøre endringer eller tillegg til de migrerte dataene


Dette kan gjøres på to måter. 

  1. Opprette manglende målepunkter/kontrakter eller målepunkter som har blitt stoppet i migrering
    1. Bruk prosedyren i 6.1.1 Last opp Excel-filer for manglende målepunkter
  2. Endring på målepunkter som har blitt migrert
    1. Dette skal i utgangspunktet gjøres via BRSer. Dersom man står fast, send henvendelse til post@elhub.no


Commissioning steps

Task ID Commissioning step Start time Deadline Execution Organization Step description Done
5.0 Making EIP ready for final migration
!!Elhub Web Portal, EMIF and Web Plugin made UNavailable to market!!
5.0.0 Stop doing daily verifications in the Web Portal 2/2
Statnett Not necessary and not possible as the Web Portal will be down.
  •  
N/A Start migration mode

1/2

16:00

1/2

17:00

Accenture

Start migration mode

Set soft blackout before starting any activities

  •  
5.0.1.ohs Turn off OHS to stop all web traffic toElhub 1/2 1/2 Accenture

Turn off OHS to stop all web traffic to Elhub

Turn off OHS to stop all web traffic to Elhub

  •  
5.0.1.fw Turn off WebPlugin access for the market parties 1/2 1/2 Accenture

Turn OFF WebPlugin access for the market parties

Turn ON/OFF WebPlugin access for the market parties

  •  
5.0.1.oag

Turn off OAG to stop all web service traffic toElhub

1/2 1/2 Accenture

Turn off OAG to stop all web service traffic to Elhub

Turn off OAG (EMIF) to stop all web service traffic to Elhub


  •  
5.0.1

Put EIP in migration mode

1/2

1/2

Accenture

Put EIP in migration mode

How to switch modes in EIP using ConfigurationManagementTool.sh - Prepare Migration Mode

  •  
5.0.2 Logical Backup EnergyIP DB

1/2

1/2

Accenture

Take a logical backup before wiping EIP

TA - ExArch - Infra - Exadata

CRB BACKUP USING LOGICAL BACKUP METHOD

  •  
5.0.3.gg Pause GG synchronization 1/2 1/2 Accenture

Pause GG synchronization

Goldengate Start and Stop steps .

  •  
5.0.3.odi Turn off scheduled procedures ODI

1/2

1/2

 Accenture

Turn off scheduled procedures ODI

Turn off/on scheduled procedures ODI

  •  
5.0.3 Delete master data from Energy IP

1/2

2/2


Accenture

Run the final wipe of EnergyIP. Note that this is not the same as the initial wipe and will leave some of the data in the system. Ensure backup step 5.0.2 is completed.

Secondary delete of data from EnergyIP

  •  
5.0.3.1 Refresh MTM MVs 2/2 2/2 Accenture

This is a dependency for PopulateGridAreaStatusTable, and has to be done after grid area data has been populated.

Refresh MTM MVs

  •  
5.0.4 Ensure DB health of EIP

2/2

2/2

Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.0.6 Delete all messages on OSB queues. 2/2
2/2 Accenture

Delete all messages on OSB queues.

Delete all messages on OSB/SOA/BPM queues.

  •  
5.0.7 Delete all messages on SOA/BPM queues. 2/2
2/2 Accenture

Delete all messages on SOA/BPM queues.

Delete all messages on OSB/SOA/BPM queues.

  •  
5.0.8 Terminate any running processes in SOA/BPM. 2/2

2/2

Accenture

Terminate any running processes in SOA/BPM.

Terminate any running processes in SOA/BPM

  •  
5.0.5 Synchronize tables after source systems Truncate 2/2 2/2 Accenture

Synchronize tables after source systems Truncate (GoldenGate)

GoldenGate Configuration Steps (initial load)

  •  
5.0.9

Turn on OAG to start all web service traffic

(BRS's will be off so this is for migration only)

2/2 2/2 Accenture

Turn on both OAG instances.

Turn on OAG (EMIF) to start all web service traffic to Elhub

  •  
5.0.10 Open DAM for file upload 2/2

2/2

23:59

Statnett

Configurationparametersneedstobe set

  •  
5.0.11 Load sanity files 2/2

2/2

23:59

Statnett Load sanity files in EDM
  •  
5.0.12 Notify certain market parties 2/2

2/2

23:59

Statnett
  •  
5.1 Making DAM ready final migration
5.1.2 Ensurepartitonson EDM tables

31/1

08:00 CET

31/1 Accenture

Ensure that partitioned tables have the correct partitions.

Partitioning metering value tables

  •  
5.1.3 Delete old migration files

31/1

31/1

Accenture

Remove old migration files

Delete EDM migration files from elhwls servers

  •  
5.1.3.a Clean the quarantine folder in DMZ 31/1 31/1 Accenture

Manually clean the quarantine folder in DMZ

Clean the quarantine folder in DMZ

  •  
5.1.6 Logical Backup DAM DB

31/1

31/1

12:00 CET

Accenture

Logical Backup before westartmigration. Will include updated config and/or packages/code.

DAM DB Backup  

  •  
5.1.4 Reminder to market parties about migration

31/1

31/1

Statnett

Send reminder on email to market parties about upcoming migration.

Send notification to market parties that does migration

  •  
5.1.1 Delete master data from EDM database

31/1

31/1


 Statnett

Remove master data from EDM database

Delete master data from EDM database


  •  
5.1.5 Verify structure data in DAM landing

31/1

31/1

Statnett

We rerun the step from initial migration just to make sure that nothing has happened.

Validating landing structure data"

  •  
5.2 Receive and validate master data and metering values
5.2.1 Ensure indexes for receiving files

31/1

31/1

Statnett

Ensure indexes for phase UPLOAD

Create and drop indexes on EDM tables

  •  
5.2.2 Verify database health for EDM database

31/1

31/1

Statnett

We need to ensure that we have no known issues in the database.

EDM database health check

  •  
5.2.3 Verify functional configuration in EDM

31/1

31/1

Statnett

Before we start uploading files, we need to ensure that all functional parameters are set correctly. Verify:

  • NBS_BRP_FROM_DATE
  • NOW_DATE
  • EDIEL_VAL_ST_DATE

Verify functional configuration in EDM

  •  
5.2.3.9 Load sanity files 31/1 31/1 Statnett Load the sanity point files and ensure all is ok.
  •  
5.2.4 Open for upload of files to EDM

31/1

31/1

23:59


Statnett

Configurationparametersneedstobe set

  •  
5.2.4.1 Notify market that DAM is ready 31/1

31/1

23:59

Statnett

SendemailtoDAM users that DAM is open.

Send notification to market parties that does migration

  •  
5.2.5 Laste opp migreringsfiler til DAM

1/2

00:00


4/2

14:00


Nettselskaper og kraftleverandører

Data om målepunkt, kunder og kontrakter, samt 1 uke måleverdier for timesavregnede (21/1 – 27/1). I tillegg minimum siste periodevolum med tilhørende start- og sluttstand, på relevante (profilavregnede) målepunkter.

DAM vil være åpent i periodene:

• Fredag 1/2 00:00 til 16:00
• Søndag 3/2 00:00 til mandag 4/2 14:00

Gjennomføring av dette steget skal rapporteres i Edielportalen.

NA
5.2.6

Receive master data and metering values


1/2

00:00


4/2

14:00


Statnett

Includes:

  • accounting points with contracts and customers
  • exchange point
  • metering values
    • special series
    • limitedamoutofprofiled reads (for EAC)

Format and integrity check

  •  
5.2.7

Sjekkpunkt

Migrering er komplett fra alle markedsaktører

5/2

12:00

N/A Elhub

Alle nettselskaper skal ha lastet opp sine migreringsfiler.


  •  
5.2.8

Rapporter til NBS

4.2

00:00

6.2

23:59

Nettselskaper

Måleverdier for 28.1. - 3.2. blir rapportert til NBS.

Gjennomføring av dette steget skal rapporteres i Edielportalen.

N/A
5.2.8.5 Create and upload connection points file 9/1 13/1 Statnett

Connection pointsarenotrecievedfrommarketparties,andneedsto be uploaded by DAM Operators. It is also used as a sanity test of file upload.

Upload connection points

  •  
5.2.9

Block upload of files to EDM

5/2

7/2

Statnett

Configurationparametersneedstobe set to -1

  •  
5.2.11 Ensure indexes for timeline check

5/2

7/2

Statnett

Ensure indexes for phase TIMELINE

Create and drop indexes on EDM tables

  •  
5.2.13 Run timeline check

5/2

7/2

Statnett

Timeline check replaces old data and updates data to set them as current (eligible for staging and consistency)

How to run timeline check

  •  
5.2.14 Ensure indexes for next phase

5/2

7/2

Statnett

Ensure indexes for phase CONSISTENCY

Create and drop indexes on EDM tables

  •  
5.2.16 Read Grid Owner reports and upload in DAM 5/2 13/1 Statnett

Read grid owner reports, create output and load into DAM.

Process EDM grid owner reports

  •  
5.3 Stage and load master data and metering values
5.3.1 Run staging integrity

5/2

7/2

Statnett

Staging integrity is used as a last step to exclude (remove) data from staging.

How to run staging integrity check

  •  
5.3.2 Stage master data, including post checks

5/2

7/2

Statnett

Stage master data.

How to stage master data

  •  
5.3.3 Run go and bs reports 5/2 7/2 Statnett Used in quality assurance
  •  
5.3.17 Check completeness and quality of migration

5/2

7/2

Statnett

We need to check the completeness and quality before going ahead.

Check completeness and quality of data in EDM

Do in parallel with backup.

  •  
5.3.6 Logical Backup DAM DB

5/2

5/2

Accenture

Backup staging result (*_RESULT) tables,stagingmetatables, and calculated_metering_values. Take logical backup if loading completely fails.

DAM DB Backup 

This step can be skipped in case of time-constraint, but gained time is limited by previous step being run in parallel.

  •  
5.3.7 Ensure DB health of EIP

5/2

7/2

Statnett

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.8 Update Migration_Sequence in DAM.

5/2

7/2

Statnett

Update Migration_Sequence table's Last_Executed_primary column to NULL in EDM where Relevance_Inclusion_Group_Id is equal or greater than 500


See exception handling chapter in


Transfer data from DAM to EIP


  •  
5.3.9.prep Monitor masterdata load 5/2 7/2

Accenture

Statnett

Eyes on glass monitoring and take remediating actions to keep load speed above threshold (TBD).

Key resources:

  • EIP (Radomir)
  • DBA
  • ODI (Ashish?)

Monitor master data load

  •  
5.3.9 LoadmasterdatatoElhub

5/2

7/2

Statnett

This follows the same process as for full load.

Transfer data from DAM to EIP

  •  
5.3.10 Load birth numbers andorganizationnumbertsto EIP

5/2

7/2

Statnett

Load data into Identity and User_Mapping using ODI.

Data is transferred using DB Insert method.

How to load masked identification numbers from EDM to EIP

  •  
5.3.11 Load ADU

5/2

7/2

Statnett

Average Daily Usage is loaded from the DAM staging table to EnergyIP.

Firstpartof this step is to retrieve the internal SP IDs of Metering Points from EIP to DAM

Load ADU (Average Daily Usage)

  •  
5.3.12 Ensure DB health of EIP

5/2

7/2

Statnett

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.14

Create virtual channels on connection points in EIP

5/2

7/2

Statnett

Create virtual channels on connection points in EIP

Create virtual channels on connection points in EIP

  •  
5.3.15 Ensure DB health of EIP

5/2

7/2

Statnett

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.15.gg Synchronize tables after source systems Truncate

Accenture

Synchronize tables after source systems Truncate (GoldenGate)

GoldenGate Configuration Steps (initial load)

DO NOT execute unless explicitly instructed.

5.3.16 Run completeness reports for exchange and accounting points, and check results.

5/2

7/2

Statnett

Reports to run:

  • ExchangePointIntegrityCheck
  • ExchangePointCount
  • AccountingPointDataIntegrityCheck
  • AccountingPointDataCount
  • ContractIntegrityCheck
  • ContractCount
  • EBOKeyCheck

How to run a completeness report or a load script

  •  
5.3.17 Check entity counts between EDM and EIP

5/2

7/2

Statnett

Checklandingvs staging vs EIP for master data

Check counts between EDM and EIP

  •  
5.3.18 Logical Backup EnergyIP DB

6/2

7/2

Accenture

Take logical database backup after loading latest master data.

TA - ExArch - Infra - Exadata

CRB BACKUP USING LOGICAL BACKUP METHOD

This step can be skipped in case of time-constraint.

  •  
5.3.20 Set x_process_milestone 6/2


7/2

Accenture

Set x_process_milestone to avoid migrated data triggering creation of these messages.

Set X_PROCESS_MILESTONE

  •  
5.3.21 Put EIP in business as usual mode 6/2


7/2

Accenture

Switch to Operations Mode

How to switch modes in EIP using ConfigurationManagementTool.sh - Prepare Operations Mode

  •  
5.3.22 Ensure DB health of EIP 6/2


7/2

Accenture

Identify and remedy issues with

  • Invalid indexes
  • Stale stats

Ensure DB health in EIP

  •  
5.3.22.b Allocate CPU for operations 6/2


7/2

Accenture

Allocate CPU for operations

CPU allocation for operations

  •  
5.3.24 Synchronize master data to OBIEE (ODI data load) 6/2


7/2

Accenture

Synchronize master data to OBIEE (ODI data load)

Use snapshot date interval from 01.02.2019 to sysdate

Run full load described in section "Steps to run FULL Load" in How to set up ODI Scheduling

  •  
5.3.25 Execute EIDG load 6/2


7/2

Accenture

Execute EIDG load (cache metering points) toensureperformanceofMTM jobs and other jobs operating with large sets of data.

Load EIDG

  •  
5.3.22.c

Turn on OHS to start web traffic to Portal and Plugin


6/2
7/2
Accenture

Turn on the portal,pluginandloginOHS instances.

Turn on OHS to start web traffic to Portal and Plugin

Note: WebPlugin Access is turned off for Market Parties at this point.

  •  
N/A End migration mode

Accenture

End migration mode

Lift soft blackout whenportalis available

  •  
5.3.23 Turn on ReadOnly BRSs

6/2

7/2

Accenture

Turn on access to BRSs thatdonot change the datainElhub. This allows the market parties to QA their datathroughBRSs,andQA their CIS systems at the same time.

List of BRSs to turn on:

  • BRS-NO-303 Request Master Data
  • BRS-NO-315 Request Metering Values
  • BRS-NO-324 Request Imbalance Settlement

Turn on Read Only BRSs

Open/Close BRS for traffic

  •  
5.3.26
Open Access to Web Plugin for market parties 6/2 7/2 Accenture

Turn ON WebPlugin access for the market parties

Turn ON/OFF WebPlugin access for the market parties

  •  


5.3.27 Set metering point range for internal market parties 6/2


7/2

Accenture

Set metering point range for internal market parties

Set metering point range for internal metering points in Elhub

  •  
5.4 Set metering point range and channel milestone
This section is done in parallel with Market Parties QA
5.3.13 Load Channel Milestones

5/2

7/2

Statnett

Report to run:

  • ChannelMilestoneLoad

Parameters:

  • ChannelType - Interval
  • OverwriteExisting - true

Channel milestone for non profiled metering points is set to Sunday February 10 00:00 so that validation for metering value continuency and reminders will work in the start up week.

creport - How to run a completeness report or a load script

  •  
5.4.2 Start doing daily verification in Elhub Web Portal 8/2
Statnett

Should run daily in the morning until final migration phase.

Sanitysjekk av Elhub Aktørportal

  •  
5.X Consistency Check and Dataquality assessment in DAM
This section should run in parallel with Market Parties QA and can be started as soon asEIDGand OBIEE load is done
5.X.1 Run consistency checks

7/2

7/2

Statnett

Run consistency on the data,includingmetalistandcore process.

This step can be skipped in case of time constraint.

How to run consistency check

  •  
5.X.2 Rundataqualityreports and publish downloadable reports in DAM and elhub.no

7/2

7/2

Statnett

The downloadable reports include an overview over failed metering points, as well as metering points with insufficient metering values. Verify that all special series have been received from grid owners.

Create EDM data-quality reports

  •  
5.X.3 Notify market parties that NCFs are ready for download

7/2

7/2

Statnett

Notify market parties by email when all checks have run, and reports have been published in DAM.

Send notification to market parties that does migration

  •  
5.X.4 Check completeness and quality of migration

7/2

7/2

Statnett

We need to check the completeness and quality before going ahead.

Check completeness and quality of data in EDM

  •  
5.X.5 Report migration statistics internally 7/2 7/2 Statnett Report migration statistics internally
  •  
5.X.6 Logical Backup DAM DB

8/2

8/2

Accenture

Take Logical database backup. Backup-migration is done.

DAM DB Backup

This step can be skipped in case of time-constraint.

  •  
5.5 Market parties QA in Elhub
!!Elhub Web Portal, EMIF and Web Plugin made available to market!!
Steps 5.5.1 to 5.5.6 (check list activities) can be run in parallell.
5.5.1 Sjekkliste for kvalitetssikring av migrerte data

8/2

8/2

16:00

Statnett

Sjekkliste som brukes internt av Elhub operatører for å kvalitetssikre data gjennom Elhub Web Portal etter endt initiell migrering og frem til Go Live.

Sjekkliste for kvalitetssikring av Elhub-operatører

  •  
5.5.2 Sjekkliste for nettselskaper

8/2

8/2

16:00 for rapportering av avvik


Netteiere

Sjekkliste som brukes internt av nettselskaper for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for nettselskaper

Gjennomføring av dette steget skal rapporteres i Edielportalen for nettselskaper.

Når man selvregistrerer at dette steget er utført i Edielportalen, skal man i Notat-feltet notere hvor mange målepunkt man mangler i Elhub.

For eksempel dersom det mangler 10 målepunkter, skriver man følgende i notatet: 10

NA
5.5.3 Sjekkliste for tjenestetilbyder

8/2

14/2 for utbedring av avvik

Tjenestetilbyder

Sjekkliste som brukes internt av tjenestetilbyder for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for tjenestetilbyder


NA
5.5.4 Sjekkliste for kraftleverandører og regulert kraftleverandør

8/2

8/2

16:00 for rapportering av avvik


Kraftleverandører og regulerte kraftleverandører

Sjekkliste som brukes internt av kraftleverandører for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for kraftleverandører

Gjennomføring av dette steget skal rapporteres i Edielportalen for kraftleverandører, ikke regulerte.

Når man selvregistrerer at dette steget er utført i Edielportalen, skal man i Notat-feltet notere hvor mange kontrakter man mangler i Elhub.

For eksempel dersom det mangler 10 kontrakter, skriver man følgende i notatet: 10

NA
5.5.5 Sjekkliste for balanseansvarlige

8/2

14/2 for utbedring av avvik

Balanseansvarlige

Sjekkliste som brukes internt av balanseansvarlige for å kvalitetssikre data gjennom Elhub Web Portal og BRSeretter endt initiell migrering.

Sjekkliste for balanseansvarlig


NA
5.5.6 Sjekkliste for tredjepart

8/2

14/2 for utbedring av avvik

Tredjeparter

Sjekkliste som brukes internt av tredjepart for å kvalitetssikre data gjennom Elhub Web Portal og BRSer etter endt initiell migrering.

Sjekkliste for tredjepart


NA
5.5.7 Logical Backup EnergyIP DB

8/2

8/2

Accenture

We make a logical DB backup after all corrections to the structure data are done.

TA - ExArch - Infra - Exadata

EnergyIp DB Backup

NA

Will be done 8.2.2019 19:00 CET as part of preparations for 9.2 deploy.

5.5.8

Sjekkpunkt

Datakvalitet i Elhub OK, avvik er håndterbart -> Point of no return

9/2

8:00

N/A
Elhub Aktører har gått gjennom oppstartssjekklister og rapportert avvik. Antall feilede målepunkter eller manglende målepunkter skal være under et håndterbart antall.
  •  
5.5.9 Sjekkliste for nettselskaper

8/2

14/2

Netteiere

Utbedring av avvik

Sjekkliste for nettselskaper

Gjennomføring av dette steget skal rapporteres i Edielportalen for nettselskaper.


NA
5.5.10 Sjekkliste for kraftleverandører og regulert kraftleverandør

8/2

14/2

Kraftleverandører og regulerte kraftleverandører


Utbedring av avvik

Sjekkliste for kraftleverandører

Gjennomføring av dette steget skal rapporteres i Edielportalen for kraftleverandører, ikke regulerte.


NA