You'll even like your Data Guard more with Flashback

Similar documents
Oracle Database 11g Data Guard

Oracle 1Z0-053 Exam Questions & Answers

INTRODUCTION EXTENDED FLASHBACK FUNCTIONS

Question No: 1 Which two statements are true for Data Guard environments with multi-tenant databases?

Flashback. A quest for zero data loss. Tommie Grove. MyDBA. January P a g e F l a s h b a c k A q u e s t f o r z e r o d a t a l o s s

Oracle Database 12c: Data Guard Administration

Database Rolling Upgrade with Transient Logical Standby Database DOAG Day High Availability Robert Bialek Principal Consultant

Oracle Database 12c: Dataguard Administration

CO Oracle Database 12c: Data Guard Administration

CO Oracle Database 11g: Data Guard Administration

Oracle 12c Dataguard Administration (32 Hours)

Oracle Database 12c: Data Guard Administration LVC

Recovery without Backup. All Data Lost?

ORACLE 11gR2 DBA. by Mr. Akal Singh ( Oracle Certified Master ) COURSE CONTENT. INTRODUCTION to ORACLE

Oracle10g Data Guard: Back to the Future

Oracle 11g Data Guard Manual Failover Steps

Oracle Database 11g for Experienced 9i Database Administrators

Maximum Availability Architecture. Oracle Best Practices for High Availability. Reducing Siebel Downtime with a Local Standby Database

Oracle Database 11g: SQL Fundamentals I

Data Guard: Setup and Administration. By Ashok Kapur Hawkeye Technology, Inc.

White Paper. Increase ROI on Disaster Recovery Infrastructure. Implementing Snapshot Standby on Oracle 11g and 10g. Abstract

ORACLE DATA GUARD REAL WORLD EXAMPLES AND TROUBLESHOOTING

UNIVERSITY AUTHORISED EDUCATION PARTNER (WDP)

ORACLE 12C - M-IV - DBA - ADMINISTRADOR DE BANCO DE DADOS II

Oracle DBA Course Content

Oracle Data Guard Tips and Tricks Direct from Oracle Development

1Z Oracle Database 12c - Data Guard Administration Exam Summary Syllabus Questions

Explore the Oracle 10g database architecture. Install software with the Oracle Universal Installer (OUI)

Oracle Database 11g: Administration Workshop I DBA Release 2

Oracle Database 10g Migration to Automatic Storage Management. An Oracle White Paper August 2005

Configuring High Availability for the PMG DB

Oracle Database Service High Availability with Data Guard?

for Backup & Recovery & Failover

DataGuard in Practice

What was new in 11g for Backup and Recovery? Contents

Oracle Database 11g: Data Guard Administration

Oracle DataGuard 10gR2

Oracle Database 12c: Backup and Recovery Workshop Ed 2 NEW

Exam : Oracle 1Z0 043

Using Recovery Manager with Oracle Data Guard in Oracle9i. An Oracle White Paper March 2004

Oracle DBA workshop I

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Oracle Database 11g Administration Workshop II

Programa de Actualización Profesional ACTI Oracle Database 11g: Administration Workshop II

ORACLE DBA TRAINING IN BANGALORE

How To Apply Archive Logs Manually In Standby Database Using Rman

Oracle Data Guard in Oracle Database 10g Release 2 Business Continuity for the Enterprise. An Oracle White Paper November 2006

Oracle Database Appliance: Implementing Disaster Recovery Solutions Using Oracle Data Guard

LOSS OF FULL DATABASE AND DATABASE RECOVERY ORACLE 11g

Actual4Test. Actual4test - actual test exam dumps-pass for IT exams

Oracle Active Data Guard - Overview

OCP Oracle Database 12c: Advanced Administration Exam Guide

An Oracle White Paper September Methods for Upgrading to Oracle Database 11g Release 2

Oracle - Oracle Database 12c: Backup and Recovery Workshop Ed 2

Converting to Transparent Data Encryption with Oracle Data Guard using Fast Offline Conversion Oracle Database 12.1 and Oracle Database 11.

An Oracle White Paper April Deploying Oracle Data Guard with Oracle Database Appliance

Oracle 1Z Upgrade Oracle9i/10g OCA to Oracle Database 11g OCP. Download Full Version :

Data Guard Deep Dive

Oracle Database 12c R2: Backup and Recovery Workshop Ed 3

Oracle12c Release 1 New Features for Administrators (5 Days)

12.1 Multitenancy in real life

<Insert Picture Here> Maximum Availability Architecture (MAA) Best Practices: Online Patching and Rolling Upgrades with Oracle Database

Disaster Recovery: Restore Database from One Server to another Server when Different Location

Oracle Database 12c R2: Backup and Recovery Workshop Ed 3

Mike Hughes Allstate Oracle Tech Lead, Oracle Performance DBA

Steps Create Physical Standby Database Oracle 9i

Enterprise Manager: Scalable Oracle Management

Oracle Database 12c R2: Backup and Recovery Workshop Ed 3

Oracle RMAN for Absolute Beginners

How to Recover the lost current control file, or the current control file is inconsistent with files that you need to recover??

Disaster Recovery Strategies for RAC on Oracle SE Arjen Visser Dbvisit Software Limited

Maximum Availability Architecture. Oracle Best Practices For High Availability

ORANET- Course Contents

Oracle Exam 11gocmu Oracle Database 11g Certified Master Upgrade Exam Version: 4.0 [ Total Questions: 671 ]

Oracle Database 12c R2: Backup and Recovery Workshop Ed 3

Online Operations in Oracle 12.2

ORACLE DATABASE: ADMINISTRATION WORKSHOP II

ORACLE 10g/9i DATA GUARD LOGICAL STANDBY DATABASE

Oracle 1Z Oracle Database 10g: Administration II. Download Full Version :

1Z Upgrade Oracle9i/10g to Oracle Database 11g OCP Exam Summary Syllabus Questions

1z z0-060 Upgrade to Oracle Database 12c

ORACLE DATABASE: ADMINISTRATION WORKSHOP II

Actual4Test. Actual4test - actual test exam dumps-pass for IT exams

Projects. Corporate Trainer s Profile. CMM (Capability Maturity Model) level Project Standard:- TECHNOLOGIES

Steps how to duplicate a database to a new machine. Version 10gR2

Question No : 1 Which three statements are true regarding the use of the Database Migration Assistant for Unicode (DMU)?

Question: 1 For which two SQL statements can you use the Flashback Table feature to revert a table to its previous state? (Choose two.

Oracle 1Z Oracle Database 12c: Data Guard Administration. Download Full version :

Oracle Database 12c: OCM Exam Preparation Workshop Ed 1

FILE / RMAN BACKUP PRODUCTS MANUAL EBOOK

Oracle Database 10g: New Features for Administrators Release 2

Installing the Oracle Database Softwar

1. Name of Course: Oracle Database 12c: Backup and Recovery Workshop

ALTER DATABASE RECOVER TO LOGICAL STANDBY KEEP IDENTITY;

DOC ORACLE STANDBY USER MANUAL ARCHIVE

Oracle Database 11g: Administration Workshop II Release 2

Insanity: doing the same thing over and over again and expecting different results. Nice Proverb

Recovering Oracle Databases

Oracle - Oracle Database 11g: Backup and Recovery Workshop (Training On Demand)

Exam : 1Z Title : Oracle Database 10g: Administration II. Ver :

Transcription:

You'll even like your Data Guard more with Flashback Hervé Schweitzer Mathias Zarick München, 26.01.2010 Baden Basel Bern Brugg Lausanne Zürich Düsseldorf Frankfurt/M. Freiburg i. Br. Hamburg München Stuttgart Wien

Agenda History of Standby Databases Flashback Database Flashback enhances Data Guard Environment Data are always part of the game. Rolling Upgrade with Transient Logical Standby Database 2

Standby Databases: A short history Oracle 7.3: Creating and mounting a standby database Oracle 8i: Automated archived redo log transport and application, TAF, open read-only of standby database Oracle 9i: Data Guard and Data Guard Broker with switchover, close log gap, delayed redo application, GUI and no-data-loss setups (sync transport) Oracle 10g: Simplified syntax, RAC support, partial failover cluster support, reuse of old primary as new standby database, automatic standby activation Oracle 11g: Active Data Guard (readonly open and log-apply), Lost-write detection, snapshot standby, automatic standby activation in MaxPerformance mode 3

11gR2 Data Guard New Features Automatic Repair of corrupted blocks with the help of a physical standby database or in a physical standby database Redo transport compression now also supports SYNC setups Configurable Real-Time Query Apply Lag Limit per session: STANDBY_MAX_DATA_DELAY = NONE n Flushing any unsent redo to standby in mount state SQL> ALTER SYSTEM FLUSH REDO TO 'PHYS_SITE1'; System altered. LogMiner (Logical Standby) supports table compression (OLTP and BASIC) and SecureFile LOBs Up to 30 standby databases 4

Agenda History of Standby Databases Flashback Database Flashback enhances Data Guard Environment Comprehensive consultancy is the through ball to your performance. Rolling Upgrade with Transient Logical Standby Database 5

Flashback in Oracle Database Three completely independent, different mechanisms exist Only Flashback database is especially relevant in a Data Guard environment Mechanism Undo segments Rename table statements Since Oracle 10g: flashback logs Flashback Type DBMS_FLASHBACK Flashback query Flashback table Recycle bin Flashback table x to before drop Flashback database 6

With Flashback you can Rewind your Database!"#!"# $ 7

Concept of Flashback Database New type of log files: Flashback Logs Contain before images of data blocks New background process RVWR Writes Flashback data (asynchronous to checkpoints, DBWR, etc.) (' (' (' (' $' $' $' $' %&' %&' %&' %&' Rewind resp. Rollback Logical data corruptions User errors Depending on DB size, significantly faster than conventional recovery Restore is not required!!"#* )* It often eliminates the need for delayed redo application, but it consumes more resources (sometimes not acceptable) Caution: Flashback is not guaranteed, DelayMins is guaranteed! 8

The Flash Recovery Area Automatically created directory structure /u02/flash_recovery_area PHYS_SITE1 +!"# archivelog autobackup backupset controlfile datafile o1_mf_07bv7sbk_.flb o1_mf_078ez34h_.flb... 2009-03-24 o1_mf_1_5839_00q8o099_.arc o1_mf_1_5840_00qbojfg_.arc... 2009-03-24 o1_mf_s_515845766_00q8vspj_.bkp o1_mf_s_515855778_00qlnpn9_.bkp 2009-03-24 o1_mf_annnn_tag20090324t100512_00q7zc9j_.bkp o1_mf_ncnnf_tag20090324t102920_00q8vmff_.bkp... o1_mf_tag20090324t095756_00q72lc2_.ctl o1_mf_sysaux_00q70otw_.dbf o1_mf_system_00q72yyp_.dbf o1_mf_undotbs_00q71o05_.dbf... 9

Configuration of Flashback Database must be running in archive log mode 1. Configuration of Flashback Recovery Area DB_RECOVERY_FILE_DEST_SIZE SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=2048M; DB_RECOVERY_FILE_DEST SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='/u02/flashback_recovery_area'; 2. Activating Flashback (database must be mounted, not open) SQL> ALTER DATABASE FLASHBACK ON; 3. Define retention target (time in minutes) SQL> ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=1440; (default) 10

Dictionary Views relevant to Flashback Database V$DATABASE SQL> select current_scn, flashback_on from v$database; CURRENT_SCN FLASHBACK_ON ----------- ------------------ 6368415 YES V$FLASHBACK_DATABASE_LOG Important for size estimation SQL> select oldest_flashback_scn old_flash_scn, to_char(oldest_flashback_time,'dd-mon-yyyy HH24:MI:SS') old_flash_time, retention_target ret, flashback_size flash_size, estimated_flashback_size est_flash_size from v$flashback_database_log OLD_FLASH_SCN OLD_FLASH_TIME RET FLASH_SIZE EST_FLASH_SIZE ------------- -------------------- ---------- ---------- -------------- 6310610 15-JUL-2009 03:20:08 1440 155770880 122044416 11

I/O Monitoring V$FLASHBACK_DATABASE_STAT SQL> select to_char(begin_time,'dd-mon-yyyy HH24:MI:SS') begin_time, to_char(end_time,'dd-mon-yyyy HH24:MI:SS') end_time, flashback_data flash_data, db_data, redo_data, estimated_flashback_size est_flash_size from v$flashback_database_stat BEGIN_TIME END_TIME FLASH_DATA DB_DATA REDO_DATA EST_FLASH_SIZE -------------------- -------------------- ---------- ---------- ---------- -------------- 16-JUL-2009 10:50:38 16-JUL-2009 10:52:42 221184 0 15872 0 16-JUL-2009 09:50:37 16-JUL-2009 10:50:38 3235840 4136960 866304 122044416 16-JUL-2009 08:50:36 16-JUL-2009 09:50:37 2539520 2899968 670720 122511360 16-JUL-2009 07:50:35 16-JUL-2009 08:50:36 2744320 3407872 748544 123199488... 12

Rewind the Database: Usage of Flashback SQL*PLUS and RMAN Commands and functionality are identical But: Better perform open resetlogs only in RMAN RMAN is recommended RMAN knows all about backups (e.g. Archive Log) that could be required for a successful Flashback operation rman target / RMAN> SHUTDOWN IMMEDIATE; RMAN> STARTUP MOUNT; RMAN> FLASHBACK DATABASE TO [SCN SEQUENCE TIME RESTORE_POINT ]; RMAN> SQL "ALTER DATABASE OPEN READ ONLY"; rman target / RMAN> SHUTDOWN IMMEDIATE; RMAN> STARTUP MOUNT; RMAN> FLASHBACK DATABASE TO [SCN SEQUENCE TIME RESTORE_POINT ]; RMAN> ALTER DATABASE OPEN RESETLOGS; 13

Agenda History of Standby Databases Flashback Database Flashback enhances Data Guard Environment Decisions have effects. Jump. Rolling Upgrade with Transient Logical Standby Database 14

Flashback and Data Guard: Learning by examples Did you ever truncate the wrong table? Did you ever need to recreate a standby (2.5 TByte ) after a primary database PITR? Did you ever need to recreate a standby after activating one? Did you ever need to clone a database for a little application or Oracle patch test? 15

Recreation of data with Flashback of Standby (1) SQL> truncate table,- prior to 10g Tablespace PITR Database PITR Activate standby database or open read-only/exp/imp (if delay) as of 10g 1. Flashback standby database (if no delay in redo apply) 2. Open read-only 3. Export table 4. Import on primary database 16

Recreation of data with Flashback of Standby (2) &01,- 2$1 31 ' %" &01,- &01,- /"), -,"), " 31 ' &#4!5,/ )4)** )-" +!"#,).,"), 2$1 31 ' %" 17

Recreation of data with Flashback of Standby (3) PRIMARY STANDBY TRUNCATE TABLE EMP; DISABLE DATABASE 'PHYS_SITE2'; SHUTDOWN IMMEDIATE STARTUP MOUNT FLASHBACK STANDBY DATABASE TO SCN scn; ALTER DATABASE OPEN READ ONLY; exp scott/blue tables=emp \ file=exp_from_standby.dmp SHUTDOWN IMMEDIATE STARTUP MOUNT imp scott/blue \ file=exp_from_standby.dmp ENABLE DATABASE 'PHYS_SITE2'; &#4!5,/ )4)** 18

PITR of Primary without Recreating Standby (1) 7 *) )) 82 6,- prior to 10g PITR ( open resetlogs) either recreate the standby database with server managed backup (rman) or user managed backup (cp/copy) as of 10g Flashback primary database Depending on the current redo apply state 1. Too short or no delay also Flashback standby database 2. PITR Resetlogs SCN not yet applied no action (recovery through resetlogs is supported) 19

PITR of Primary without Recreating Standby (2) &01,- 31 ' &01,- +!"#,* " &01,- 31 ' )-" +!"# 2$1 31 ' %" &#4!5,/ )4)** 20

PITR of Primary without Recreating Standby (3) PRIMARY SELECT current_scn FROM V$DATABASE; UPDATE hr.employees SET SALARY=1; ALTER SYSTEM SWITCH LOGFILE; SHUTDOWN IMMEDIATE STARTUP MOUNT EXCLUSIVE FLASHBACK DATABASE TO SCN scn; STANDBY ALTER DATABASE OPEN RESETLOGS; SELECT resetlogs_change#-2 FROM V$DATABASE; DISABLE DATABASE 'PHYS_SITE2'; SHUTDOWN IMMEDIATE; STARTUP MOUNT; FLASHBACK STANDBY DATABASE TO SCN scn; ENABLE DATABASE 'PHYS_SITE2'; &#4!5,/ )4)** 21

Concept of Database Reinstate (1) %8- + %8- %" %",- '!"/!+,- 6 - prior to 10g backup of the new primary database duplicate with backup recreate standby database as of 10g reinstate database (only with manual intervention after failover immediate ) 22

Concept of Database Reinstate (2) &01,- %8- %8- %" %" &01 + %8-,- %" 2$1 31 ' %" FAILOVER TO 'PHYS_SITE2' 2$1 31 ',- %" REINSTATE DATABASE 'PHYS_SITE1' &01 2$1 31 ',- %" &#4!5,/ )4)** 23

Test application patch install on Standby (1) 6 9,,8! prior to 10g backup of the primary DB or clone the database with 10g open standby database temporarily (read/write) for testing purposes Flashback standby database to state before open (read/write) 24

Test application patch install on Standby (2) Prerequisites Data Guard configuration ready (SUCCESS) and enough space in the flashback area for the guaranteed restore point (db_recovery_file_dest(_size) must be defined) No use of Data Guard Broker commands, directly in sqlplus Attention: Perform only with a second standby database, otherwise no disaster protection (activation stops redo-transfer (ORA-16009)) Guarantee that normal clients cannot connect to the temporarily opened standby 25

Test application patch install on Standby (3),4,!- --+-* ),-* )7 - +!"#,!- )-" */),- ),,- ",!- &#4!5,/ )4)** 26

Test application patch install on Standby (4) PRIMARY STANDBY EDIT DATABASE 'PHYS_SITE2' SET STATE='LOG-APPLY-OFF'; DISABLE DATABASE 'PHYS_SITE2'; CREATE RESTORE POINT before_patch GUARANTEE FLASHBACK DATABASE; ALTER DATABASE ACTIVATE STANDBY DATABASE; SHUTDOWN IMMEDIATE STARTUP MOUNT ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE; ALTER DATABASE OPEN; : #,,--,!!-) &#4!5,/ )4)** 27

Test application patch install on Standby (5) PRIMARY STANDBY SHUTDOWN IMMEDIATE STARTUP MOUNT FLASHBACK DATABASE TO RESTORE POINT before_patch; ALTER DATABASE CONVERT TO PHYSICAL STANDBY; SHUTDOWN IMMEDIATE STARTUP MOUNT DROP RESTORE POINT before_patch; ENABLE DATABASE 'PHYS_SITE2'; EDIT DATABASE 'PHYS_SITE2' SET STATE='ONLINE'; &#4!5,/ )4)** 28

Snapshot Standby Database with 11g Temporarily convert a physical standby to a normal database DGMGRL> convert database 'PHYS_SITE2' to snapshot standby; Converting database "PHYS_SITE2" to a Snapshot Standby database, please wait... Database "PHYS_SITE2" converted successfully Now, do your tests on the standby site Afterwards, convert again back to physical standby DGMGRL> convert database 'PHYS_SITE2' to physical standby; Converting database "PHYS_SITE2" to a Physical Standby database, please wait... Operation requires shutdown of instance "PHYS" on database "PHYS_SITE2"... Operation requires startup of instance "PHYS" on database "PHYS_SITE2"... Continuing to convert database "PHYS_SITE2"... Operation requires shutdown of instance "PHYS" on database "PHYS_SITE2"... Operation requires startup of instance "PHYS" on database "PHYS_SITE2"... Database "PHYS_SITE2" converted successfully Compared to manual solution of 10g, redo transfer is not stopped during snapshot period 29

Snapshot Standby Database with 11g Configuration DGMGRL> show configuration Configuration Name: PHYS Enabled: YES Protection Mode: MaxAvailability Databases: PHYS_SITE1 - Primary database PHYS_SITE2 - Snapshot standby database PHYS_SITE3 - Physical standby database V$DATABASE View SQL> select open_mode, database_role from v$database; OPEN_MODE DATABASE_ROLE ---------- ---------------- READ WRITE SNAPSHOT STANDBY Fast-Start Failover: DISABLED Current status for "PHYS": SUCCESS 30

Test Application Patch Install on Standby 11g / 10g )7 -,!,!- --+-* ),-* )7 -,!- +!"#,!-,!- " ),,- )-" */),-,!- 31 11 10 &#4!5,/ )4)**

Agenda History of Standby Databases Flashback Database Flashback enhances Data Guard Environment Our experience and service competence form the basis of your success. Rolling Upgrade with Transient Logical Standby Database 32

Rolling Upgrade Starting with Oracle 10g Release 1 (10.1.0.3), you can use a Logical Standby database with SQL Apply to perform a Rolling Upgrade of your Oracle Software and Database With Rolling Upgrade you can patch your database with a very short downtime (only the time to make a Switchover) Rolling Upgrade is also available for unsupported object types. However it needs more attention and no changes are possible on unsupported objects during the upgrade During a rolling upgrade (mixed mode state), you can perform tests on the standby database and if necessary abort the Rolling Upgrade procedure 33

Rolling Upgrade - Concept - - 8- ;9,, <<<= 8-! "# $# <<<= 8- <<<= * ;// <<< %" - - 8- ; 9,, 8- <<<= <<< <<<= <<< 34

Rolling Upgrade - Requirements Rolling Upgrade is only available for patch or release (major release, patch set, etc..) validated with Rolling Upgrade support Rolling Upgrade does not support an Oracle Data Guard broker configuration The LOG_ARCHIVE_DEST_n parameter for the logical standby database must be set to OPTIONAL to ensure that the primary database can proceed during upgrade of the standby database The COMPATIBLE parameter must have the same value on both databases The Data Guard protection mode must be set to MAXIMUM PERFORMANCE or MAXIMUM AVAILABILITY 35

Rolling Upgrade with Physical Standby Sounds better than it is... Rolling Upgrade with Transient Logical Standby You still need a Logical Standby database with all it s limitations What is different? The Logical Standby keeps the DBID We have 2 DIFFERENT databases with the SAME DBID SQL> alter database recover logical standby keep identity; keep identity is a 11g feature, it does not work for upgrade 10.x 11.1 For Upgrades from 10g use manual steps described in MAA White Paper Database Rolling Upgrade Using Transient Logical Standby Oracle Database 10g Release 2 36

Rolling Upgrade with Transient Logical Standby 11g Create a Data Guard Configuration without Broker create restore point pre_upgrade guarantee flashback database; Primary Phys. Standby alter database recover logical standby keep identity; @ catupgrd.sql; Primary Log. Standby Switchover Log. Standby Primary flashback database to restore point pre_upgrade; -- Restart Database with New Oracle Home 11.1.0.7 alter database convert to physical standby; -- Restart Database Phys. Standby Re-create Broker Configuration Maybe switchover Primary 11.1.0.6 11.1.0.7 37

Rolling Upgrade Conclusion Rolling upgrade is a procedure to perform an upgrade with less downtime Before beginning with a Rolling Upgrade, ensure that you have a lot of experience with Logical Standby Databases and Data Guard configurations Not supported with a Data Guard Broker Configuration Unsupported objects need attention during upgrade 38

Conclusion are a perfect combination of features Easy Recreation of Data from Past Recreation of Standby Databases is much easier Snapshot Standby Database is very valuable for Application Tests Rolling Upgrade can reduce downtime but needs to be tested and prepared very well 39

The Trivadis HA Package TVD-HA TM Currently TVD-HA TM consists of three main components DGADD for setup of a new environment DGMON for monitoring vip.sh for management of virtual IPs for Data Guard Efficiently creates Data Guard environments with high demands on quality assurance Monitors Data Guard environments Manages Virtual IPs within a Data Guard environment Supports Oracle versions 10g and above 40

The Trivadis Standby Package - TVD-Standby TM High Data Availability for Standard Edition (One), Express Edition Physical Standby Database Support Basic functionality of Data Guard log transport recovery (also with delay) But: No Zero data loss Role Changes Graceful Switchover Failover ASM and RAC Support Monitor Environment, Housekeeping 41

Thank you!? www.trivadis.com Baden Basel Bern Brugg Lausanne Zurich Düsseldorf Frankfurt/M. Freiburg i. Br. Hamburg Munich Stuttgart Vienna