RDS MIGRATION TOOL FAQ

Transcription

1 RDS MIGRATION TOOL FAQ I. General: 1.1 Why isn t the task percentage complete metric accurately reflecting the percent done? The percent complete metric is not 100% accurate as it is only an estimate. In actuality in some cases it can even go backwards as the task runs. This has to do with more information on the amount of data to be moved becoming available as the task progresses. 1.2 Why am I unable to connect to an RDS Database? There are several reason you may not be able to connect to an RDS database I. The username/password combination may be incorrect II. Check that the RDS Endpoint was used for the Server name field when creating the RDS MIGRATION TOOL Endpoint III. Check that the port specified when creating the RDS MIGRATION TOOL Endpoint is the one the RDS database is listening for connections on IV. Check that the security groups associated with the RDS Database allow connections from the RDS MIGRATION TOOL instance V. Check that the security groups associated with the RDS MIGRATION TOOL Instance allow connections to the RDS database. VI. Check to see if the RDS database is publicly accessible, this may be need if the RDS MIGRATION TOOL instance and the RDS database are not in the same VPC 1.3 My password with special characters doesn t seem to be working what can I do? There are cases where RDS MIGRATION TOOL isn t passing special characters in passwords correctly to databases. This definitely includes the semi-colon character. In order to connect to the database that is having problems try these solutions. First modify the endpoint that is having the password issue and enclose the password in {}. If enclosing the password in {} doesn t work unfortunately you will have to change the password and remove the special characters. 1.4 What happened to all of my foreign keys and secondary indexes? RDS MIGRATION TOOL does not migrate/create any database objects other tables and primary keys. 1.5 How can I get missing secondary objects into my target database?

2 To migrate these objects in your database please look at using native database tools for homogenous migrations or the Schema Conversion Tool for heterogeneous migrations. 1.6 How can I combine several tables with the same structure into a single table in the target database? To combine several tables into one simply create a task for each source table. The target endpoint can be reused among each task. If the tables have different names and/or schemas transformation rules can be used so that they all map to the same target. 1.7 Do I pay for idle RDS MIGRATION TOOL Instances, specifically ones that are not actively running tasks? Yes you currently pay for all launched RDS MIGRATION TOOL instances whether they are migrating data or not. This is the same as both EC2 and RDS. 1.8 How can I increase the speed of the initial load? There a few things you can do that can help increase speed of the initial load component of a task I. Ensure your target is not multi-az, you can turn that on when you're done with the load. II. Turn off backups/logging on the target during the load, turn it back on when the load is complete. III. Make sure that you have disabled all secondary database objects on the target, this includes triggers, Foreign Keys, and secondary indexes. IV. If you have LOBS in your tables make sure that the task is optimized for LOBS V. Check to ensure that your task/s are not overloading the RDS MIGRATION TOOL instance s CPU or memory VI. Finally using provisioned IOPS on the target may help. 1.9 How can I make my task with LOBs run faster? Tasks by default are set to run in Full LOB support mode. While this will move all of the LOBS in your tables it is also slow. To increase the speed with which your migration task runs, you should create a new task and set the task to use Limited Size LOB mode. When you choose this mode you need to ensure that the setting of the MAX LOB parameter is correct. This parameter should be set to the largest LOBS size for all of your tables Are default values on columns migrated?

3 RDS MIGRATION TOOL does not migrated default column values to the target database when RDS MIGRATION TOOL is used to create the target tables. If these values are required it is recommend to pre-create your tables on the target and select Do nothing or Truncate for the task setting of Target table preparation mode How can I connect to a database outside of my Amazon AWS VPC? In order to use RDS MIGRATION TOOL you need to have network connectivity between the RDS MIGRATION TOOL instance and your source and target database systems. This usually means you need be able to route connections from your VPC to your datacenter. This network connectivity can be achieved through a variety of means. Please review Setting Up a Network for Database Migration for some of the more common network configurations that will work. It is not an exhaustive list of all configurations that will work, only the most common. There may be other configurations that are not documented that will also work Why can t I seem to see find the logs for my task? Logs are not enabled by default. When you create a task you must make sure you select Enable logging under task settings on the task creation screen if you want to have access to your task logs Why does my task seem to start fine and run for a while fine then mysteriously stop working? This behavior is quite often seen when the RDS MIGRATION TOOL instance is undersized for the task or tasks running on it. Please review your instance metrics underneath the monitoring tab of your RDS MIGRATION TOOL instance to ensure that you have not exhausted all available memory and gone into SAWP and/or overloaded the CPU for the instance type you had selected. If you find that you have you can simply scale up the RDS MIGRATION TOOL instance without needing to re-create everything Why am I now getting errors about primary key violations after I restarted my task? This can usually be traced back to the settings associated with your task. You should check to see if your Target table preparation mode is set to Do nothing, in this case RDS MIGRATION TOOL does not do any prep or cleanup from a previous run. In order to restart you task and avoid these errors you need to remove rows inserted into the target tables from the previous run Why did my task do the Full Load fine, but now it appears stuck when replicating changes?

4 Check to see if you set the task s Target table preparation mode is set to Do nothing or Truncate. If your task is set this way, then check to see if you also created primary keys and/or unique indexes on the target tables. RDS MIGRATION TOOL tasks with these settings will not create either primary keys or unique indexes on the target tables as you have told RDS MIGRATION TOOL that you are handling the required setup of the target tables. Running CDC without these indexes will basically result in Full Table scans being performed for each update What is causing the initial load of my schemas to fail and the error Operation:getSchemaListDetails:errType=, status=0, errmessage=, errdetails= to be returned? This is usually caused by the database user that RDS MIGRATION TOOL is connecting as not having the required permissions in the source endpoint Is it possible to do a partial load of a table? Yes it is! You must setup a source filter in your table selection rules of the Table Mappings JSON used for the task. Using these filters you can build where clauses for what data you want or do not want to migrate 1.18 How can I throttle the RDS MIGRATION TOOL task during the Full Load portion? You can throttle the load by changing the task setting MaxFullLoadSubTasks from the default value of 8 to a lower number during task creation. This parameter controls how many tables the Full Load process loads in parallel Why is that my tasks keep taking longer and longer to start? If your first task starts quickly and each additional task starts to take longer and longer to start then you are probably hitting a resource limit on you RDS MIGRATION TOOL instance. In particular you should examine your allocated versus your used IOPS especially if you are trying to run multiple tasks with Redshift endpoints as they are IO intensive. You can either increase the IOPS for your storage, or split your tasks across multiple RDS MIGRATION TOOL instances Why am I seeing weird errors related to tasks starting or dying mysteriously that don t seem to be addressed in this document? The cause of these types errors are extremely varied, however we have found that a large number of these types of errors are caused by inadequate resources available on the RDS MIGRATION TOOL instance you are using. Please check your RDS MIGRATION TOOL Instance s

5 usage of CPU, Memory, Swap, and IOPS to ensure that your instance has enough of these resources for the tasks you have defined to run on it Why is my task with 100,000 tables failing? The task most likely has too many tables in it. Please reduce the number of tables in the task. If possible try to reduce the number of tables in the task to less than 60, Is there a limit to the numbers of tables a task can migrate? There is no set limit on number of tables per task. The number of tables per task is dependent upon a number of things including how many columns are in each table. However we have generally found that limiting the number of tables in a task to less than 60,000 is a good rule of thumb Why did RDS MIGRATION TOOL restart loading my table from the beginning? RDS MIGRATION TOOL will restart the loading of table from the beginning for any table that has not finished the initial load if for whatever reason the RDS MIGRATION TOOL task is restarted. Any tables that have completed their initial load will not be reloaded if a task is restarted, again only tables that had their full loads inflight when the task is interrupted are affected. 2 Oracle: 2.1 Why can t I pull data from views? To be able to extract data from views you must add the following to the Extra Connection Attributes in the advanced section of the Oracle source endpoint. exposeviews=true 2.2 Why are changes to the LOBS In my tables on Oracle 12c not being migrated? RDS MIGRATION TOOL uses LogMiner by default to capture changes. However on Oracle 12c, LogMiner does not support LOB columns. To capture changes to LOBS on Oracle 12c change to using the BinaryReader. 2.3 How do I use BinaryReader instead of LogMiner for capturing changes?

6 To enable the use of BinaryReader instead of LogMiner for capturing changes you need to perform the following actions I. Add uselogminerreader=n to the Extra Connection Attributes of the source Oracle endpoint II. Make sure you grant the following additional privilege to the RDS MIGRATION TOOL user connecting to the source Oracle endpoint SELECT ON V_$TRANSPORTABLE_PLATFORM 2.4 How do I resolve the error Oracle CDC stopped Oracle CDC maximum retry counter exceeded.? This error indicates that the needed archive logs were removed from your database server prior to RDS MIGRATION TOOL being able to mine them for changes. Please check your log retention policies in place on your database server and increase them. For an RDS database this can be accomplished by executing the following RDS procedure which in the example increases log retention to 24 hours. Exec rdsadmin.rdsadmin_util.set_configuration( archivelog retention hours,24); 2.5 Why aren t changes to LOBs captured for certain tables? Currently in order to capture changes to LOBS a primary key is required on the table. 2.6 Is there any way I can capture changes to LOBS in tables that don t have primary keys? Yes there are several methods you can employ in order to be able to capture changes from these tables. I. Add a primary key to the table. This can be as simple as adding an ID column and populating it with a sequence using a trigger. II. Create a materialized view of the table that includes a system generated ID as the primary key and migrate the materialized view rather than the table III. Create a logical standby, add a primary key to the table, and migrate from the logical standby 2.7 Can I get RDS MIGRATION TOOL to automatically add the required Supplemental Logging to my tables? Yes you can have RDS MIGRATION TOOL automatically add the required Supplemental Logging to your tables. We do not automatically add supplemental logging as we aim to do zero changes in the source endpoint by default. To have RDS MIGRATION TOOL do this you will need to add

7 the following to the Extra Connection Attributes in the advanced section of the target Oracle endpoint. addsupplementallogging=y 2.8 Why am I get ORA-12899: value too large for column Errors? These are normally caused by a mismatch in either charactersets used in the source and target or NLS settings. The NLS_Semantics parameters are most likely different between the source and target this is especially true for the column named in the message. A usual cause is when the source parameter is set to CHAR semantics and the target is set to BYTE. 2.9 Why am I getting ORA errors from RDS MIGRATION TOOL? The ORA are coming from the Oracle database side of the connection. Finding the root cause of the problem is never a simple or easy process as the source of this error can be from many different components. However a lot of these seem to be related to the database configuration or the configuration of a piece of networking gear between RDS MIGRATION TOOL and the database. Sometime setting the sqlnet.ora parameters inbound_connect_timeout and inbound_connect_timeout_listenername to 0 can stop the ORA error. It may be useful to try setting these parameters if not set already and seeing if they eliminate the ORA Why are my NUMBERs being interpreted incorrectly? The Oracle NUMBER data type is converted into various RDS MIGRATION TOOL datatypes depending on the precision and scale of the NUMBER. These conversions are documented here Source Data Types for Oracle. The way the NUMBER type is converted can also be affected by using Extra Connection Attributes for the source Oracle endpoint. These Extra Connection Attributes are documented in Using Extra Connection Attributes with AWS Database Migration Service: Oracle. 3 MySQL: 3.1 My CDC task is failing because my RDS Database is reporting binary logging is disabled, how do I resolve this? To resolve this issue please enable backups for your RDS MySQL database. 3.2 Migration is working but I m not seeing any data in the target?

8 Verify that autocommit is on in the target, it s required for the service. Alternatively autocommit can be set on a per session basis by adding the following to the Extra Connection Attributes in the advanced section of the target MySQL, Aurora, or Mariadb endpoint. Initstmt= SET AUTOCOMMIT=1 3.3 Is it possible to disable foreign keys during the load? It is possible to disable foreign key checks on MySQL by adding the following to the Extra Connection Attributes in the advanced section of the target MySQL, Aurora, or Mariadb endpoint. Initstmt=SET FOREIGN_KEY_CHECKS=0 3.4 Why is RDS MIGRATION TOOL converting some of my characters into the? character? This is usually seen when the source database is using characters that have been encoded using utf8mb4 charactersets. RDS MIGRATION TOOL currently does not support the migration of utf8mb I got a message about a Bad event in my logs and now I am no longer seeing changes being replicated, what happened? This is most likely from performing an unsupported DDL operation on the source database. This then caused an event the replication engine was unable to skip. The easiest fix for this is to perform full restart of the task from the beginning, which will reload the tables and start capturing changes at a point after the unsupported DDL was issued. 3.6 Why can t I capture changes in my RDS MySQL 5.5 database? In order to support change capture on a MySQL database full image row based binary logging must be enabled. This is only possible for RDS MySQL version 5.6 and above. Therefore, if you wish to use change capture on an RDS instance of version 5.5 or below, you ll need to upgrade to version 5.6 first. 3.7 Why does my task using a view as a source run to completion in under a minute, but no rows are copied over, and there are no errors? Unfortunately at this time a "view" in Postgres is not a supported data source for RDS MIGRATION TOOL.

9 3.8 How can I increase my binary log retention on RDS MySQL? In order to increase your log retention for RDS MySQL you need to run the following, which will set log retention to 24 hours. Call mysql.rds_set_configuration( binlog retention hours, 24); 3.9 Why am I seeing this message Some changes from the source database had no impact when applied to the target database. See attrep_apply_exceptions table for details. In my logs? When updating a column s value to its existing value, a zero rows affected message is returned from MySQL (unlike Oracle and SQL Server that perform an update of one row) Why am I getting errors about identifiers being too long of the form TARGET_LOAD E: RetCode: SQL_ERROR SqlState: HY000 NativeError: 1059 Message: MySQLhttp://ODBC 5.3(w) Driverhttp://mysqld Identifier name XXXXXXXXXXXXXXXXXXXXXXX is too long ODBC general error. (ar_odbc_stmt.c:4054)? When RDS MIGRATION TOOL is set to create the tables and primary keys in the target system it currently does not use the same names for the Primary Keys that were used in the source system. Instead the Primary key name is based upon the table s name. When the table name is long it can result in the auto generated identifier being longer than the allowed limits for MySQL. The solution to this problem currently is to pre-create the tables and Primary Keys in the target and use a task that is set to Do nothing or Truncate only in regards to the target tables Why am I see messages of the form [SOURCE_CAPTURE ]E: Column XXXXXXXXXX uses an unsupported character set [120112] A field data conversion failed. (mysql_endpoint_capture.c:2154)? These errors are commonly seen because the source tables and/or databases are using UTF8MB4. The UTF8MB4 character set is not supported at this time. In addition to checking your table definitions, please also check you database parameters related to connections and defaults for you instance. The following command can be used to see these parameters. SHOW VARIABLES LIKE %char% 4 PostgreSQL: 4.1 Why is the data in my columns of a user defined data type not being migrated correctly?

10 When replicating from a PostgreSQL source, AWS RDS MIGRATION TOOL creates the target table with the same data types for all columns, apart from columns with user-defined data types. In such cases, the data type is created as character varying in the target. 4.2 Why is RDS MIGRATION TOOL throwing the following error about not find a schema SQL_ERROR SqlState: 3F000 NativeError: 7 Message: ERROR: no schema has been selected to create in;? Check your Table mapping JSON, you most likely are using a wild card for schema and the source database you are using doesn t support that. 4.3 Why am I not seeing deletes and/or updates on my table being replicated? Check to see if your source table has a primary key. In the event that a table does not have a Primary Key, DELETE and UPDATE record operations will be ignored. 4.4 Are truncate statements propagated? Change processing of TRUNCATE operations is not supported. 4.5 Can I prevent PostgresSQL from capturing DDL? Yes you can prevent PostgreSQL from capturing DDL by adding the following to Extra Connection Attributes in the advanced section of the target PostgreSQL endpoint. captureddls=n 4.6 Can I select what schema the Database objects for capturing DDL are created? Yes you can control the schema that the database objects related to capturing DDL are created in. In order to do this add the following to Extra Connection Attributes in the advanced section of the target PostgreSQL endpoint. ddlartifactsschema=xyzddlschema 4.7 I migrated my tables from Oracle and now I can t seem to find them? When performing a migration from Oracle you will most likely need to supply transformation rules under the table mapping section of your task to convert the case of your table names. Oracle and PostgreSQL default to UPPERCASE and LOWERCASE respectively for table names,

11 migrating your data without using the transformation rules means that you will need to enclose your table names in quotes when referencing them. 4.8 Why am I not getting the tables and/or columns I expected when using a table mapping This is most likely due to the fact that by default Postgres is case insensitive for string comparisons. For example, if you have a select using test and two objects named Test and TEST you would by default be selecting both of them. 4.9 Why does my task using a view as a source run to completion in under a minute, but no rows are copied over, and there are no errors? Unfortunately at this time a "view" in Postgres is not a supported data source for RDS MIGRATION TOOL. 5 SQLServer: 5.1 Does the user RDS MIGRATION TOOL connects with need any special permissions? The RDS MIGRATION TOOL user requires the SysAdmin role in order to operate correctly when capturing changes. 5.2 How can I get RDS MIGRATION TOOL to capture changes from my RDS SQLServer? At this time RDS MIGRATION TOOL cannot capture changes from an RDS SQLServer database. However RDS MIGRATION TOOL can capture changes from a SQLServer database running onpremise or on EC Why do I keep getting errors capturing changes from my non-rds SQLServer database? These errors are usually related to all of the required pre-requisites for capturing changes not being met. In particular the most common pre-requisite that is often overlooked is the need for a FULL Database backup. This will be evident by looking for the following message in your task log SOURCE_CAPTURE E: No FULL database backup found (under the 'FULL' recovery model). To enable all changes to be captured, you must perform a full database backup Changes may be missed. (sqlserver_log_queries.c:2623). You should also review the RDS MIGRATION TOOL documentation: Using a Microsoft SQL Server Database as a Source for AWS Database Migration Service

12 5.4 What happened to my Identity columns? Identity columns are not preserved when using the Migration Service to create your target schema, you will need to add them after the load is complete. 5.5 What am I doing wrong, I keep getting the following error, RetCode: SQL_ERROR SqlState: HY000 NativeError: Message: This edition of SQL Server does not support publications.? You are most likely trying to use SQLServer Express as a source. However at this time SQLServer Express is not a supported database endpoint option for RDS MIGRATION TOOL. 5.6 Why are changes in my source SQLServer database not being reflected in my target endpoint? RDS MIGRATION TOOL requires a source SQLServer database to be in either FULL or BULK LOGGED Data recovery model in order to consistently capture changes. The SIMPLE model is not sufficient. The SIMPLE recovery model logs the minimal information to allow users to recover their database. All inactive log entries are automatically truncated when a checkpoint occurs. All operations are still being logged, but as soon as a checkpoint occurs the log is automatically truncated, which means that it becomes available for re-use and older log entries can be over-written. In this case events may be lost. This is why one of the over models is required. Please review the RDS MIGRATION TOOL Documentation: Using a Microsoft SQL Server Database as a Source for AWS Database Migration Service 6 Redshift: 6.1 Can I load my Redshift Cluster that is in a different region from my RDS MIGRATION TOOL instance? Unfortunately this cannot be done. RDS MIGRATION TOOL requires that the RDS MIGRATION TOOL Instance and the Redshift cluster be in the same region. 6.2 Why am I getting errors like Relation "attrep_apply_exceptions" already exists; and only some of my tables are being migrated? This can occur when you have mistakenly created your Redshift endpoint as a Postgres endpoint. You will need to modify your endpoint and change the Target engine from postgres to redshift.

13 6.3 What is going on I keep getting errors related to tables whose names start with awsrds Migration Tool_changes? Please check that you do not have two tasks currently trying to load data into the same Redshift endpoint at the same time. Due to the way temporary tables are named, this can result in the types of errors you are seeing. 6.4 Why am I see tables with names like RDS Migration Tool.awsRDS Migration Tool_changes XXXX being created in my Redshift cluster? These tables are temporary tables that are created when data is being loaded from files stored in S3. Because of the way the Redshift COPY command works these tables are need to store data when it is first loaded and before it is placed in its final target table. 6.5 What Permissions are needed to use a Redshift Endpoint? In order to use RDS MIGRATION TOOL with a Redshift target endpoint you need to provide RDS MIGRATION TOOL with a user that can connect to your Redshift cluster and has the following permissions: I. CRUD (Select, Insert, Update, Delete) II. Bulk Load III. Create, Alter, Drop (if required by the task's definition)