Cloud Infrastructure Maintenance Updates

Oracle performs the updates to all of the Oracle-managed infrastructure components on Exadata Cloud Infrastructure.

You may manage contacts who are notified regarding infrastructure maintenance, set a maintenance window to determine the time your quarterly infrastructure maintenance will begin, and also view scheduled maintenance runs and the maintenance history of your Exadata Cloud Infrastructure in the Oracle Cloud Infrastructure Console. For details regarding the infrastructure maintenance process and configuring the maintenance controls refer to the following:

About Oracle-managed Exadata Cloud Infrastructure Maintenance

Oracle performs patches and updates to all of the Oracle-managed system components on Exadata Cloud Infrastructure.

Oracle patches and updates include the physical database server hosts, Exadata Storage Servers, Network Fabric Switches, management switch, power distribution units (PDUs), integrated lights-out management (ILOM) interfaces, and Control Plane Servers. This is referred to as infrastructure maintenance.

The frequency of the updates depends on the region type, as follows:

  • Commercial regions: Oracle performs full quarterly infrastructure updates and monthly security infrastructure updates.
  • Government regions: Oracle performs monthly full infrastructure maintenance updates.

In all but rare exceptional circumstances, you receive advance communication about these updates to help you plan for them. If there are corresponding recommended updates for your VM cluster virtual machines (VMs), then Oracle provides notifications about them.

Wherever possible, scheduled updates are performed in a manner that preserves service availability throughout the update process. However, there can be some noticeable impact on performance and throughput while individual system components are unavailable during the update process.

For example, database server patching typically requires a reboot. In such cases, wherever possible, the database servers are restarted in a rolling manner, one at a time, to ensure that the service remains available throughout the process. However, each database server is unavailable for a short time while it restarts, and the overall service capacity diminishes accordingly. If your applications cannot tolerate the restarts, then take mitigating action as needed. For example, shut down an application while database server patching occurs.

Overview of the Quarterly Infrastructure Maintenance Process

By default, infrastructure maintenance updates the Exadata database server hosts in a rolling fashion, followed by updating the storage servers.

You can also choose non-rolling maintenance to update database and storage servers. The non-rolling maintenance method first updates your storage servers at the same time, then your database servers at the same time. Although non-rolling maintenance minimizes maintenance time, it incurs full system downtime while the storage servers and database servers are being updated.

Rolling infrastructure maintenance begins with the Exadata database server hosts. For the rolling maintenance method, database servers are updated one at a time. Each of the database server host's VMs is shut down, the host is updated, restarted, and then the VMs are started, while other database servers remain operational. This rolling maintenance impacts older applications not written to handle a rolling instance outage. This process continues until all servers are updated.

After database server maintenance is complete, storage server maintenance begins. For the rolling maintenance method, storage servers are updated one at a time and do not impact VM cluster VM's availability. However, the rolling storage server maintenance can result in reduced IO performance as storage servers are taken offline (reducing available IO capacity) and resynced when brought back online (small overhead on database servers). Properly sizing the database and storage infrastructure to accommodate increased work distributed to database and storage servers not under maintenance will minimize (or eliminate) any performance impact.

Note

While databases are expected to be available during the rolling maintenance process, the automated maintenance verifies Oracle Clusterware is running but does not verify that all database services and pluggable databases (PDBs) are available after a server is brought back online. The availability of database services and PDBs after maintenance can depend on the application service definition. For example, a database service, configured with certain preferred and available nodes, may be relocated during the maintenance and wouldn't automatically be relocated back to its original node after the maintenance completes. Oracle recommends reviewing the documentation on Achieving Continuous Availability for Your Applications on Exadata Cloud Systems to reduce the potential for impact to your applications. By following the documentation's guidelines, the impact of infrastructure maintenance will be only minor service degradation as database servers are sequentially updated.

Oracle recommends that you follow the Maximum Availability Architecture (MAA) best practices and use Data Guard to ensure the highest availability for your critical applications. For databases with Data Guard enabled, Oracle recommends that you separate the maintenance windows for the infrastructure instances running the primary and standby databases. You may also perform a switchover prior to the maintenance operations for the infrastructure instance hosting the primary database. This allows you to avoid any impact on your primary database during infrastructure maintenance.

Prechecks are performed on the Exadata Cloud Infrastructure components prior to the start of the maintenance window. The goal of the prechecks is to identify issues that may prevent the infrastructure maintenance from succeeding. The Exadata infrastructure and all components remain online during the prechecks. An initial precheck is run approximately 5 days prior to the maintenance start and another precheck is run approximately 24 hours prior to maintenance start. If the prechecks identify an issue that requires rescheduling the maintenance notification is sent to the maintenance contacts.

Note

Do not perform major maintenance operations on your databases or applications during the patching window, as these operations could be impacted by the infrastructure maintenance operations

Time Estimates for Quarterly Maintenance Windows

The time taken to update infrastructure components varies depending on the number of database servers and storage servers in the Exadata infrastructure, the maintenance method and whether custom action has been enabled.

The approximate times provided are estimates. Time for custom action, if configured, is not included in the estimates. Database server maintenance time may vary depending on the time required to shutdown each VM before the update and then start each VM and associated resources after the update of each node before proceeding to the next node. The storage server maintenance time will vary depending on the time required for the ASM rebalance, which is not included in the estimates below. If issues are encountered during maintenance this may also delay completion beyond the approximate time listed. In such a situation, if Oracle cloud operations determine resolution would extend beyond the expected window, they will send a notification and may reschedule the maintenance.

Note

The timeframes mentioned below can change if Oracle cloud operations determine that additional maintenance work is needed. If additional time is necessary, Oracle will send a customer notification in advance to inform customers that additional time will be required for the next quarterly maintenance window.

Table 5-1 Approximate Times for Exadata Infrastructure Maintenance

Exadata Shape Configuration Rolling Patching Method (Approximate Time) Non-Rolling Patching Method (Approximate Time)
Quarter rack 5-6 hours 4-7 hours
Half rack 10 hours 4-7 hours
Full rack 20 hours 4-7 hours
Flexible shapes (X8M and higher) 1.5 hours per compute node + 1 hour per storage node 4-7 hours

Overview of Monthly Security Maintenance

Security maintenance, performed alongside the quarterly maintenance, is executed in months when important security updates are needed and includes fixes for vulnerabilities with CVSS scores greater than or equal to 7.

Note

For more information about the CVE release matrix, see Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1).

To view the CVE release matrix specific to an Exadata Infrastructure version, click the Exadata version, for example, Exadata 23. Version-specific CVE release matrices are listed in the Notes column of the table.

Security maintenance, when needed, is scheduled to be applied during a 21-day window that begins between the 18th-21st of each month and will run till the 9th-12th of the next month. Customers will receive notification of the proposed schedule at least 7 days before the start of the monthly maintenance window and can reschedule monthly maintenance to another date in the window if desired. The monthly security maintenance process updates database servers to fix critical security vulnerabilities and critical product issues. Monthly maintenance also updates storage servers to an Exadata Storage Software image that resolves known security vulnerabilities and product issues.

Updates to database servers are applied online via Ksplice technology, and have no impact to workloads running on the compute (database) servers, as database server security updates are applied online to the host server while your VM and all processes within the VM, including databases, remain up and running. Servers and VMs are not restarted. Updates to storage servers are applied in a rolling fashion. As with quarterly maintenance, the impact of rebooting storage servers should be minimal to applications.

While updating your services infrastructure, some operations including memory, and storage scaling, operating system and Grid Infrastructure patching (including prechecks), and elastic expansion of compute and storage servers may be blocked.
Note

CPU scaling and VM startup/shutdown are the only operations supported during monthly infrastructure maintenance.
Please plan to defer these operations until after the updates are complete. Application of security updates takes about 15 minutes per DB server host, plus 60 minutes per storage server depending on the I/O activity. If you attempt an affected operation, the console will notify you of the ongoing security updates. No software is updated in the guest VMs.

Understanding Monthly and Quarterly Maintenance in the Same Month

Special considerations are made when both quarterly and monthly security maintenance are scheduled to run in the same month. Quarterly maintenance will reapply any security fixes already applied by security maintenance, and neither quarterly nor monthly maintenance will apply a storage server update if the existing storage server version is the same or newer than the version contained in the update.

  • The contents of the updates applied during quarterly maintenance are determined at the start of the maintenance quarter and use the latest Exadata release from the month prior to the start of the maintenance quarter. If any additional security fixes are available at that time, those updates are included in the quarterly maintenance. That image is then used throughout the quarter. For example, the January release is used for quarterly maintenance in Feb, March, and April.
  • When quarterly maintenance is applied it is possible there are security updates previously installed on the database servers are not included in the quarterly maintenance release to be applied. In that case, the automation will apply the same security fixes to new release installed by the quarterly maintenance so there will not be any regression in security fixes. If the current image on the storage server is the same or newer than that to be applied by the quarterly or monthly security maintenance, that maintenance will be skipped for the storage servers.

If quarterly maintenance is scheduled within 24 hours of the time the monthly is scheduled, the scheduled monthly maintenance will be skipped and the monthly update will instead be applied immediately following the quarterly maintenance.

  • When scheduled at the same time, the monthly update is executed immediately following the completion of the quarterly maintenance.
  • If monthly maintenance is scheduled to begin 0-24 hours ahead of the quarterly maintenance, then the monthly maintenance will not execute as scheduled, but instead, wait and be executed immediately following the quarterly maintenance. If the quarterly maintenance is subsequently rescheduled, then the monthly security maintenance will begin immediately. Oracle, therefore, recommends scheduling quarterly and monthly maintenance at the same time. As a result, if you reschedule the quarterly at the last moment, the monthly maintenance will run at the scheduled time instead of immediately upon editing the schedule. You can also reschedule the monthly security maintenance when rescheduling the quarterly maintenance as long as you keep the monthly within the current maintenance window. Monthly maintenance can be rescheduled to another time in the maintenance window, but cannot be skipped.

Monthly Security Maintenance before Quarterly Maintenance

  • To apply security maintenance before quarterly maintenance, reschedule the monthly security maintenance to occur more than 24 hours prior to the quarterly maintenance. The security maintenance will online apply security patches to the database servers with no impact to applications, and apply an update to the storage servers with minimal to no impact (may be slight performance degradation) on applications. The quarterly maintenance will follow as scheduled, and will perform rolling maintenance on the database servers, which will impact applications not written to handle a rolling reboot. As part of the quarterly maintenance, it will apply the same security updates to the database server that are already installed on the system (no security regression).
  • If you are concerned about getting the latest security updates applied, schedule the monthly security maintenance to run after the new monthly maintenance window opens (usually on the 21st of the month).
  • The impact of the monthly security maintenance rebooting the storage servers should be minimal, so impact to the applications during this month will only be due to the restart of the database servers during the quarterly maintenance. However, if you must coordinate a maintenance window with your end users for the security maintenance, this will require two maintenance windows.

Quarterly Maintenance before Monthly Security Maintenance

  • To run the quarterly maintenance before the monthly security maintenance, reschedule the security maintenance to run no earlier than 24 hours before the quarterly maintenance is scheduled to start. The security maintenance will be deferred until the quarterly maintenance is completed. The quarterly maintenance will perform rolling maintenance on the database servers, which will impact applications not written to handle a rolling reboot. The quarterly maintenance may or may not skip the storage server patching. That depends on if it is newer or older than the release currently installed. In most cases, the version installed should be newer than the version associated with the quarterly maintenance. Exceptions to this rule may occur if it is the first month of a maintenance quarter, or you skipped the security maintenance in one or more prior months. The security maintenance will run either immediately after the quarterly maintenance is completed, or when scheduled, whichever is later. It will apply online updates to the database servers (no application impact) and will likely update the storage servers in a rolling manner. In some corner cases. the quarterly maintenance may contain the same storage server release as the security maintenance and the security maintenance storage server updates will be skipped.
  • The impact to end users of running the quarterly maintenance before the security maintenance should be roughly the same as running the security maintenance first. The quarterly maintenance will be a disruptive event, but the security maintenance rebooting the storage servers should cause minimal disruption, and the security maintenance is applied to the database servers online. However, if you must coordinate a maintenance window with your end users for the security maintenance, this will require two maintenance windows. You can schedule those two maintenance windows to run back-to-back, to appear as single maintenance window to end users. To do this, reschedule the security maintenance to start at the same time (or up to 24 hours prior) as the quarterly maintenance. The security maintenance will be deferred until the quarterly maintenance is completed. Assuming you have been regularly applying monthly security maintenance, the storage servers will be skipped by the quarterly maintenance and will be updated by the security maintenance immediately upon the completion of the quarterly maintenance.

Minimizing Maintenance Windows

  • To minimize the number of maintenance windows (you have to negotiate those with end users), schedule the quarterly maintenance and monthly maintenance at the same time. The security maintenance will be blocked. The quarterly maintenance will update the database servers in a rolling manner and will most likely skip the storage server. The security maintenance will follow up immediately and update the database servers online and the storage servers in a rolling manner. The result is a single database and storage server restart in a single maintenance window.
  • There are two exceptions to this. 1. If the quarterly and monthly maintenance contain the same storage server release, the quarterly maintenance will apply the storage server update, and the security maintenance will be skipped. From your perspective, this is still a single rolling reboot in a single maintenance window. 2. The currently installed release on the storage servers is older than that contained in the quarterly maintenance, which in turn is older than that in the security maintenance. That would cause the quarterly maintenance to update the storage, and then the security maintenance to do it as well. This can only happen if you skipped a prior month's security maintenance, because it requires the current image to be at least 2 months out of date. In such a scenario, you may want to schedule the security maintenance first and then the quarterly maintenance. This would result in one storage server reboot, but two distinct maintenance windows — the first for the security maintenance, and then later the quarterly maintenance.
  • To minimize the impact to your end users, always apply the monthly security updates, and in months where both are scheduled, schedule them at the same time.
Note

If the Exadata Infrastructure is provisioned before Oracle schedules the security maintenance, then it will be eligible for security maintenance.

Any time before the scheduled monthly Exadata Infrastructure maintenance, you can reschedule it.

Using the Console to Configure Oracle-Managed Infrastructure Updates

Software updates are scheduled quarterly and monthly. You can use the the console to schedule and plan for them.

Full Exadata Cloud Infrastructure software updates are scheduled on a quarterly basis for commercial regions, and monthly for government regions. In addition, important security updates are scheduled monthly. While you cannot opt-out of these infrastructure updates, Oracle alerts you in advance through the Cloud Notification Portal and allows scheduling flexibility to help you plan for them.

For quarterly infrastructure maintenance, you can set a maintenance window to determine when the maintenance will begin. You can also edit the maintenance method, enable custom action, view the scheduled maintenance runs and the maintenance history, and manage maintenance contacts in the in the Exadata Infrastructure Details page of the Oracle Cloud Infrastructure Console.

To set the automatic quarterly maintenance schedule for Exadata Cloud Infrastructure

To view or edit the properties of the next scheduled quarterly maintenance for Exadata Cloud Infrastructure

To view the maintenance history of an Exadata Cloud Infrastructure resource

To set the node patching order for a scheduled infrastructure maintenance run

This task describes how to set the node patching order for a scheduled infrastructure maintenance run for a cloud Exadata infrastructure or Exadata DB system resource.

Note

By default, all scheduled maintenance runs are initially set to use rolling patching. To use non-rolling patching, you must change this setting for each maintenance run after it is scheduled.

  1. Open the navigation menu. Click Oracle Database, then click Exadata on Oracle Public Cloud.
  2. Navigate to the cloud Exadata infrastructure or DB system you want to access:
    • Cloud Exadata infrastructure (new resource model): Under Oracle Exadata Database Service on Dedicated Infrastructure, click Exadata Infrastructure. In the list of infrastructure resources, find the infrastructure you want to access and click its highlighted name to view its details page.

    • DB systems: Under Bare Metal, VM, and Exadata, click DB Systems. In the list of DB systems, find the Exadata DB system you want to access, and then click its name to display details about it.

  3. On the resource details page, under Maintenance, click the view link in the Next Quarterly Maintenance field.
  4. On the Maintenance page, click the edit link in the Maintenance Method field for a scheduled cloud Exadata infrastructure maintenance run.
  5. In Update Exadata Infrastructure Node Patching Order, change the maintenance method to either Rolling or Non-rolling as needed.

Monitor Infrastructure Maintenance Using Lifecycle State Information

The lifecycle state of your Exadata Infrastructure resource enables you to monitor when the maintenance of your infrastructure resource begins and ends.

In the Oracle Cloud Infrastructure Console, you can see lifecycle state details messages on the Exadata Infrastructure Details page when a tooltip is displayed beside the Status field. You can also access these messages using the ListCloudExadataInfrastructures API, and using tools based on the API, including SDKs and the OCI CLI.

During infrastructure maintenance operations, you can expect the following:
  • If you specify a maintenance window, then patching begins at your specified start time. The infrastructure resource's lifecycle state changes from Available to Maintenance in Progress.
    Note

    The prechecks are now done prior to the start of the maintenance.
  • When Exadata database server maintenance starts, the infrastructure resource's lifecycle state is Maintenance in Progress, and the associated lifecycle state message is, The underlying infrastructure of this system (dbnodes) is being updated.
  • When storage server maintenance starts, the infrastructure resource's lifecycle state is Maintenance in Progress, and the associated lifecycle state message is, The underlying infrastructure of this system (cell storage) is being updated and this will not impact Database availability.
  • After storage server maintenance is complete, the networking switches are updated one at a time, in a rolling fashion.
  • When maintenance is complete, the infrastructure resource's lifecycle state is Available, and the Console and API-based tools do not provide a lifecycle state message.

Receive Notifications about Your Infrastructure Maintenance Updates

There are two ways to receive notifications. One is through email to infrastructure maintenance contacts and the other one is to subscribe to the maintenance events and get notified.

Oracle schedules maintenance run of your infrastructure based on your scheduling preferences and sends email notifications to all your infrastructure maintenance contacts. You can login to the console and view details of the schedule maintenance run. Appropriate maintenance related events will be generated as Oracle prepares for your scheduled maintenance run, for example, schedule reminder, patching started, patching end, and so on. For more information about all maintenance related events, see Oracle Cloud Exadata Infrastructure Events. In case, if there are any failures, then Oracle reschedules your maintenance run, generates related notification, and notifies your infrastructure maintenance contacts.

For more information about Oracle Cloud Infrastructure Events, see Overview of Events. To receive additional notifications other than the ones sent to infrastructure maintenance contacts, you can subscribe to infrastructure maintenance events and get notified using the Oracle Notification service, see Notifications Overview.

Managing Infrastructure Maintenance Contacts

Learn to manage your Exadata infrastructure maintenance contacts.

To manage maintenance contacts in an Exadata Cloud Infrastructure

Manage contacts for Exadata infrastructure maintenance notifications using the Console.

To prevent an Exadata infrastructure administrator from being overwhelmed by system update notifications, you can specify up to 10 email addresses of people to whom maintenance notifications are sent.

  1. Open the navigation menu. Click Oracle Database, then click Oracle Exadata Database Service on Dedicated Infrastructure.
  2. In the Oracle Exadata Database Service on Dedicated Infrastructure section, click Exadata Infrastructure to display a list of Exadata infrastructures in the default compartment. You can select a different compartment from the Compartment drop-down located in the List Scope section.
  3. In the list of Exadata infrastructure resources, find the infrastructure you want to access and click its highlighted name to view its details page.
  4. In the Maintenance section, click Manage in the Customer Contacts field to display the Manage Contacts dialog.
  5. Click the Add Contacts button to display a field in which to enter a valid email address. You can have up to 10 maintenance contacts for each Exadata infrastructure.
  6. To edit an email address, in the Manage Contacts dialog, select the box preceding the email address you want to edit and click the Edit button.
  7. To remove an email address from the list, in the Manage Contacts dialog, select the box preceding the email address you want to remove and click the Remove button.

Using the API to Manage Exadata Cloud Infrastructure Maintenance Controls

Use these API operations to manage Exadata Cloud Infrastructure maintenance controls and resources.

For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.

Use these API operations to manage Exadata Cloud Infrastructure maintenance controls.

Cloud Exadata infrastructure resource (new resource model):