Troubleshooting Instances Using Instance Console Connections

You can remotely troubleshoot malfunctioning instances using console connections. For example:

  • An imported or customized image that does not complete a successful boot
  • A previously working instance that stops responding

Two types of instance console connections exist: serial console connections and VNC console connections.

Important

Instance console connections are for troubleshooting purposes only. To connect to a running instance for administration and general use, instead use a Secure Shell (SSH) or Remote Desktop connection. See Connecting to an Instance.

You can connect to the serial console quickly and easily using the Cloud Shell integration. If you're using Cloud Shell to connect to the serial console, the only prerequisite is having the correct permissions. If you want to make a local connection either to the serial console or to the VNC console, follow these configuration steps.

  1. Make sure you have the correct permissions.
  2. Complete the prerequisites, including creating your SSH key pair.
  3. Create the instance console connection.
  4. Connect to the serial console or connect to the VNC console.
  5. If you're trying to connect to the serial console and you think the connection isn't working, test your connection to the serial console using Cloud Shell.
Note

Some issues can be diagnosed using information in the console history. Console history lets you see serial output from your instance without having to connect to the instance remotely.

Required IAM Policies

To use Oracle Cloud Infrastructure, an administrator must be a member of a group granted security access in a policy  by a tenancy administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don't have permission or are unauthorized, verify with the tenancy administrator what type of access you have and which compartment  your access works in.

To create instance console connections, an administrator needs to grant user access to manage instance console connections and to read instances through an IAM policy. The resource name for instance console connections is instance-console-connection. The resource name for instances is instance. The following policies grant users the ability to create instance console connections:

Allow group <group_name> to manage instance-console-connection in tenancy
Allow group <group_name> to read instance in tenancy

Instance console connections also support network sources. The following policies grant users the ability to create instance console connections with a network source:

Allow group <group_name> to manage instance-console-connection in tenancy where request.networkSource.name='example-network-source'
Allow group <group_name> to read instance in tenancy where request.networkSource.name='example-network-source'

If you're new to policies, see Getting Started with Policies and Common Policies.

Before You Begin

Complete these prerequisites before creating the instance console connection.

Important

If you're using Cloud Shell to connect to the serial console, the only prerequisite is having the correct permissions.

Installing an SSH Client and a Command-line Shell (Windows)

Windows does not include an SSH client by default. If you are connecting from a Windows client, you need to install an SSH client. You can use PuTTY plink.exe with Windows PowerShell or software that includes a version of OpenSSH such as:

The instructions in this topic frequently use PuTTY and Windows PowerShell.

If you want to make the console connection from Windows with Windows PowerShell, PowerShell might already be installed on your Windows operating system. If not, follow the steps at the link. If you are connecting to the instance from a Windows client using PowerShell, plink.exe is required. plink.exe is the command link connection tool included with PuTTY. You can install PuTTY or install plink.exe separately. For installation information, see http://www.putty.org.

Creating SSH Key Pairs

To create the secure console connection, you need an SSH key pair. The method to use for creating key pairs depends on your operating system. When connecting to the serial console, you must use an RSA key. The instructions in this section show how to create an RSA SSH key pair.

Creating the SSH key pair for Linux

For detailed instructions about creating an SSH key pair to use on Linux, see Managing Key Pairs on Linux Instances.

Creating the SSH key pair for Windows using PuTTY

If you are using a Windows client to connect to the instance console connection, use an SSH key pair generated by PuTTY.

Signing in to an instance from the serial console (optional)

To troubleshoot instances and see serial output using the serial console, you don't need to sign in. To connect to a running instance for administration and general use with Secure Shell (SSH) or Remote Desktop connection, see Connecting to an Instance.

If you want to sign in to an instance using an instance console connection, you can use Secure Shell (SSH) or Remote Desktop connection to sign in. If you want to sign in with a username and password, you need a user account with a password. Oracle Cloud Infrastructure does not set a default password for the opc user. Therefore, if you want to sign in as the opc user, you need to create a password for the opc user. Otherwise, add a different user with a password and sign in as that user.

Connecting Through Firewalls

If your system is behind a firewall, the system must be able to reach the console servers. The client system connecting to the serial console must be able to reach the serial or VNC console server (for example, instance-console.us-ashburn-1.oci.oraclecorp.com) over SSH using port 443, directly or through a proxy.

Supported Instance Types

Serial console connections are supported on the following types of instances:

  • Virtual machine (VM) instances launched in September 2017 or later.
  • Bare metal instances launched in November 2017 or later.

VNC console connections are supported on the following types of instances:

  • VM instances launched on October 13, 2017 or later

  • Most bare metal instances are supported, with the following exceptions.

Using Cloud Shell to Connect to the Serial Console

You can connect to the serial console quickly and easily using the Cloud Shell integration. Cloud Shell is a web browser-based terminal accessible from the Console. The Cloud Shell integration automatically creates the instance console connection and a temporary SSH key. The only prerequisite for connecting to the serial console from Cloud Shell is granting users the correct permissions. For an introductory walkthrough of using Cloud Shell, see Using Cloud Shell.

By default, Cloud Shell limits network access to Oracle Cloud Infrastructure internal resources in your tenancy home region, unless you enable the Cloud Shell managed Public Network. Your administrator can configure an Identity policy to enable Cloud Shell Public Network. For more information, see Cloud Shell Networking.

Note

You cannot use Cloud Shell for VNC console connections. You can only use it for serial console connections.

When you are finished with the serial console and have terminated the SSH connection, you should delete the serial console connection. If you do not disconnect from the session, Oracle Cloud Infrastructure terminates the serial console session after 24 hours and you must reauthenticate to connect again.

Creating the Instance Console Connection

Before you can make a local connection to the serial console or VNC console, you need to create the instance console connection. When you use Cloud Shell to connect the the serial console, the instance console connection is created automatically.

Note

Instance console connections are limited to one client at a time. If the client fails, the connection remains active for approximately five minutes. During this time, no other client can connect. After five minutes, the connection is closed, and a new client can connect. During the five-minute timeout, any attempt to connect a new client fails with the following message:
channel 0: open failed: administratively prohibited: console access is limited to one connection at a time
Connection to <instance and OCID information> closed.
  1. Open the navigation menu  and select Compute. Under Compute, select Instances.
  2. Click the instance that you're interested in.
  3. Under Resources, click Console connection.

  4. Click Create local connection.
  5. Upload the public key portion for the SSH key. You have three options for adding the SSH key.
    • Generate a key pair for me: You can have Oracle Cloud Infrastructure generate an SSH key pair to use. If you are using PowerShell or PuTTY to connect to the instance from a Windows client, you cannot use the generated SSH key pair without first converting it to a .ppk file.
    • Upload public key file: Browse to a public key file on your computer. If you followed the prerequisite steps in Creating SSH Key Pairs to create a key pair, use this option to navigate to the .pub file.
    • Paste public key: Paste the content of your public key file into the text box.
  6. Click Create console connection.

    When the console connection has been created and is available, the state changes to Active.

Making a Local Connection to the Serial Console

After you create the console connection for the instance, you can connect to the serial console using a Secure Shell (SSH) connection. When making a local connection to the serial console, you must use an RSA key. You can use the same SSH key for the serial console that was used when you launched the instance, or you can use a different SSH key.

Tip

You can also use Cloud Shell to connect to the serial console.

When you are finished with the serial console and have terminated the SSH connection, you should delete the serial console connection. If you do not disconnect from the session, Oracle Cloud Infrastructure terminates the serial console session after 24 hours and you must reauthenticate to connect again.

Validating Server Host Keys

When you first connect to the serial console, you're prompted to validate the fingerprint of the server host key. The fingerprint of the server host key is the SHA256 hash of the server host's public SSH key. The server SSH handshake response is signed with the associated private key. Validating the server host key's fingerprint protects against potential attacks.

When you make a manual connection to the serial console, the fingerprint of the server host key is not automatically validated. To manually validate the fingerprint, compare the fingerprint value displayed in the Oracle Cloud Infrastructure Console to the value of the RSA key fingerprint that appears in the terminal when you connect.

To find the fingerprint of the server host key in the Console, on the Instance Details page, under Resources, click Console connection. The table displays the fingerprint of the server host key. The fingerprint in the Console should match the value of the RSA key fingerprint shown in the terminal when you connect to the serial console.

The server host keys are periodically rotated for security purposes. Key rotation reduces the risk posed when keys are compromised by limiting the amount of data encrypted or signed by one key version. When your key is rotated and you try to connect to the serial console, a warning appears indicating a potential attack. The warning includes a Host key verification failed error and a line number in your .ssh/known_hosts file. Delete that line in your .ssh/known_hosts file and then reconnect to the serial console. You are then prompted to accept a new server host key fingerprint.

Connecting from Mac OS X and Linux Operating Systems

Use an SSH client to connect to the serial console. Mac OS X and most Linux and UNIX-like operating systems include the SSH client OpenSSH by default.

Note

The minimum version required for OpenSSH to connect to the serial console from Linux and MacOS is OpenSSH 7.2.

Connecting from Windows Operating Systems

The steps to connect to the serial console from Windows PowerShell are different from the steps for OpenSSH. The following steps do not work in the Windows terminal.

Important

If you are connecting to the instance from a Windows client using PowerShell, plink.exe is required. plink.exe is the command link connection tool included with PuTTY. You can install PuTTY or install plink.exe separately. For more information, see Installing an SSH Client and a Command-line Shell (Windows).
Note

The minimum version required for OpenSSH to connect to the serial console from Windows is PuTTY (0.75).

Connecting to the VNC Console

After you create the console connection for the instance, you need to set up a secure tunnel to the VNC server on the instance, and then you can connect with a VNC client.

The VNC console connection uses SSH port forwarding to create a secure connection from your local system to the VNC server attached to your instance's console. Although this method is a secure way to use VNC over the internet, owners of multiuser systems should know that opening a port on the local system makes it available to all users on that system until a VNC client connects. For this reason, we don't recommend using this product on a multiuser system unless you take proper actions to secure the port or you isolate the VNC client by running it in a virtual environment, such as Oracle VM VirtualBox.

Exiting the Instance Console Connection

Tagging Resources

Apply tags to resources to help organize them according to business needs. Apply tags at the time you create a resource, or update the resource later with the wanted tags. For general information about applying tags, see Resource Tags.

Troubleshooting Instances from Instance Console Connections

You can troubleshoot using the boot process of a compute instance with an instance console connection. Troubleshooting options are available for Linux and Windows.

Troubleshooting Linux with Instance Console Connections

The following describes instance console troubleshooting options for instances running one of the following OSs: Oracle Autonomous Linux 8.x, Oracle Autonomous Linux 7.x, Oracle Linux 9.x, Oracle Linux 8.x, Oracle Linux 7.x, and Oracle Linux Cloud Developer 8.x.

Tip

Other Linux distributions and SSH clients might require different steps. For example, see details for Ubuntu recovery mode here: Using recovery modes.

After connecting to an instance console, you can perform various tasks, such as:

  • Boot into maintenance mode.
  • Edit system configuration files.
  • Add or reset the SSH keys for the opc user.
  • Reset the password for the opc user.

These tasks require you to boot into a bash shell in maintenance mode.

Troubleshooting Windows Instances from Instance Console Connections

The following Windows tools can help you troubleshoot a compute instance from an instance console connection.

Using Windows Special Administration Console

The Windows Special Administration Console (SAC) allows you to access a PowerShell console or command prompt from the serial terminal. By connecting to the instance's serial console and using SAC, you can interrupt the boot process and boot Windows in safe mode.

When you use SAC, you can create multiple user sessions or channels and switch between them. This feature enables you to use SAC commands while concurrently running command-line commands or viewing setup logs.

To use SAC in the serial console, first enable it on your Windows server. After you activate SAC, it provides a special console on the serial port. When enabled, the SAC> prompt appears in the serial output.

Boot menu commands

If the instance has the boot menu enabled and is rebooted after connecting through SSH, the Windows boot menu should display in the serial console output. The following list includes commands that you can use with the boot menu:

Boot menu
  • Enter: When the boot menu initiates and the operating system is highlighted, starts the operating system.
  • Tab: Switches to the Tools menu.
  • Esc: Exits the boot menu and restarts the instance.
  • Esc and then 8 or F8: Displays advanced options for the selected item.
  • Esc + left arrow: Returns to the initial boot menu.
Advanced boot options

Disable SAC and the boot menu

You can use either PowerShell or command prompt to disable SAC and the boot menu.

Using Windows Recovery Console

When encountering issues where the Windows operating system is unable to start or is experiencing repeated unexpected restarts, utilizing the Windows Recovery Environment (WinRE) is an effective solution. This environment is designed to assist in troubleshooting and rectifying common problems such as disk corruption, missing or damaged system files, or unfinished updates.

WinRE automatically engages under certain conditions, including:

  • Two unsuccessful attempts to boot Windows.
  • Two abrupt shutdowns following boot completion within a two-minute interval.
  • Two restarts occurring within two minutes of booting.
  • Errors related to Secure Boot (excluding Bootmgr.efi issues).
  • BitLocker errors on devices only supporting touch.
  • Accessing manually
Important

To use Windows Recovery Console, you must first set up a VNC console. For details, see: Connecting to the VNC Console.
Tip

The Windows Recovery Partition is a dedicated section of a boot disk that contains essential tools and resources for troubleshooting, repairing, and recovering the Windows operating system. The Windows recovery console is located on the Windows recovery partition.

Security Considerations

When working with Windows recovery console, be aware of these security considerations:

  • You can run most tools within Windows recovery console without selecting an administrator account and entering the password.
  • When booted into the recovery environment, encrypted files won't be accessible unless the user has the key to decrypt the volume.
  • By default, networking is disabled in Windows recovery console. You can turn on networking when you need it. For better security, disable networking when you don't need connectivity.