Oracle Cloud Infrastructure Documentation

Transferring Data To and From File Storage

Many common use cases for Oracle Cloud Infrastructure (OCI) File Storage include the transfer of a large amount of data. Based on the origination, destination, and the direction of the data transfer, the best method to accomplish that transfer can vary.

The following table provides recommendations for common File Storage data transfer scenarios, including the migration of on-premises data, copying File Storage data from one region to another, copying File Storage data within a region, and copying File Storage data to Object Storage.

For general information about private connections between OCI and on-premises data, see FastConnect and Site-to-Site VPN.

File Storage Data Transfer Scenarios
Transfer Data From... To... Recommended Method Prerequisites and Considerations
On-premises storage File Storage on OCI

Linux users can use instance-to-instance streaming and the fpsync tool.

Windows users can mount the Windows on-premises file share as a Common Internet File System (CIFS) share on an OCI Oracle Linux instance and use a copy tool such as fss-parallel-tools or fpsync to transfer data.

For more information, see Transferring On-Premises Data to File Storage.

Ensure that network connectivity is established between source and destination instances.
OCI File Storage File Storage in another region

Use one of the following methods, depending on the specifics of the use case:

  1. Use File System Replication. See Using Replication for Data Transfer for more information.

  2. Use instance-to-instance streaming and the fpsync tool. For more information, see Using Instance-to-Instance Streaming to Transfer File Storage Data.

If using replication, see replication's Limitations and Considerations.

If using instance-to-instance streaming, ensure that network connectivity is established between source and destination instances.
OCI File Storage File Storage within the same availability domain

Use one of the following methods, depending on the specifics of the use case:

  1. Use File System Replication. See Using Replication for Data Transfer for more information.

  2. Use the parcp command in our suite of parallel tools to transfer data.

    See the parcp usage examples in Using File Storage Parallel Tools, particularly how to use parcp as an effective alternative for rsync.

If using replication, see replication's Limitations and Considerations.

If using parcp, ensure that both source and destination file systems are mounted in the instance.
OCI File Storage OCI Object Storage

Use rclone. See Backing Up Snapshots to Object Storage Using rclone for more information.

Instance should be able to connect to the Object Storage bucket.

Transferring On-Premises Data to File Storage

The following scenarios have been proven effective for Linux and Windows users when transferring large amounts of data from on-premises storage to OCI File Storage. They might not be applicable to all environments or meet all requirements.

For Linux Users

Use the fpsync tool to perform an initial copy of on-premises data to OCI File Storage. Then, incremental data changes can be synchronized using rsync because fpsync can't delete files and folders in the destination that don't exist in the source.

The fpsync tool is a parallel wrapper of rsync. Linux users can download fpsync from a yum repository. The commands differ depending on the version of Linux.

  1. Download from the repository.

    Linux 7 users can install the tool using the following command:

    sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

    Linux 8 users can install the tool using the following command:

    sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
  2. Install the tool:
    sudo yum install fpart -y

Before beginning the data transfer, complete the following prerequisites:

  • Ensure that network connectivity is established between the on-premises data source and OCI. Use FastConnect or Site-to-Site VPN connection to enable fast instance-to-instance streaming over SSH.
  • Create an Oracle Linux instance in OCI.
  • Attach or mount the on-premises storage share on a Linux server. A dedicated instance is recommended.

In this scenario, we suggest that the initial copy uses fpsync. Later, incremental syncs use rsync because fpsync doesn't have the --delete option.

  1. Mount the file system in the Oracle Linux instance.
  2. Run the following command from the on-premises Linux server where the source share is attached or mounted to perform the initial copy: 
    fpsync -vv -n `nproc` -f 2000 -o "-arxH --progress --log-file fpsync.log -e ssh" /<source>/ <user>@<oci_linux_instance>:/file_storage_destination/
  3. (Optional) If you need to run an incremental sync until a specific date, you can schedule the following rsync command as a cron job: 
    rsync -arxH --delete --progress --log-file rsync.log -e ssh" /source/ <user>@<oci_linux_instance>:/file_storage_destination/
For more fpsync options, see the fpsync man page.

For Windows Users

Don't use Windows copy utilities for large data copies to File Storage. Instead, use an OCI Linux instance to perform large copies using the parallel tools that come with an OCI Linux instance. Mount the Windows share as a Common Internet File System (CIFS) share and use a copy tool such as parcp or fpsync to transfer data.

Because the SMB protocol and the NFS protocol used by File Storage aren't compatible, an instance that can mount both the NFS file system and the SMB share is used to create a bridge between them.

  1. Identify or create a Linux instance in Oracle Cloud Infrastructure that has network access to both the File Storage file system and the Windows SMB share.
  2. Open a terminal on the instance.

  3. Type the following to install the Common Internet File System (CIFS) utility and verify its installation:

    $sudo yum install -y cifs-utils
$rpm -qa|grep cifs

  4. Mount the Windows SMB share as a CIFS share.

    1. Create a mount point directory. For example:

      sudo mkdir /mnt/win_share

    2. Mount the CIFS share:

      sudo mount -t cifs -o username=<win_share_user> //<win_IP_address/hostname>/<share_name> /mnt/<win_share_dir>

      For example: 

      $sudo mount -t cifs -o username=opc //win2012/c /mnt/win_share
  5. Copy the files from the CIFS share to the mounted file system using the parcp utility from the File Storage Parallel Tools suite or fpsync. For installation information about parcp, see Using File Storage Parallel Tools. For information about fpsync, visit the fpsync man page.

    For example:

    parcp -P 32 /mnt/win_share /mnt/MyFilesystem
    fpsync -v -n 8 -f 101 -o "-lptgoD -v --numeric-ids -e ssh" /mnt/win_share /mnt/MyFilesystem

Using Instance-to-Instance Streaming to Transfer File Storage Data

The fpsync tool is a parallel wrapper of rsync. You can use fpsync and instance-to-instance streaming to transfer data between mounted File Storage file systems.

To install fpsync, enable the Oracle Linux developer repository, which includes the fpsync utility, on the OCI instance using a command such as the following. The command differs based on the version of Oracle Linux in use:

yum --enablerepo ol7_developer_EPEL install -y fpart
yum --enablerepo ol8_developer_EPEL install -y fpart

After installing the tool, use an instance-to-instance streaming command such as this to stream data:

fpsync -o "-e ssh --progress" /<fss_src_region>/test <ssh_user>@<remote_ip>:/<fss_dest_region>/

For more information and options, see the fpsync man page.

An example showing the performance difference between the two approaches follows:

# date; time fpsync -o "-e ssh --progress --log-file ~/speedtest.log" /fss_src_bom/test root@DR_hyd:/fss_dest_hyd/ ; date
Sun Mar 13 15:22:58 GMT 2022

real 0m1.467s
user 0m0.111s
sys 0m0.075s
Sun Mar 13 15:23:00 GMT 2022

# ls -ltrd test
drwxr-xr-x. 2 root root 1 Mar 13 15:22 test
# du -sh test
1001M test
# cp -r test test1

# date; time fpsync -o "--progress --log-file ~/speedtest1.log" /fss_src_bom/test1 /fss_dest_hyd/ ; date
Sun Mar 13 15:25:16 GMT 2022

real 1m28.847s
user 0m3.688s
sys 0m1.439s
Sun Mar 13 15:26:44 GMT 202