Configuring the Cache for File Systems

Configure the cache for file systems using Storage Gateway.

Storage Gateway caches frequently retrieved data on the local host, minimizing the number of REST API calls to Storage Gateway and enabling faster data retrieval. You configure the cache for a file system when you create the file system. See Creating Your First Storage Gateway File System and Adding a Storage Gateway File System.

About File System Cache

The file system cache serves as both a read cache and a write buffer for data storage and retrieval. The read cache contains frequently retrieved data that’s accessible locally for read operations. The write buffer contains data that has been copied to the disk cache and queued for upload to your Oracle Cloud Infrastructure tenancy.

When you retrieve data from Oracle Cloud Infrastructure, the data is stored in the Storage Gateway read cache. The read cache allows subsequent I/O operations to that data at local disk speed.

When the read cache is full or reaches the configured limit, Storage Gateway removes files from the cache based on a least recently used (LRU) algorithm. Files pending upload to your tenancy are not removed from cache. You can also preserve files that you do not want removed from cache.

For more information on how to preserve files in the read cache, see Preserving Files in the File System Cache.

When an application transfers files through an NFS share, the files are written to the write buffer. The write buffer can contain many files that are queued and pending upload. If the host on which Storage Gateway is installed fails or Storage Gateway stops abruptly, the pending upload operations persist on the local disk. When Storage Gateway restarts, the pending upload operations resume and the data is uploaded to Oracle Cloud Infrastructure.

Configuring Local Storage for File Systems and Cache

Storage Gateway uses local storage attached to the server (or virtual server) for hosting the file systems and cache. Files written to a file system in Storage Gateway are uploaded to the associated Object Storage bucket, with a portion of the file set maintained locally in the file system as a warm cache.

For optimal performance, reliability, and fault tolerance, follow these guidelines when configuring the local Storage Gateway storage:

Allocate dedicated local storage for each Storage Gateway file system, and associated metadata and logs.
Multiple disks (hard disk drives or solid-state drives) in a RAID10 set provide an optimal balance of performance, reliability, and fault tolerance. Alternatively, you can use RAID6.

Important

Avoid RAID0 or single disk (no RAID) because of the potential for data loss upon disk failure.
Provision sufficient space to accommodate the read cache and the write buffer (for ingesting new files) without ever becoming more than 80% full.

In general, provision file system cache storage that is at least 1.5 times the size of the file set that you want to hold in the read cache. For example, assume that the entire file set requires 50 TB of space. You expect frequent access to 10% (5 TB) of that file set. Ensure that the file system cache storage has at least 7.5 TB of usable capacity. If the cache size reaches a near-full threshold, any data ingestion results in an out of space error in Storage Gateway.
When you provision local storage at installation time, Oracle recommends that you configure the read cache to be equal to the file system cache size minus the desired write buffer size. If the file system cache is less than 300 GB, Storage Gateway generates a warning message.

Determining File System Cache Size

The Storage Gateway file system cache serves as both a read cache and a write buffer. You can specify the maximum size of the read cache. The write buffer uses any remaining available space in the file system cache. You do not explicitly specify a size for the write buffer.

Important

Oracle recommends that you configure the read cache to be equal to the file system cache size minus the desired write buffer size.

Read Cache Size

The default maximum read cache size is 300 GB if the file system cache size is greater than 300 GB. Changing the default maximum read cache size is optional. The appropriate size depends on Storage Gateway workload. While the default setting for the read cache is appropriate for most workloads, consider increasing the size if Storage Gateway must retrieve a significant amount of data from the cloud.

Use the following guidelines to determine the appropriate setting for the read cache size:

Do not set the read cache maximum equal to the size of the file system cache. Doing so allocates 100% of the space for the read cache and leaves no capacity for ingesting new files. If there is no available space for new file ingestion, Storage Gateway stops ingesting data and begins evicting files from the read cache to create space. Always preserve some space in the file system cache for ingestion.
Set the read cache size to equal the amount of data that you anticipate to be accessed frequently, while leaving enough capacity for the write buffer.

Note

If the total file system cache size is less than 300 GB, Storage Gateway automatically sets the read cache maximum size to 20% of the file system cache. The system does not honor custom read cache configuration settings for a file system cache less than 300 GB.

After you calculate the optimal file system cache size, you can configure the read cache when creating the file system or adjust it after monitoring the workload. See Adding a Storage Gateway File System and Changing the Properties of a Storage Gateway File System.

Write Buffer Size

Optimizing the space available for the write buffer is an important part of determining the appropriate file system cache size. The write buffer size increases when data is ingested in Storage Gateway and decreases after the data is uploaded to the cloud.

Important

When the write buffer uses all available file system cache space, further data ingestion is blocked until a portion of the existing files are uploaded and evicted from the cache.
Oracle recommends that you allow a minimum of 300 GB for the write buffer under any circumstances.

Use the following guidelines to determine the space needed for the write buffer:

Identify the amount of data to be uploaded in Storage Gateway. If a large amount of data is uploaded, the Storage Gateway write buffer can reach its maximum size. Exceeding the write buffer leads to I/O failure as the file system cache has no space available. If you cannot regulate data ingestion, you can increase the file system cache space to avoid I/O failure. You can regulate I/O by pausing after a certain amount of data is ingested or by periodically allowing uploads to complete before ingesting more data. For example, you can use this approach for backups run as cron jobs when the file system cache space is less than the amount of data to be ingested.
Calculate the amount of data that is ingested on any typical day or week in Storage Gateway. Also, calculate the amount of data that is uploaded over a time period, based on the available bandwidth or historical data. Ensure that the difference between these calculations does not exceed the write buffer size.
Some applications can handle I/O failure, and then resume writing data. In this case, consider setting the cache size to the amount of data that you’d like the application to tolerate before the cache space can be reclaimed.

As stated earlier, you want to avoid completely filling the file system cache. The write buffer grows by the difference between the ingest rate and the upload rate. The file system cache size must be larger than the read cache plus the total number of bytes buffered at any point during the job. If you have workloads that upload large amounts of data in parallel, you can use the following equation to determine the amount of space needed by the write buffer.

WB >= D * (1 - UR/IR) + E

WB = Recommended write buffer size

D = Total uploaded dataset size

UR = Upload rate (The upload rate is the lesser of the actual upload rate or the disk read speed.)

IR = Ingestion rate (disk write speed)

E = Extra margin (Oracle recommends at least 50 GB.)

Note

This equation applies only if the upload rate is less than the ingest rate.

Run the following command to measure your system's ingestion rate:

sudo docker exec -it ocisg python /opt/oracle/gateway/python/packages/ocisg_helper/disk_speed_test.py

Example output:

Write speed:
2.1 GB copied in 9.4 seconds (228 MB/s)
Read speed:
2.1 GB copied in 6.8 seconds (315 MB/s)

If read and writes occur in parallel during the job, the read and write speeds are about 50% of the returned values.

Run the following command to measure your system's upload rate:

sudo docker exec ocisg cat /mnt/gateway/cache-phoenix/:::diag:oci-network-speed-test

Example output:

Average Upload Speed = 125 MB/s

Example 1

Daily dataset is 500 GB

Upload rate = 2 MB/s

Disk read = 600 MB/s

Ingestion rate (Disk write) = 600 MB/s

Apply the equation:

WB >= 500 GB * (1 - (2/600)) + 50 GB

WB >= 549 GB

In this case, the upload rate is very slow compared to the ingestion rate, so the entire dataset needs to fit in the write buffer.

Example 2

Daily dataset is 1 TB

Upload rate = 300 MB/s

Disk read = 400 MB/s

Ingestion rate (Disk write) = 100 MB/s

In this case, we recommend the minimum write buffer size of 300 GB, since the upload rate is higher than the ingestion rate.

Example 3

Daily dataset is 5 TB

Upload rate = 250 MB/s

Disk read = 400 MB/s

Ingestion rate (Disk write) = 300 MB/s

Apply the equation:

WB >= 5 TB * (1 - (250/300)) + 50 GB

WB >= 0.88 TB

WB is 880 GB.

Note

Storage Gateway begins throttling I/O when the free cache space falls below 15 GB. Determine if the application is able to handle the throttling or if you want to provision more cache space.

Preserving Files in the File System Cache

When you write a file to your file system, the file is initially stored in the file system cache, and then uploaded to your Oracle Cloud Infrastructure tenancy. After a file has been uploaded, the cache manager can remove the file from the file system cache. To meet the cache threshold specified for the file system, cache is reclaimed using the Least Recently Used (LRU) cache management policy. If you want specific files to be available in the cache for quick access, you can pin the files to the file system cache. Once pinned, files are not removed from the file system cache until you explicitly unpin them. You can view the Maximum Read Cache Size in GiB for a selected file system in the management console under Settings.

You can pin files connected to both Standard and Archive storage tiers to file system cache. Files that you write to a file system are always uploaded to your tenancy, regardless of whether the files are pinned to the cache.

If the file that you want to pin to cache is not present in the cache, the file is automatically downloaded to the cache if the file system is connected to a Standard storage tier. If that file belongs to a file system connected to an Archive storage tier, you must first restore the file before the file can be downloaded to the cache. See Restoring Storage Gateway Files and Objects from Archive Storage for details.

Important

By default, the cache pinning feature is enabled on all file systems.
When selecting the files for cache pinning, consider the overall cache threshold and calculate the residual cache space that remains available for normal cache operations. For example, assume that your cache threshold is 1 TB and you estimate that the files you want to pin to cache occupy 300 GB. That leaves 700 GB of usable space in your cache after pinning the files.
When you restore a file from the Archive storage tier, the file moves to the Standard storage tier. The file remains in Standard storage for 24 hours or the retention duration you specify. The continued availability of the file in the cache depends on the LRU operation. However, if you pin such a file to the cache, the restored file remains in the cache until you unpin the file.

Enabling and Managing Cache Pinning

To perform cache pinning operations for a file system, run the following command from the NFS client on which the file system is mounted:

cat /path/to/mountpoint/<file_path>:::cache:cache_command[:argument]

The following table lists the cache pinning operations and the corresponding command and argument for each operation:


Operation	Cache Command	Argument
Enable cache pinning for a file system. By default, cache pinning is enabled for all file systems.	`set-preserve-option`	`true`
Get the cache pinning status for a file system.	`get-preserve-option`	No argument
Disable cache pinning for a file system.	`set-preserve-option`	`false`
List the files that are pinned to the cache.	`list-preserve`	No argument
Remove deleted files from the preserve list.	`list-preserve-update`	No argument
Add a file to the preserve list.	`add-preserve`	No argument
Remove a file from the preserve list.	`remove-preserve`	No argument
Clear the preserve list.	`clear-preserve`	No argument

Example Commands

To enable cache pinning for the myFS file system:

cat /mnt/gateway/myFS/:::cache:set-preserve-option:true

To get the cache pinning status for myFS:
```
cat /mnt/gateway/myFS/:::cache:get-preserve-option
```
If cache pinning is enabled for the file system, the output of this command is true. Otherwise, the output is false.

To disable cache pinning for the myFS file system:

cat /mnt/gateway/myFS/:::cache:set-preserve-option:false

To add a file myFile of the myFS file system to the preserve list:
```
cat /mnt/gateway/myFS/myFile:::cache:add-preserve
```
To find out which files are added to the preserve list of the myFS file system:
```
cat /mnt/gateway/myFS/:::cache:list-preserve
```
Sample output of the preceding command:
```
["/doNotDelete.txt", "/myFileMetadata", "/myFile"]
```

To remove the file myFile from the preserve list

cat /mnt/gateway/myFS/myFile:::cache:remove-preserve

To update the preserve list when the output of the cache:list-preserve command indicates that a pinned file has been removed from the file system:
```
cat /mnt/gateway/myFS/:::cache:list-preserve-update
```
Sample of the original preserve list:
```
["/doNotDelete.txt", "/myFileMetadata"]
```
Output of the cache:list-preserve command after the file myFileMetadata is removed from the cache:
```
["/doNotDelete.txt", "Status: 1 files appear to no longer exist. Please run list-preserve-update"]
```
Output of the cache:list-preserve-update command:
```
["/doNotDelete.txt"]
```

To clear the preserve list for a file system:

cat /mnt/gateway/myFS/:::cache:clear-preserve

Oracle Cloud Infrastructure Documentation