Upload Logs on Demand
If you want to ingest log files into Oracle Logging Analytics without continuously collecting them using the Management Agent, then you can perform an on-demand upload. You can perform as many on-demand uploads as necessary to upload any logs that you want to analyze.
The following are features of on-demand upload:
-
You can upload a single raw log file or any archive file (
.zip
,.gz
,.tgz
,.tar
) containing multiple log files. The number of files inside an archive should be less than 2000, including directories, if any. -
The maximum file size for a single upload (single file or a ZIP file) is 1 GB. The uncompressed size of the file should be less than 10 GB.
-
You can name each upload for easy reference. Reusing the name, you can upload files at different times to the same upload name.
-
There is a limit of 10000 on the number of unique upload names allowed per tenancy in a region.
-
Additional metadata can be attached with each log record by providing a metadata file along with the log data.
-
The upload configuration information and its corresponding processing status is available for 90 days.
Prerequisites: Before beginning to upload log data on-demand, ensure to collect the following information:
- Log source name of a new log source or use an
Oracle-defined log source that matches your log format. See
Oracle-defined Sources and Create a Source.
To perform on-demand upload, the source should be of the type System Event Messages (Syslog), File, or Oracle Diagnostic Logs (ODL).
- Log group OCID where you will store these logs to control the user access control to the logs. See Create Log Groups to Store Your Logs.
- Optionally, if you want to map the logs that you are uploading, then the entity OCID. See Create an Entity to Represent Your Log-Emitting Resource.
Permission Required for On-Demand Upload
Topics:
- Allow Users to Perform On-Demand Upload Create, Get, and List Operations
- Allow Users to Perform On-Demand Upload Delete Operation
Allow Users to Perform On-Demand Upload Create, Get, and List Operations
You can enable the users to perform on-demand upload Create, Get, and List operations by selectively giving only the required permissions to perform those on-demand upload tasks, giving individual resource-type permissions, or giving broader aggregate level permissions. So you can select any one of the three sets of policy statements provided below, as suitable for your use-case.
The following IAM policy statements are for providing the specific permissions to the user group for create, get, and list operations during on-demand upload:
allow group <group_name> to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <log_group_compartment>
allow group <group_name> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <entity_compartment>
allow group <group_name> to {LOG_ANALYTICS_SOURCE_READ} in tenancy
allow group <group_name> to use loganalytics-ondemand-upload in tenancy
The following IAM policy statements are for providing permissions at the level of individual resource-types to use on-demand upload:
allow group <group_name> to use loganalytics-ondemand-upload in tenancy
allow group <group_name> to use loganalytics-log-group in compartment <log_group_compartment>
allow group <group_name> to read loganalytics-source in tenancy
allow group <group_name> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <entity_compartment>
On the other hand, the following IAM policy statements are for providing permissions at Oracle Logging Analytics aggregate resources level to use on-demand upload:
allow group <group_name> to use loganalytics-features-family in tenancy
allow group <group_name> to use loganalytics-resources-family in tenancy/compartment
group_name
in all the above policy statements
refers to the user group that must be given the required permissions.
Allow Users to Perform On-Demand Upload Delete Operation
You can enable the users to perform on-demand upload Delete operation by selectively giving only the required permissions to perform the on-demand upload delete task, giving individual resource-type permissions, or giving broader aggregate level permissions. So you can select any one of the three sets of policy statements provided below, as suitable for your use-case.
The following IAM policy statements are specifically for providing the permission to the user group for the delete operation:
allow group <group_name> to use loganalytics-ondemand-upload in tenancy
allow group <group_name> to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in compartment <log_group_compartment>
allow group <group_name> to {LOG_ANALYTICS_QUERY_VIEW} in tenancy
allow group <group_name> to {COMPARTMENT_QUERY} in tenancy
The following IAM policies statements are for providing permissions at the level of individual resource-types for on-demand upload delete operation:
allow group <group_name> to use loganalytics-ondemand-upload in tenancy
allow group <group_name> to manage loganalytics-log-group in compartment <log_group_compartment>
allow group <group_name> to read loganalytics-query in tenancy
allow group <group_name> to read compartments in tenancy
The following IAM policies statements are for providing permissions at Oracle Logging Analytics aggregate resources level for on-demand upload delete operation:
allow group <group_name> to use loganalytics-features-family in tenancy
allow group <group_name> to manage loganalytics-resources-family in tenancy/compartment
allow group <group_name> to read compartments in tenancy
group_name
in all the above policy statements refers
to the user group that must be given the required permissions.
On-Demand Upload Using Console
You can upload your log files using the On-Demand Upload (ODU) wizard that's available on the service console of Oracle Logging Analytics.
On-Demand Upload Using CLI
You can use the OCI CLI to upload your log files to Oracle Logging Analytics through a command-line interface. This simple interface enables you to automate your uploads by integrating the CLI into your application.
For information about using the CLI, see Command Line Interface (CLI).
For a complete list of flags and options available for CLI commands, see Command Line Reference: Logging Analytics - Upload.
Run the following CLI commands to manage loganalytics-ondemand-upload:
-
Upload Log File:
oci log-analytics upload upload-log-file --namespace-name <namespace_name> --log-source-name <log-source-name> --upload-name <upload-name> --filename <file_name> --opc-meta-loggrpid <opc-meta-loggrpid> --file <path_to_log_file>
Sample response of the above command:
{ "data": { "name": null, “reference” : “32817130200562135", “timeCreated” : “2020-06-01T12:00:00.000Z”, "time-earliest-log-entry": null, "time-latest-log-entry": null, "time-updated": null, "warnings-count": null } }
-
Delete Upload:
oci log-analytics upload delete --namespace-name <namespace_name> --upload-reference <upload-reference>
-
List Uploads:
oci log-analytics upload list --namespace-name <namespace_name>
-
Get Upload:
oci log-analytics upload get --namespace-name <namespace_name> --upload-reference <upload-reference>
-
List Upload Files:
oci log-analytics upload list-upload-files --namespace-name <namespace_name> --upload-reference <upload-reference>
-
Delete Upload File:
oci log-analytics upload delete-upload-file --namespace-name <namespace_name> --upload-reference <upload-reference> --file-reference <file-reference>
Verify an On-Demand Upload Using Console
Upon completing an on-demand upload of the log data, you can view the summary of uploads and verify the file status.
- Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.
- On the left panel, under Resources, click Uploads. This displays the latest on-demand uploads.
- To view the processed data of the upload, click on the menu icon corresponding to the upload > select View in Log Explorer.
- To view the list of files in the upload along with their processing status, click the upload name.
Delete Uploaded Log Files Using Console
Upon completing an on-demand upload of the log data, you can view the summary of uploads and verify the file status. If, in case you notice that the file upload failed or you no longer want to keep the files for this upload, then you can delete them.
Add Additional Metadata to the Upload
Create a metadata json file and name it
uploads_metadata.json
. Following is an example metadata file:
{
"field1":"value1",
"field2":"value2"
}
In the above file field1 and field2 are Oracle-defined or user defined fields. See Create a Field. All these fields along with the values provided will be added to each parsed log record.
Create a zip archive file containing the actual log data in one of the
supported file formats, and uploads_metadata.json
file. The
uploads_metadata.json
file must be located in the top hierarchy of
the zip.