Distributed Tracing for Functions
Find out how to enable tracing and view function traces when debugging with OCI Functions.
When a function is invoked but doesn't run or perform as expected, you need to investigate the issue at a detailed level. The distributed tracing feature observes the function's execution as it moves through the different components of the system. You can trace and instrument standalone functions to debug execution and performance issues. You can also use function tracing to debug issues with complete serverless applications comprising multiple functions and services, such as:
- a function calling another function
- a function calling other services such as the Object Storage service
- a function that serves as a backend for an API gateway deployed in the API Gateway service
- a function triggered in response to an event by the Events service, Notifications service, or Connector Hub
The OCI Functions tracing capabilities are provided by the Oracle Cloud Infrastructure Application Performance Monitoring service. Features in Application Performance Monitoring (APM) enable you to identify and troubleshoot failures and latency issues in the functions you create and deploy.
In the Application Performance Monitoring service:
- An APM domain contains the systems monitored by Application Performance Monitoring. An APM domain is an instance of a collector of trace and span data which stores, aggregates, displays, and visualizes the data.
- A trace is the complete flow of a request as it passes through all the components of a distributed system in a given time period. It consists of an entire tree of spans all related to the same single overall request flow.
- A span is an operation or a logical unit of work with a name, start time, and duration, within a trace. A span is a time segment associated with the duration of a unit of work within the overall request flow.
The Application Performance Monitoring Trace Explorer enables you to visualize the entire request flow and explore trace and span details for diagnostics. You can view and monitor slow traces and traces with errors. To isolate and identify trace issues, you can drill down into specific spans, such as page loads, AJAX calls, and service requests. For more information about the Application Performance Monitoring service, see Application Performance Monitoring.
To enable tracing for a function, you must:
- Set up a policy to give the OCI Functions service permission to access APM domains, if the policy does not exist already (see Policy Statements to Give the OCI Functions Service and OCI Functions Users Access to Tracing Resources).
- Set up an APM domain.
- Enable tracing for the Functions application and select the APM domain you created.
- Enable tracing for one or more functions.
When you enable tracing for a function, OCI Functions automatically generates a "default function invocation span." The default span captures information about the function's execution context including the overall time taken to process the request and return a response to the caller. In addition to the default function invocation span, you can add code to functions to define custom spans. Use custom spans to capture more function-specific information to help with debugging. For example, you might define custom spans to capture the start and end of specific units of work. For example, units of work could include getting the database password from the Vault, opening a database connection, and retrieving records from the database.
Four variables have been added to the OCI Functions context that provide helpful tracing information. These variables include:
FN_APP_NAME:
The function application name.FN_FN_NAME:
The function name.OCI_TRACE_COLLECTOR_URL
: The APM domain URL with data key.OCI_TRACING_ENABLED:
Is tracing enabled?- When retrieved from environment variables, returns 0 or 1.
- When retrieved from the function context, returns
true
orfalse
as appropriate for the language used.
Required IAM Policy for Enabling Tracing
Before you can enable tracing, the group to which you belong must have permission to access existing APM domains or to create APM domains. In addition, OCI Functions must have permission to access APM domains. See Policy Statements to Give the OCI Functions Service and OCI Functions Users Access to Tracing Resources.
Using the Console to Enable Tracing and View Function Traces
A couple of steps are required to enable tracing and to view function traces for the Oracle Cloud Infrastructure Application Performance Monitoring (APM) service. First, enable tracing for the application containing the function. Then, enable tracing for one or more functions. You can then view function traces in the APM Trace Explorer.
Using the Console to Enable Tracing
To enable tracing, follow these steps.
- Sign in to the Console as a functions developer.
- Open the navigation menu and click Developer Services. Under Functions, click Applications.
- Select the region and compartment containing the Functions application.
The Applications page shows all the applications in the compartment you selected.
- Select the Functions application for which you want to enable tracing.
- To enable tracing for the application:
- Under Resources, click Traces.
- Select the Trace Enabled option and specify:
- Compartment: The compartment in which to create the trace. By default, the current compartment.
- APM Domain: The APM domain (defined in the Application Performance Monitoring service) in which to create the trace. To use an existing APM Domain, select an existing APM domain from the list. Or, to create a new APM domain, click APM Domain. For more information about APM domains, see Getting Started with Application Performance Monitoring. Note
The APM Domain needs to have both public and private data keys for function tracing to work. If the keys do not exist, you can create them through the console interface.
- Click Enable Trace to enable tracing for the application.
Having enabled tracing for the Functions application, you can now enable tracing for one or more functions in the application.
- To enable tracing for specific functions in the application:
- Under Resources, click Functions.
-
Select the Enable Trace option beside one or more function(s) for which you want to enable tracing.
The Enable Trace option is only shown if you have previously enabled tracing for the application. Note the following:
- If the Enable Trace option is not shown, you must enable tracing for the application. If you haven't already enabled tracing for the application, see the previous step.
- If you previously enabled tracing for the application but later disabled it, an Enable application tracing link is shown. Click the Enable application tracing link to re-enable tracing for the application (see the previous step). Having re-enabled tracing for the application, you can then enable tracing for specific functions.
When you have enabled tracing for the application and one or more functions, you can view function traces.
Using the Console to View Function Traces
To view the traces for functions that have tracing enabled:
- Sign in to the Console as a functions developer.
- Open the navigation menu and click Developer Services. Under Functions, click Applications.
- Select the region and compartment containing the Functions application with functions for which you want to view function traces.
The Applications page shows all the applications in the compartment you selected.
- Select the application containing the functions for which you want to view traces.
- To see traces for functions:
- To see traces for all the functions that have tracing enabled in the
application:
- Under Resources, click Traces.
- Click the name of the trace. Note
A trace name is only shown if you have already enabled tracing for the application.
- To see the trace for a specific function that has tracing enabled:
- Under Resources, click Functions.
- Click the Note
The View Trace option is only shown if you have already enabled tracing for the function.
beside the function, and then click View Trace.
The traces for the functions you selected are shown in the APM Trace Explorer. By default, a trace is shown for the default function invocation span, and any custom spans defined for the function.
- To see traces for all the functions that have tracing enabled in the
application:
- In the APM Trace Explorer:
- Click a trace to see the spans for that trace.
- Click a span to see the details captured for that span.
For more information about using the APM Trace Explorer, see Use Trace Explorer.
Tracing a Chain of Functions
By default, function tracing provides a trace for an entire function invocation. However, often with modern cloud applications, you need to chain function invocations. OCI Functions tracing provides the ability trace the execution of a function invoked by another function. This ability means you can examine the execution of each function in a chain of calls in a single tree of spans in APM trace explorer.
To trace a chain of functions, you need to propagate the X-B3 headers X-B3-TraceId
, X-B3-SpanId
, X-B3-ParentSpanId
, and X-B3-Sampled
in the function invocation request from your function code.
After the function has run, the trace data from your functions is collected and available in APM Trace Explorer. For more information about using the APM Trace Explorer, see Use Trace Explorer.
Here's an example of how you can trace a chain of functions. If you want to try you this example, you need to create two sample functions. Follow these steps to set up your functions.
- Create your tracing Python function:
fn init --runtime python <your-function-name-1>
- Create your "Hello World!" Python function:
fn init --runtime python <your-function-name-2>
- Deploy both functions:
fn -v deploy --app <app-name>
- Get the second functions OCID and invoke endpoint:
fn inspect function your-app-name your-function-name-2
- Create JSON file to pass the required information into the first function. For
example, your
test.json
file might look like this:{ "function_ocid": "ocid1.fnfunc.oc1.iad.aaaaaaaaxxxxxxxxxxx", "function_endpoint": "https://xxxxxxxxx.us-ashburn-1.functions.oci.oraclecloud.com", "function_body": "", "__comment": "Alternatively, you can set function_body to { \"name\": \"Oracle\" }" }
- When the first function is invoked, you can pass the second functions information
using
test.json
:fn invoke <app-name> <your-function-name-1> < test.json
Now you are ready to update the first function with the required code updates.
Configure Packages
Update your requirements.txt
file to include the following
packages:
fdk
oci
Save the file.
Update your Function Code to Propagate the X-B3 Headers
The Python function calls the handler
function and passes in the
JSON information from the invoke command. The handler
function is
broken in into several small blocks to simplicity. The complete source file is
provided at the bottom of this section.
Load the JSON Data
In this first part, the JSON data is loaded from the function invocation.
import io
import json
import logging
import oci
from fdk import response
def handler(ctx, data: io.BytesIO=None):
app_name = ctx.AppName()
func_name = ctx.FnName()
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler")
try:
body = json.loads(data.getvalue())
function_endpoint = body.get("function_endpoint")
function_ocid = body.get("function_ocid")
function_body = body.get("function_body")
except (Exception) as ex:
print('ERROR: Missing key in payload', ex, flush=True)
raise
Create Invoke Client and Gather Header Information
Create the Functions invoke client using the OCI Python SDK and Functions resource
principals. Then, retrieve the tracing_context
and extract the
required information to create the HTTP headers.
signer = oci.auth.signers.get_resource_principals_signer()
client = oci.functions.FunctionsInvokeClient(config={}, signer=signer, service_endpoint=function_endpoint)
#
# Zipkin X-B3- header propagation
#
tracing_context = ctx.TracingContext()
trace_id = tracing_context.trace_id()
span_id = tracing_context.span_id()
parent_span_id = tracing_context.parent_span_id()
is_sampled = tracing_context.is_sampled()
Propagate the X-B3 Headers
The OCI Python SDK lets you set custom headers. Use this technique to
pass the X-B3 headers in to the second function invocation. Header information is
passed for trace_id
, span_id
,
parent_span_id
, and is_sampled
. Finally, the
second function is invoked with client
and the response is passed
to this function's response.
# if tracing is enabled, is_sampled will be true in the tracing context
if is_sampled:
# To propagate headers in the OCI SDK in the request to the next function,
# add the X-B3- headers in the request. This header will be included in ALL
# subsequent calls made.
if trace_id is not None:
client.base_client.session.headers['X-B3-TraceId'] = trace_id
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | trace_id: " + trace_id)
if span_id is not None:
client.base_client.session.headers['X-B3-SpanId'] = span_id
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | span_id: " + span_id)
if parent_span_id is not None:
client.base_client.session.headers['X-B3-ParentSpanId'] = parent_span_id
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | parent_span_id: " + parent_span_id)
client.base_client.session.headers['X-B3-Sampled'] = str(int(is_sampled))
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | is_sampled: " + str(int(is_sampled)))
else:
# function.trace is DISABLED
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | function tracing is DISABLED")
resp = client.invoke_function(function_id=function_ocid, invoke_function_body=function_body)
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | Response: " + resp.data.text)
return response.Response(
ctx,
response_data=resp.data.text,
headers={"Content-Type": "application/json"}
)
Here is the complete source code for the sample Python function.
#
# oci-invoke-function-python version 2.0.
#
# Copyright (c) 2021 Oracle, Inc.
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
#
import io
import json
import logging
import oci
from fdk import response
def handler(ctx, data: io.BytesIO=None):
app_name = ctx.AppName()
func_name = ctx.FnName()
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler")
try:
body = json.loads(data.getvalue())
function_endpoint = body.get("function_endpoint")
function_ocid = body.get("function_ocid")
function_body = body.get("function_body")
except (Exception) as ex:
print('ERROR: Missing key in payload', ex, flush=True)
raise
signer = oci.auth.signers.get_resource_principals_signer()
client = oci.functions.FunctionsInvokeClient(config={}, signer=signer, service_endpoint=function_endpoint)
#
# Zipkin X-B3- header propagation
#
tracing_context = ctx.TracingContext()
trace_id = tracing_context.trace_id()
span_id = tracing_context.span_id()
parent_span_id = tracing_context.parent_span_id()
is_sampled = tracing_context.is_sampled()
# if tracing is enabled, is_sampled will be true in the tracing context
if is_sampled:
# To propagate headers in the OCI SDK in the request to the next function,
# add the X-B3- headers in the request. This header will be included in ALL
# subsequent calls made.
if trace_id is not None:
client.base_client.session.headers['X-B3-TraceId'] = trace_id
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | trace_id: " + trace_id)
if span_id is not None:
client.base_client.session.headers['X-B3-SpanId'] = span_id
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | span_id: " + span_id)
if parent_span_id is not None:
client.base_client.session.headers['X-B3-ParentSpanId'] = parent_span_id
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | parent_span_id: " + parent_span_id)
client.base_client.session.headers['X-B3-Sampled'] = str(int(is_sampled))
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | is_sampled: " + str(int(is_sampled)))
else:
# function.trace is DISABLED
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | function tracing is DISABLED")
resp = client.invoke_function(function_id=function_ocid, invoke_function_body=function_body)
logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | Response: " + resp.data.text)
return response.Response(
ctx,
response_data=resp.data.text,
headers={"Content-Type": "application/json"}
)
Adding Custom Spans to Functions
With function tracing enabled, the default function invocation span provides a trace for the entire function invocation. The default span can provide good information, but when investigating your code you might want to dig deeper. Custom spans are added directly to your code and allow you to define spans for a method or a block of code. The resulting data provides a better picture of your function as it runs.
Before you can use custom spans, you must enable tracing for your application and functions using the Oracle Cloud Infrastructure Application Performance Monitoring (APM) service. To set up tracing, you must:
- Set up a policy to give the OCI Functions service permission to access APM domains, if the policy does not exist already (see Policy Statements to Give the OCI Functions Service and OCI Functions Users Access to Tracing Resources).
- Set up an APM domain.
- Enable tracing for the Functions application and select the APM domain you created.
- Enable tracing for one or more functions.
These steps have already been covered. However, a couple more things are required for custom spans:
- Select a distributed tracing client library, for example Zipkin.
- Add client libraries to your function dependencies.
- In your function code, use the
OCI_TRACING_ENABLED
function context variable to check if tracing is enabled. - In your function code, use the
OCI_TRACE_COLLECTOR_URL
function context variable to send your custom spans to your APM domain. - Add instrumentation to your function code.
To use custom spans, you must have the following minimum versions of the Fn Project FDKs:
- Java FDK: 1.0.129
- Python FDK: 0.1.22
- Node FDK: 0.0.20
Here's an example of how to use Zipkin to add custom spans to your Java function. If you want to try you this example, you can create a Java "Hello World!" function and add custom span code. To create a sample function:
- Create a Java function:
fn init --runtime java apm-fn-java
- For simplicity, remove the
src/test
directory.
Configure Maven
Add the following dependencies to the <dependencies> section of your Maven
pom.xml
file.
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-sender-urlconnection</artifactId>
<version>2.16.3</version>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
<version>2.16.3</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave</artifactId>
<version>5.13.3</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-core</artifactId>
<version>4.13.6</version>
</dependency>
Save the file.
The HandleRequest Method
Observations about the method follow the handleRequest
source
code.
package com.example.fn;
import brave.Span;
import brave.Tracer;
import brave.Tracing;
import brave.propagation.*;
import brave.sampler.Sampler;
import com.fnproject.fn.api.tracing.TracingContext;
import com.github.kristofa.brave.IdConversion;
import zipkin2.reporter.Sender;
import zipkin2.reporter.brave.AsyncZipkinSpanHandler;
import zipkin2.reporter.urlconnection.URLConnectionSender;
public class HelloFunction {
Sender sender;
AsyncZipkinSpanHandler zipkinSpanHandler;
Tracing tracing;
Tracer tracer;
String apmUrl;
TraceContext traceContext;
public String handleRequest(String input, TracingContext tracingContext) {
try {
intializeZipkin(tracingContext);
// Start a new trace or a span within an existing trace representing an operation
Span span = tracer.newChild(traceContext).name("MainHandle").start();
System.out.println("Inside Java Hello World function");
try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
method1();
method2();
method3();
} catch (RuntimeException | Error e) {
span.error(e); // Unless you handle exceptions, you might not know the operation failed!
throw e;
} finally {
span.finish(); // note the scope is independent of the span. Always finish a span.
tracing.close();
zipkinSpanHandler.flush();
}
} catch (Exception e) {
return e.getMessage();
}
return "Hello, AppName " + tracingContext.getAppName() + " :: fnName " + tracingContext.getFunctionName();
}
- The
TracingContext tracingConext
object passes in all the APM-related information needed to make connections to the APM service. - The
intializeZipkin
method is called which updates thetracingContext
and creates atracer
object which is used to set up custom spans. - A
span
is created for the parent custom span. Then three methods are called in the scope of the parent span. - Notice in the
finally
block all the tracing objects are closed out.
The initializeZipkin Method
Observations about the intializeZipkin
method follow the source
code.
public void intializeZipkin(TracingContext tracingContext) throws Exception {
System.out.println("Initializing the variables");
apmUrl = tracingContext.getTraceCollectorURL();
sender = URLConnectionSender.create(apmUrl);
zipkinSpanHandler = AsyncZipkinSpanHandler.create(sender);
tracing = Tracing.newBuilder()
.localServiceName(tracingContext.getServiceName())
.sampler(Sampler.NEVER_SAMPLE)
.addSpanHandler(zipkinSpanHandler)
.build();
tracer = tracing.tracer();
tracing.setNoop(!tracingContext.isTracingEnabled());
traceContext = TraceContext.newBuilder()
.traceId(IdConversion.convertToLong(tracingContext.getTraceId()))
.spanId(IdConversion.convertToLong(tracingContext.getSpanId()))
.sampled(tracingContext.isSampled()).build();
}
- The
traceContext
is passed in to create all the objects used to create custom spans. - The
apmURL
is retrieved from thegetTraceCollectorURL()
method. The URL is the endpoint to the APM domain and is used to create thetracer
object which builds the custom spans. - A builder takes the
zipkinSpanHandler
and the service name to create atracer
object. Thistracer
object is used to create custom spans.
Creating Custom Spans
With the tracer
object initialized, custom spans can be created.
public void method1() {
System.out.println("Inside Method1 function");
TraceContext traceContext = tracing.currentTraceContext().get();
Span span = tracer.newChild(traceContext).name("Method1").start();
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
span.finish();
}
}
- The
method1
method creates a custom span named "Method1."
Here is the complete source code for the sample Java tracing function.
package com.example.fn;
import brave.Span;
import brave.Tracer;
import brave.Tracing;
import brave.propagation.*;
import brave.sampler.Sampler;
import com.fnproject.fn.api.tracing.TracingContext;
import com.github.kristofa.brave.IdConversion;
import zipkin2.reporter.Sender;
import zipkin2.reporter.brave.AsyncZipkinSpanHandler;
import zipkin2.reporter.urlconnection.URLConnectionSender;
public class HelloFunction {
Sender sender;
AsyncZipkinSpanHandler zipkinSpanHandler;
Tracing tracing;
Tracer tracer;
String apmUrl;
TraceContext traceContext;
public void intializeZipkin(TracingContext tracingContext) throws Exception {
System.out.println("Initializing the variables");
apmUrl = tracingContext.getTraceCollectorURL();
sender = URLConnectionSender.create(apmUrl);
zipkinSpanHandler = AsyncZipkinSpanHandler.create(sender);
tracing = Tracing.newBuilder()
.localServiceName(tracingContext.getServiceName())
.sampler(Sampler.NEVER_SAMPLE)
.addSpanHandler(zipkinSpanHandler)
.build();
tracer = tracing.tracer();
tracing.setNoop(!tracingContext.isTracingEnabled());
traceContext = TraceContext.newBuilder()
.traceId(IdConversion.convertToLong(tracingContext.getTraceId()))
.spanId(IdConversion.convertToLong(tracingContext.getSpanId()))
.sampled(tracingContext.isSampled()).build();
}
public String handleRequest(String input, TracingContext tracingContext) {
try {
intializeZipkin(tracingContext);
// Start a new trace or a span within an existing trace representing an operation
Span span = tracer.newChild(traceContext).name("MainHandle").start();
System.out.println("Inside Java Hello World function");
try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
method1();
method2();
method3();
} catch (RuntimeException | Error e) {
span.error(e); // Unless you handle exceptions, you might not know the operation failed!
throw e;
} finally {
span.finish(); // note the scope is independent of the span. Always finish a span.
tracing.close();
zipkinSpanHandler.flush();
}
} catch (Exception e) {
return e.getMessage();
}
return "Hello, AppName " + tracingContext.getAppName() + " :: fnName " + tracingContext.getFunctionName();
}
public void method1() {
System.out.println("Inside Method1 function");
TraceContext traceContext = tracing.currentTraceContext().get();
Span span = tracer.newChild(traceContext).name("Method1").start();
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
span.finish();
}
}
public void method2() {
System.out.println("Inside Method2 function");
TraceContext traceContext = tracing.currentTraceContext().get();
Span span = tracer.newChild(traceContext).name("Method2").start();
try {
Thread.sleep(400);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
span.finish();
}
}
public void method3() {
System.out.println("Inside Method3 function");
TraceContext traceContext = tracing.currentTraceContext().get();
Span span = tracer.newChild(traceContext).name("Method3").start();
try {
Thread.sleep(300);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
span.finish();
}
}
}
Here's an example of how to use Zipkin to add custom spans to your Python function. If you want to try you this example, you can create a Python "Hello World!" function and add custom span code. To create a sample function:
- Create a Python function:
fn init --runtime python apm-fn-python
Configure Packages
Update your requirements.txt
file to include the following
packages:
fdk
requests
py_zipkin
Save the file.
Creating Handler Class and Parent Custom Span
The Python function calls the handler
function and passes in the
function context to create custom spans.
def handler(ctx, data: io.BytesIO = None):
tracing_context = ctx.TracingContext()
with zipkin_span(
service_name=tracing_context.service_name(),
span_name="Customer Code",
transport_handler=(
lambda encoded_span: transport_handler(
encoded_span, tracing_context
)
),
zipkin_attrs=tracing_context.zipkin_attrs(),
encoding=Encoding.V2_JSON,
binary_annotations=tracing_context.annotations()
):
name = "World"
try:
body = json.loads(data.getvalue())
name = body.get("name")
except (Exception, ValueError) as ex:
logging.getLogger().info('error parsing json payload: ' + str(ex))
logging.getLogger().info("Inside Python Hello World function")
time.sleep(0.005)
example(ctx)
return response.Response(
ctx, response_data=json.dumps(
{"message": "Hello {0}".format(name)}),
headers={"Content-Type": "application/json"}
)
- The
tracing_context
is passed from the function context and contains all the information needed to create and configure custom spans.Note
If tracing is not enabled, the tracing context is an empty object. With an empty tracing context, theis_sampled
flag is set toNone
andpy_zipkin
does not emit spans. - The
with zipkin_span
statement is used to create spans.- The information in
tracing_context
is used to get theservice_name
, call thetransport_handler
, and set thezipking_attrs
. - A custom span name is specified just by setting
span_name
. - Tracing attributes required for Zipkin are retrieved from the tracing
context:
tracing_context.zipkin_attrs()
.
- The information in
- With the custom span setup, the main block runs boilerplate "Hello World!" code.
With the only exception, a call to the
example
function.
The transport_handler Function
The transport_handler
function communicates with the APM domain with
messages about span execution.
# transport handler, needed by py_zipkin
def transport_handler(encoded_span, tracing_context):
return requests.post(
tracing_context.trace_collector_url(),
data=encoded_span,
headers={"Content-Type": "application/json"},
)
- The
trace_collector_url
is returned from the function context. This URL provides the communication endpoint for your custom spans to the APM domain.
Creating a Custom Span in Example Function
The example function demonstrates the creation of a custom span.
def example(ctx):
with zipkin_span(
service_name=ctx.TracingContext().service_name(),
span_name="Get ADB Password from OCI Vault",
binary_annotations=ctx.TracingContext().annotations()
) as example_span_context:
try:
logging.getLogger().debug("Get ADB Password from OCI Vault")
time.sleep(0.005)
# throwing an exception to show how to add error messages to spans
raise Exception('Request failed')
except (Exception, ValueError) as error:
example_span_context.update_binary_annotations(
{"Error": True, "errorMessage": str(error)}
)
else:
FakeResponse = namedtuple("FakeResponse", "status, message")
fakeResponse = FakeResponse(200, "OK")
# how to update the span dimensions/annotations
example_span_context.update_binary_annotations(
{
"responseCode": fakeResponse.status,
"responseMessage": fakeResponse.message
}
)
- The
with zipkin_span
statement is used to identify the custom span and give it a name. - The
example_span_context
block raises an exception and returns an error message.
Here is the complete source code for the sample Python tracing function.
import io
import json
import logging
from fdk import response
import requests
import time
from py_zipkin import Encoding
from py_zipkin.zipkin import zipkin_span
from collections import namedtuple
# transport handler, needed by py_zipkin
def transport_handler(encoded_span, tracing_context):
return requests.post(
tracing_context.trace_collector_url(),
data=encoded_span,
headers={"Content-Type": "application/json"},
)
def handler(ctx, data: io.BytesIO = None):
tracing_context = ctx.TracingContext()
with zipkin_span(
service_name=tracing_context.service_name(),
span_name="Customer Code",
transport_handler=(
lambda encoded_span: transport_handler(
encoded_span, tracing_context
)
),
zipkin_attrs=tracing_context.zipkin_attrs(),
encoding=Encoding.V2_JSON,
binary_annotations=tracing_context.annotations()
):
name = "World"
try:
body = json.loads(data.getvalue())
name = body.get("name")
except (Exception, ValueError) as ex:
logging.getLogger().info('error parsing json payload: ' + str(ex))
logging.getLogger().info("Inside Python Hello World function")
time.sleep(0.005)
example(ctx)
return response.Response(
ctx, response_data=json.dumps(
{"message": "Hello {0}".format(name)}),
headers={"Content-Type": "application/json"}
)
def example(ctx):
with zipkin_span(
service_name=ctx.TracingContext().service_name(),
span_name="Get ADB Password from OCI Vault",
binary_annotations=ctx.TracingContext().annotations()
) as example_span_context:
try:
logging.getLogger().debug("Get ADB Password from OCI Vault")
time.sleep(0.005)
# throwing an exception to show how to add error messages to spans
raise Exception('Request failed')
except (Exception, ValueError) as error:
example_span_context.update_binary_annotations(
{"Error": True, "errorMessage": str(error)}
)
else:
FakeResponse = namedtuple("FakeResponse", "status, message")
fakeResponse = FakeResponse(200, "OK")
# how to update the span dimensions/annotations
example_span_context.update_binary_annotations(
{
"responseCode": fakeResponse.status,
"responseMessage": fakeResponse.message
}
)
Here's an example of how to use Zipkin to add custom spans to your Node.js function. If you want to try you this example, you can create a Node "Hello World!" function and add custom span code. To create a sample function:
- Create a Node function:
fn init --runtime node apm-fn-node
Configure Node Dependencies
Update your package.json
file to include the following packages:
{
"name": "apm-tracing-node-fdk-simple-trace-final",
"version": "1.0.0",
"description": "Example APM tracing function",
"main": "func.js",
"author": "",
"license": "Apache-2.0",
"dependencies": {
"@fnproject/fdk": ">=0.0.13",
"node-fetch": "^2.6.1",
"zipkin": "^0.22.0",
"zipkin-transport-http": "^0.22.0"
}
}
Save the file.
Update Handle Method
Key observations about the fdk.handle
method follow the source
code.
// ZipkinJS core components.
const {
ExplicitContext,
Annotation,
Tracer,
TraceId,
BatchRecorder,
jsonEncoder,
sampler,
option
} = require('zipkin');
// An HTTP transport for dispatching Zipkin traces.
const {HttpLogger} = require('zipkin-transport-http');
fdk.handle(async function(input, ctx){
tracer = createOCITracer(ctx);
var result;
// Start a new 'scoped' server handling span.
await tracer.scoped(async function () {
// Fetch some resource
result = await tracer.local('fetchResource', () => {
return fetchResource();
});
// Perform some processing
result = await tracer.local('processResource', () => {
return someComputation(result);
});
// Update some resource
result = await tracer.local('updateResource', () => {
return updateResource(result);
});
await flush();
});
return result;
})
- The
tracer
is created and then used to create a parent custom span. Then child spans are created for thefetchResource
,processResource
, andupdateResource
functions.
Reviewing the createOCITracer Function
Key observations about the function follow the source code.
/**
* Creates a basic Zipkin Tracer using values from context of the function
* invocation.
*
* @param {*} ctx The function invocation context.
* @returns A configured Tracer for automatically tracing calls.
*/
function createOCITracer (ctx) {
// An OCI APM configured Tracer
//
const tracingCxt = ctx.tracingContext
const tracer = new Tracer({
ctxImpl: new ExplicitContext(),
recorder: new BatchRecorder({
logger: new HttpLogger({
// The configured OCI APM endpoint is available in the function
// invocation context.
endpoint: tracingCxt.traceCollectorUrl,
jsonEncoder: jsonEncoder.JSON_V2
})
}),
// APM Dimensions that should be included in all traces can be configured
// directly on Tracer.
defaultTags: createOCITags(ctx),
// A custom sampling strategy can be defined.
sampler: createOCISampler(ctx),
localServiceName: tracingCxt.serviceName,
supportsJoin: true,
traceId128Bit: true
})
// The initial function invocation trace identifiers can be added directly.
// If this is not defined a default TraceId is created.
const traceId = createOCITraceId(tracer, ctx)
tracer.setId(traceId)
return tracer
}
- The function context (
ctx
) is passed to this function which provides the information required to connect to the APM domain. If you follow the function calls, you can see how the tracing IDs and fields are built.
Here is the complete source code for the sample Node tracing function.
const fdk = require('@fnproject/fdk')
// ZipkinJS core components.
const {
ExplicitContext,
Tracer,
TraceId,
BatchRecorder,
jsonEncoder,
sampler,
option
} = require('zipkin')
// An HTTP transport for dispatching Zipkin traces.
const { HttpLogger } = require('zipkin-transport-http')
fdk.handle(async function (input, ctx) {
var tracer = createOCITracer(ctx)
var result
// Start a new 'scoped' server handling span.
await tracer.scoped(async function () {
// Fetch some resource
result = await tracer.local('fetchResource', () => {
return fetchResource()
})
// Perform some processing
result = await tracer.local('processResource', () => {
return someComputation(result)
})
// Update some resource
result = await tracer.local('updateResource', () => {
return updateResource(result)
})
await flush()
})
return result
})
// ----------------------------------------------------------------------------
// App Simulation Functions
//
/**
* Simulate fetching some required resource. This could be another OCI service
* or an external call.
*
* @returns A Promise with the success or failure of the operation.
*/
function fetchResource () {
return simulate(1000, { fetchResource: 'OK' })
}
/**
* Simulate some work. This could be another OCI service.
*
* @returns A Promise with the success or failure of the operation.
*/
async function someComputation (toReturn) {
var i
for (i = 0; i < 5; i++) {
await simulate(1000)
}
toReturn.processResource = 'OK'
return toReturn
}
/**
* Simulate updating some resource. This could be another OCI service or an
* external call.
*
* @returns A Promise with the success or failure of the operation.
*/
async function updateResource (toReturn) {
await simulate(500)
toReturn.updateResource = 'OK'
return toReturn
}
/**
* A helper function to simulate an operation that takes a specified amount of time.
*
* @param {*} ms The simulated time for the activity in milliseconds.
* @returns A promise that resolves when the simulated activity finishes.
*/
function simulate (ms, result) {
return new Promise(resolve => setTimeout(resolve, ms, result))
}
/**
* Functions service may freeze or terminate the container on completion.
* This function gives extra time to allow the runtime to flush any pending traces.
* See: https://github.com/openzipkin/zipkin-js/issues/507
*
* @returns A Promise to await on.
*/
function flush () {
return new Promise(resolve => setTimeout(resolve, 1000))
}
// ----------------------------------------------------------------------------
// OpenZipkin ZipkinJS Utility Functions
//
/**
* Creates a basic Zipkin Tracer using values from context of the function
* invocation.
*
* @param {*} ctx The function invocation context.
* @returns A configured Tracer for automatically tracing calls.
*/
function createOCITracer (ctx) {
// An OCI APM configured Tracer
//
const tracingCxt = ctx.tracingContext
const tracer = new Tracer({
ctxImpl: new ExplicitContext(),
recorder: new BatchRecorder({
logger: new HttpLogger({
// The configured OCI APM endpoint is available in the function
// invocation context.
endpoint: tracingCxt.traceCollectorUrl,
jsonEncoder: jsonEncoder.JSON_V2
})
}),
// APM Dimensions that should be included in all traces can be configured
// directly on Tracer.
defaultTags: createOCITags(ctx),
// A custom sampling strategy can be defined.
sampler: createOCISampler(ctx),
localServiceName: tracingCxt.serviceName,
supportsJoin: true,
traceId128Bit: true
})
// The initial function invocation trace identifiers can be added directly.
// If this is not defined a default TraceId is created.
const traceId = createOCITraceId(tracer, ctx)
tracer.setId(traceId)
return tracer
}
/**
* A ZipkinJS 'TraceId' can be created directly from the function invocation
* context.
*
* @param {*} ctx The function invocation context.
* @returns A ZipkinJS 'TraceId' created from the invocation context.
*/
function createOCITraceId (tracer, ctx) {
const tracingCxt = ctx.tracingContext
if (tracingCxt.traceId && tracingCxt.spanId) {
return new TraceId({
traceId: tracingCxt.traceId,
spanId: tracingCxt.spanId,
sampled: new option.Some(tracingCxt.sampled),
debug: new option.Some(tracingCxt.debug),
shared: false
})
} else {
return tracer.createRootId(
new option.Some(tracingCxt.sampled),
new option.Some(tracingCxt.debug)
)
}
}
/**
* A ZipkinJS 'TraceId' can be crated directly from the function invocation
* context.
*
* This configurations will automatically add the function meta-data as APM
* dimensions to each trace. Function environment variable and other dimensions
* could also be added.
*
* @param {*} ctx The function invocation context.
* @returns A map of key-value pairs, that will be added as APM
* dimensions to the traces.
*/
function createOCITags (ctx) {
return {
appID: ctx.appID,
appName: ctx.appName,
fnID: ctx.fnID,
fnName: ctx.fnName
}
}
/**
* A ZipkinJS 'Sampler' can be created directly from the function invocation
* context.
*
* This configuration will only create a trace if the function is configured
* for tracing.
*
* @param {*} ctx The function invocation context.
* @returns A ZipkinJS 'TraceId' created from the invocation context.
*/
function createOCISampler (ctx) {
return new sampler.Sampler((traceId) => ctx.tracingContext.isEnabled)
}
Using the API
For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.
Use these API operations to enable and disable tracing for applications and the functions they contain: