edx_django_utils.monitoring.internal package

Contents

edx_django_utils.monitoring.internal package#

Subpackages#

Submodules#

edx_django_utils.monitoring.internal.backends module#

Telemetry abstraction and backends that implement it.

Only a certain subset of the monitoring functions have been made configurable via this module.

class edx_django_utils.monitoring.internal.backends.DatadogBackend#

Bases: TelemetryBackend

Send telemetry to Datadog via ddtrace.

API reference: https://ddtrace.readthedocs.io/en/stable/api.html

create_span(name)#

Start a tracing span with the given name, returning a context manager instance.

The caller must use the return value in a with statement or similar so that the span is guaranteed to be closed appropriately.

Implementations should create a new child span parented to the current span, or create a new root span if not currently in a span.

record_exception()#

Record the exception that is currently being handled.

set_attribute(key, value)#

Set a key-value attribute on a span. This might be the current span or it might the root span of the process, depending on the backend.

set_local_root_span_name(name, group=None, priority=None)#

Sets the name, group, and priority for a span.

tag_root_span_with_error(exception)#

Tags the root span with the given exception. This is primarily useful for Datadog as New Relic handles this behavior correctly. Unclear if this is also needs to be implemented for OTEL.

class edx_django_utils.monitoring.internal.backends.NewRelicBackend#

Bases: TelemetryBackend

Send telemetry to New Relic.

API reference: https://docs.newrelic.com/docs/apm/agents/python-agent/python-agent-api/guide-using-python-agent-api/

create_span(name)#

Start a tracing span with the given name, returning a context manager instance.

The caller must use the return value in a with statement or similar so that the span is guaranteed to be closed appropriately.

Implementations should create a new child span parented to the current span, or create a new root span if not currently in a span.

record_exception()#

Record the exception that is currently being handled.

set_attribute(key, value)#

Set a key-value attribute on a span. This might be the current span or it might the root span of the process, depending on the backend.

set_local_root_span_name(name, group=None, priority=None)#

Sets the name, group, and priority for a span.

tag_root_span_with_error(exception)#

Tags the root span with the given exception. This is primarily useful for Datadog as New Relic handles this behavior correctly. Unclear if this is also needs to be implemented for OTEL.

class edx_django_utils.monitoring.internal.backends.OpenTelemetryBackend#

Bases: TelemetryBackend

Send telemetry via OpenTelemetry.

API reference: https://opentelemetry-python.readthedocs.io/en/latest/

create_span(name)#

Start a tracing span with the given name, returning a context manager instance.

The caller must use the return value in a with statement or similar so that the span is guaranteed to be closed appropriately.

Implementations should create a new child span parented to the current span, or create a new root span if not currently in a span.

record_exception()#

Record the exception that is currently being handled.

set_attribute(key, value)#

Set a key-value attribute on a span. This might be the current span or it might the root span of the process, depending on the backend.

set_local_root_span_name(name, group=None, priority=None)#

Sets the name, group, and priority for a span.

tag_root_span_with_error(exception)#

Tags the root span with the given exception. This is primarily useful for Datadog as New Relic handles this behavior correctly. Unclear if this is also needs to be implemented for OTEL.

class edx_django_utils.monitoring.internal.backends.TelemetryBackend#

Bases: ABC

Base class for telemetry sinks.

abstractmethod create_span(name)#

Start a tracing span with the given name, returning a context manager instance.

The caller must use the return value in a with statement or similar so that the span is guaranteed to be closed appropriately.

Implementations should create a new child span parented to the current span, or create a new root span if not currently in a span.

abstractmethod record_exception()#

Record the exception that is currently being handled.

abstractmethod set_attribute(key, value)#

Set a key-value attribute on a span. This might be the current span or it might the root span of the process, depending on the backend.

abstractmethod set_local_root_span_name(name, group=None, priority=None)#

Sets the name, group, and priority for a span.

abstractmethod tag_root_span_with_error(exception)#

Tags the root span with the given exception. This is primarily useful for Datadog as New Relic handles this behavior correctly. Unclear if this is also needs to be implemented for OTEL.

edx_django_utils.monitoring.internal.backends.configured_backends()#

Produce a list of TelemetryBackend instances from Django settings.

edx_django_utils.monitoring.internal.middleware module#

Middleware for monitoring.

class edx_django_utils.monitoring.internal.middleware.CachedCustomMonitoringMiddleware(get_response)#

Bases: MonitoringSupportMiddleware

DEPRECATED: Use MonitoringSupportMiddleware instead.

This is the old name for the MonitoringSupportMiddleware. We are keeping it around for backwards compatibility until it can be fully removed.

class edx_django_utils.monitoring.internal.middleware.CookieMonitoringMiddleware(get_response)#

Bases: object

Middleware for monitoring the size and growth of all our cookies, to see if we’re running into browser limits.

get_log_message_and_monitor_cookies(request)#

Add logging and custom attributes for monitoring cookie sizes.

For cookie size monitoring, we don’t log raw contents of cookies because that might cause a security issue—we just want to see if any cookies are growing out of control. However, there is also an option for encrypted logging of cookie data where there appears to be some kind of data corruption occurring.

Useful NRQL Queries:

# Always available SELECT * FROM Transaction WHERE cookies.header.size > 6000

Attributes that are added by this middleware:

For all requests:

cookies.header.size: The total size in bytes of the cookie header

If COOKIE_HEADER_SIZE_LOGGING_THRESHOLD is reached:

cookies.header.size.computed

Related Settings (see annotations for details):

  • COOKIE_HEADER_SIZE_LOGGING_THRESHOLD

  • COOKIE_SAMPLING_REQUEST_COUNT

  • UNUSUAL_COOKIE_HEADER_PUBLIC_KEY

  • UNUSUAL_COOKIE_HEADER_LOG_CHUNK

Returns: The message to be logged. This is returned, rather than directly

logged, so that it can be processed at request time (before any cookies may be changed server-side), but logged at response time, once the user id is available for authenticated calls.

Log all headers when corrupt cookies are detected (if settings permit).

This log data is encrypted using the log-sensitive utility.

  • Logging requires that UNUSUAL_COOKIE_HEADER_PUBLIC_KEY is set.

  • Output is split across multiple lines using UNUSUAL_COOKIE_HEADER_LOG_CHUNK.

class edx_django_utils.monitoring.internal.middleware.DeploymentMonitoringMiddleware(get_response)#

Bases: object

Middleware to record environment values at the time of deployment for each service.

static record_django_version()#

Record the installed Django version as custom attribute

static record_python_version()#

Record the Python version as custom attribute

class edx_django_utils.monitoring.internal.middleware.FrontendMonitoringMiddleware(get_response)#

Bases: object

Middleware for adding the frontend monitoring scripts to the response.

inject_script(content, script)#

Add script to the response, if body tag is present.

class edx_django_utils.monitoring.internal.middleware.MonitoringMemoryMiddleware(get_response)#

Bases: MiddlewareMixin

Middleware for monitoring memory usage.

Make sure to add below the request cache in MIDDLEWARE.

guid_key = 'guid_key'#
memory_data_key = 'memory_data'#
process_request(request)#

Store memory data to log later.

process_response(request, response)#

Logs memory data after processing response.

class edx_django_utils.monitoring.internal.middleware.MonitoringSupportMiddleware(get_response)#

Bases: MiddlewareMixin

Middleware to support monitoring.

  1. Send process request, response, and exception signals to enable plugins to add custom monitoring.

  2. Middleware batch reports cached custom attributes at the end of a request.

  3. Middleware adds error span tags to the root span.

Make sure to add below the request cache in MIDDLEWARE, but as early in the stack of middleware as possible.

This middleware will only call on the telemetry collector if there are any attributes to report for this request, so it will not incur any processing overhead for request handlers which do not record custom attributes.

classmethod accumulate_attribute(name, value)#

Accumulate a custom attribute (name and value) in the attributes cache.

classmethod accumulate_metric(name, value)#

Deprecated method replaced by accumulate_attribute.

process_exception(request, exception)#

Django middleware handler to process an exception

process_request(request)#

Django middleware handler to process a request

process_response(request, response)#

Django middleware handler to process a response

edx_django_utils.monitoring.internal.middleware.split_ascii_log_message(msg, chunk_size)#

Generator that splits a message string into chunks of at most chunk_size characters.

Message must consist of single-byte (ASCII-range) characters, otherwise chunks will not reliably be capped at the chunk_size.

A small collation tag will be added to the end of each chunk, and it may not be possible to reliably predict the length of a log message’s prefix (in its final output format), so the chunk_size should be set considerably smaller than max log message size.

edx_django_utils.monitoring.internal.transactions module#

This is a collection of transaction-related monitoring utilities.

Usage:

from edx_django_utils.monitoring import ignore_transaction … # called from inside a view you want ignored ignore_transaction()

Please remember to expose any new methods in the __init__.py file.

edx_django_utils.monitoring.internal.transactions.ignore_transaction()#

Ignore the transaction in monitoring. Only works for NewRelic.

This allows us to ignore code paths that are unhelpful to include, such as /health/ checks.

edx_django_utils.monitoring.internal.utils module#

Defines several custom monitoring helpers, some of which work with the CachedCustomMonitoringMiddleware.

Usage:

from edx_django_utils.monitoring import accumulate … accumulate(‘xb_user_state.get_many.num_items’, 4)

There is no need to do anything else. The custom attributes are automatically cleared before the next request.

We try to keep track of our custom monitoring at: https://openedx.atlassian.net/wiki/spaces/PERF/pages/54362736/Custom+Attributes+in+New+Relic

At this time, the custom monitoring will only be reported to New Relic.

edx_django_utils.monitoring.internal.utils.accumulate(name, value)#

Accumulate monitoring custom attribute for the current request.

The named attribute is accumulated by a numerical amount using the sum. All attributes are queued up in the request_cache for this request. At the end of the request, the monitoring_utils middleware will batch report all queued accumulated attributes to the monitoring tool (e.g. New Relic).

Parameters:

  • name (str) – The attribute name. It should be period-delimited, and increase in specificity from left to right. For example: ‘xb_user_state.get_many.num_items’.

  • value (number) – The amount to accumulate into the named attribute. When accumulate() is called multiple times for a given attribute name during a request, the sum of the values for each call is reported for that attribute. For attributes which don’t make sense to accumulate, use set_custom_attribute instead.

edx_django_utils.monitoring.internal.utils.background_task(*args, **kwargs)#

Handles monitoring for background tasks that are not passed in through the web server like celery and event consuming tasks.

This function only supports New Relic.

For more details, see: https://docs.newrelic.com/docs/apm/agents/python-agent/python-agent-api/backgroundtask-python-agent-api/

edx_django_utils.monitoring.internal.utils.function_trace(function_name)#

Wraps a chunk of code that we want to appear as a separate, explicit, segment in our monitoring tools.

edx_django_utils.monitoring.internal.utils.increment(name)#

Increment a monitoring custom attribute representing a counter.

Here we simply accumulate a new custom attribute with a value of 1, and the middleware should automatically aggregate this attribute.

edx_django_utils.monitoring.internal.utils.record_exception()#

Record a caught exception to the monitoring system.

Note: By default, only unhandled exceptions are monitored. This function can be called to record exceptions as monitored errors, even if you handle the exception gracefully from a user perspective.

edx_django_utils.monitoring.internal.utils.set_custom_attribute(key, value)#

Set monitoring custom attribute.

This is not cached.

edx_django_utils.monitoring.internal.utils.set_custom_attributes_for_course_key(course_key)#

Set monitoring custom attributes related to a course key.

This is not cached.

edx_django_utils.monitoring.internal.utils.set_monitoring_transaction_name(name, group=None, priority=None)#

Sets the name, group, and priority for the current root span.

Current root span refers to the most ancestral span in the current process, rather than to the trace root, which may be outside of the process.

Group and priority may not be supported by all backends.

Module contents#

Contents