Skip to main content
Version: 1.4

System Health Metrics

Wayfinder provides a set of metrics that give you signals about the health of the system and any issues you may need to investigate. These metrics are emitted in a standard kubernetes way, so you can surface them using your observability tools.

Controller metrics


The following labels are used in controller metrics.

controllerName of the controller. When omitted controller="cluster".
nameName of the object being reconciled. When omitted name="eks-dev".
namespaceNamespace of the object being reconciled. When omitted namespace="dev-team".
result“error”, “requeue”, “requeue_after”, “success”
statusValue of the status field on the status subresource, e.g. “Pending”, “Failed”, “Success”
severityError severity
typeError type

Available metrics

Wayfinder exposes controller metrics using a Prometheus-compatible metrics endpoint.

controller_runtime_reconcile_totalTotal number of reconciliations per controllercontroller, result
controller_runtime_reconcile_errors_totalTotal number of reconciliation errors per controller (see Note)controller
controller_runtime_reconcile_time_secondsLength of time per reconciliation per controllercontroller
controller_runtime_max_concurrent_reconcilesMaximum number of concurrent reconciles per controller.controller
controller_runtime_active_workersNumber of currently used workers per controllercontroller

Note: Same as controller_runtime_reconcile_total{result=”error”}

Additional controller metrics coming soon

wf_controller_reconcile_totalTotal number of reconciliations per controller and objectcontroller, namespace, name, result
wf_controller_reconcile_errors_totalTotal number of reconciliation errors per controller and objectcontroller, namespace, name, severity, type
wf_controller_component_reconcile_totalTotal number of reconciliations per controller, object and (status) componentcontroller, namespace, name, component, result, status
wf_controller_component_reconcile_errors_totalTotal number of reconciliation errors per controller, object and (status component)controller, namespace, name, component, severity, type
wf_controller_reconcile_time_secondsLength of time per reconciliation per controller and objectcontroller, namespace, name
wf_controller_reconcile_interval_secondsLength of time since the last reconciliation happened per controller and objectcontroller, namespace, name
wf_controller_sync_period_secondsDefault sync period for a controllercontroller

Example reconciliation loop with recording metrics

NameRecorded metric
Controller startswf_controller_sync_period_seconds
Reconciliation starts for dev-team/eks-dev clusterwf_controller_reconcile_interval_seconds
Reconcile “Component A”wf_controller_component_reconcile_total{component=”Component A”,result=”error|requeue|requeue_after
Reconcile “Component B”wf_controller_component_reconcile_total{component=”Component B”,result=”error|requeue|requeue_after|success”}
Reconciliation finishedwf_controller_reconcile_total{result=”error|requeue|requeue_after|success”}

API metrics

The Wayfinder API produces a number of metrics around inbound requests and handling.

NameDescriptionRecorded metric
Policy ErrorsThe number of errors encountered trying to add a policypolicy_add_errors
Policy Engine ErrorsA counter on the number of errors encountered in the policy enginepolicy_errors
Policy Evaluation SummaryA summary of the policy evaluation time in secondspolicy_evaluation_seconds|summary
Policy Evaluation Find MatchingA summary of the latency encountered when finding matching policiespolicy_find_matches_seconds|summary
Policies Out of SyncA counter of the number of policies found out of syncpolicy_out_of_sync
HTTP Request Average SummaryThe average latency on requests to the apiserverhttp_request_avg_sec|
HTTP Request Code TotalThe total number of http requests broken down by http codehttp_request_code_total
HTTP Request Error TotalThe total number of http requests that have not been successfulhttp_request_error_total
HTTP Total Number of RequestsThe total number of http requests to the apiserverhttp_request_total

Database metrics

NameDescriptionRecorded metric
Database Total Creation CounterA counter or the create operations in the dbdb_create_counter
Database Total Deletion CounterA counter or the delete operations in the dbdb_delete_counter
Database Latency Summary on deletionsThe latency on delete operations to the dbdb_delete_latency_sec
Overall Database Total ErrorsA counter of the number of errors encountered by the dbdb_error_counter
Database Latency Summary on selecting recordsThe latency on get operations to the dbdb_get_latency_sec
Database Latency Summary on selects / listingThe latency on list operations to the dbdb_list_latency_sec
Database Latency Summary on updates / insertThe latency on set operations to the dbdb_set_latency_sec
Database UpdatesA counter or the update and add operations in the dbdb_update_counter