An alert watches one metric of one endpoint and notifies your team when the metric crosses a threshold. Monocle evaluates every alert each minute, sends one notification when it fires, and one when it resolves.
Create an alert
Open an endpoint from the Requests page and click Create alert in its header. The method, route, and service are filled in for you. You can also click New alert on the Alerts page and pick the endpoint yourself.
The condition reads as one sentence:
Alert when p95 latency of POST /api/checkout in api is above 800 ms over the last 5 min
| Field | Values |
|---|---|
| Metric | See Metrics. p95 latency by default. |
| Endpoint | A matched route. The picker lists the busiest ones of the last 7 days. Raw paths from Unmatched paths can't be used. |
| Service | The service that handles the endpoint. Required, because two services can serve the same route. |
| Comparison | above or below. The comparison is strict, so a value equal to the threshold doesn't breach. |
| Threshold | A number of 0 or more, in the unit of the metric |
| Window | 5 min (default), 10 min, 15 min, or 30 min |
Then choose who to notify under Notify, and optionally rename the alert under Name. The default name sums up the condition, such as p95 latency · POST /api/checkout > 800ms.
Metrics
| Metric | Unit | What it measures |
|---|---|---|
| p50 latency | ms | Median duration of the requests in the window |
| p95 latency | ms | 95th percentile duration |
| p99 latency | ms | 99th percentile duration |
| Average latency | ms | Average duration |
| Error rate | % | Share of requests with a status of 400 or above |
| 5xx error rate | % | Share of requests with a status of 500 or above |
| Request rate | req/min | Requests in the window divided by its length, in minutes |
Only the requests that matched the route of the endpoint count.
Advanced options
| Option | Default | Description |
|---|---|---|
| Must breach for | 2 min | How long the condition must hold before the alert fires: Fire immediately, 2 min, 5 min, or 10 min |
| Minimum requests | 20 | A window with fewer requests never fires (1 to 100,000). Not used by Request rate. |
| Alert if no traffic | Off | On, a window with fewer requests than the minimum counts as a breach. Off, such a window resolves a firing alert. |
To catch an endpoint that stops receiving traffic, use Request rate below a threshold. A window with no request has a rate of 0.
Preview before you save
While you edit the condition, the Preview runs the alert on the recent traffic of the endpoint. It marks the periods where the alert Would have fired, with the number of times and the total duration. Windows with fewer requests than the minimum show as Under N requests (ignored).
The preview covers the last 24 hours (24h) by default, or the last hour (1h). p50 latency and p99 latency are limited to the last hour. The preview checks a window every 5 minutes while the alert checks one every minute, so the count is an approximation.
For latency metrics, Monocle suggests a threshold equal to the metric over the preview range, multiplied by 1.5. For example, Suggested from your 24h p95 × 1.5. The suggestion fills the threshold until you type your own value. Click the Use link next to it to apply it again.
How alerts are evaluated
Every minute, Monocle computes the metric over the window of each alert. The window ends 2 minutes before the current minute, so late data has time to arrive.
- A window that breaches the condition moves the alert to Pending.
- If the condition keeps breaching for the Must breach for duration, the alert moves to Firing and Monocle sends a notification. With Fire immediately, the first breach fires.
- The first window that doesn't breach resolves the alert, and Monocle sends a second notification. A Pending alert goes back to OK without a notification.
An alert sends nothing between these two notifications, even if the value gets worse. Monocle records the peak value of the incident instead.
If Monocle received no data at all from any service in the last minute, it skips the evaluation, so an ingestion delay doesn't fire or resolve alerts. After 3 failed evaluations in a row, the alert shows Error and keeps its state until an evaluation succeeds.
Notifications
Under Notify, click Add channel and pick one or more destinations. An alert needs at least one.
- Channels: the Slack, Discord, and webhook notification channels of your organization, up to 20 per alert. Only organization admins can create a channel. They can do it from this menu with New notification channel.
- App members: every member with access to the project gets an email and an in-app notification. Each member can turn off the email under Endpoint alerts in their email preferences.
The default channels of your organization don't apply to alerts. Each alert has its own destinations.
The firing message names the alert, the metric, its current value, the threshold, the number of requests in the window, the service, and the environment. It links to the endpoint and to its slowest traces. Emails use the subjects [Alert] <name> and [Resolved] <name>. Webhook channels receive the alert.firing and alert.resolved events. See Webhook payloads.
Manage your alerts
The Alerts page lists every alert of the environment, and the sidebar shows how many are firing. Filter the list with All, Firing, OK, or Paused. Each row shows the condition, a trend of the metric, its current value, and when the alert last fired.
| Status | Meaning |
|---|---|
| OK | The last window didn't breach |
| Pending | The condition breaches, but not for the Must breach for duration yet |
| Firing | The alert fired and hasn't resolved |
| Paused | The alert is not evaluated |
| Error | The last evaluations failed. The state is frozen until one succeeds. |
| Needs service | The alert was created before alerts required a service. Edit it to choose one. |
Click an alert to open its page:
- The current value, the threshold, and the number of requests in the window.
- A chart of the value of each evaluation, with the threshold, the incidents, and the deployments of the period. Monocle keeps evaluations for 30 days.
- Incidents: each time the alert fired, with its duration, its peak value, why it resolved, and a link to the slowest traces of the period.
- Last notification: the last message sent, and where it went.
Pause, edit, or delete an alert
From the alert page, or from the menu of a row on the Alerts page:
- Pause stops the evaluation, for example during a planned migration. A firing alert resolves and sends its resolved notification. Resume starts the evaluation again.
- Edit opens the same form. A change to the condition resets the alert to OK and resolves an open incident. If you change the metric or the endpoint, the history of the alert starts over. The chart, the trend, and the current value drop the old values, which may be in another unit. Past incidents stay listed. A change to the name or the destinations keeps the state.
- Delete removes the alert and its incident history. No resolved notification is sent.
Limits
- 50 alerts per environment, paused ones included.
- 50 active alerts per organization. Paused alerts don't count.
- Any member with access to the project can create, edit, pause, and delete its alerts.