Auto-Remediation Workflows

Auto-Remediation Workflows

Gen2 had limited auto-remediation (custom webhooks from problem notifications). Gen3 Workflows enable full auto-remediation pipelines — detect, analyze, decide, act, notify.

Auto-Remediation Pattern

Davis Problem Trigger
  → DQL: Query problem context (what entity, what metrics)
  → DQL: Check entity health history (is this recurring?)
  → DQL: Get related problems (blast radius)
  → JavaScript: Decision engine (monitor/investigate/remediate/escalate)
  → Action: Email notification OR HTTP call to remediation API

Decision Engine (JavaScript Task)

// Analyze problem context and decide action
export default async function({ execution_id }) {
  const problem = await fetch(`/platform/automation/v1/executions/${execution_id}/tasks/get_problem`);
  const health = await fetch(`/platform/automation/v1/executions/${execution_id}/tasks/check_health`);
  const history = await fetch(`/platform/automation/v1/executions/${execution_id}/tasks/get_history`);

  const problemData = problem.result.records[0];
  const historyCount = history.result.records.length;

  let action = "MONITOR";
  let reason = "New problem, monitoring";

  if (historyCount > 3) {
    action = "ESCALATE";
    reason = `Recurring problem (${historyCount} times in 24h)`;
  } else if (problemData['event.category'] === 'RESOURCE') {
    action = "INVESTIGATE";
    reason = "Resource problem — check capacity";
  }

  return { action, reason, problem_name: problemData['event.name'] };
}

Available Workflow Actions

Action Type                             Use Case
──────────────────────────────────────  ──────────────────────────────────────
dynatrace.email:send-email              Email notification
dynatrace.slack:slack-send-message      Slack notification
dynatrace.jira:jira-create-issue        Create Jira ticket
dynatrace.servicenow:create-incident    Create ServiceNow incident
HTTP action (any URL)                   Custom API calls, webhooks
DQL query                               Enrich context, check conditions
JavaScript                              Custom logic, formatting, decisions

Service User for Workflows

Workflows run under a service user — a non-human identity with its own permissions. This is critical for production workflows.

Setup:
1. Create service user via IAM API
2. Create IAM policy with required scopes
3. Bind policy to service user's auto-created group
4. Set service user as workflow "actor" (NOT "owner"!)

⚠️ NEVER set owner = service user — permanently locks out the workflow
   Only set actor = service user