Grail: The New Data Lakehouse

What is Grail?

Grail is Dynatrace's unified data lakehouse. In Gen2, data was stored in separate backends — metrics in one place, logs in another, traces in a third. In Gen3, everything goes into Grail and is queryable with DQL.

GEN2 Data Storage                       GEN3 Grail
──────────────────────────────────────  ──────────────────────────────────────
Metrics → Timeseries DB                 Grail → timeseries command
Logs → Log storage (Elasticsearch)      Grail → fetch logs
Traces → PurePath storage               Grail → fetch spans
Entities → Topology DB                  Grail → smartscapeNodes
User sessions → USQL storage            Grail → fetch user.events
Events → Event store                    Grail → fetch events
Business events → (didn't exist)        Grail → fetch bizevents
Security events → (limited)             Grail → fetch security.events
Problems → Problem feed                 Grail → fetch dt.davis.problems

Buckets and Tables

Grail organizes data into buckets and tables:

• Default bucket — where all standard data lands (logs, metrics, spans, events)
• Custom buckets — you can create buckets with different retention policies
• Tables — each data type has its own table within a bucket

Bucket: default
  ├── logs          (application + infrastructure logs)
  ├── spans         (distributed traces)
  ├── events        (Davis events, custom events)
  ├── bizevents     (business events)
  └── metrics       (timeseries data)

Bucket: custom_long_retention
  └── logs          (compliance logs, 5-year retention)

Retention

Gen2 had fixed retention per data type. Gen3 lets you configure retention per bucket:

GEN2 Retention                          GEN3 Retention
──────────────────────────────────────  ──────────────────────────────────────
Logs: 35 days (fixed)                   Logs: configurable per bucket
Metrics: 10 years (fixed)               Metrics: configurable per bucket
Traces: 10 days (fixed)                 Spans: configurable per bucket
User sessions: 35 days (fixed)          User events: configurable per bucket

OpenPipeline

Gen3 introduces OpenPipeline — a processing engine that sits between data ingestion and storage. It replaces log processing rules, metric extraction, and data routing.

Data Source → OpenPipeline → Routing → Bucket/Table
                  │
                  ├── Parse (extract fields)
                  ├── Filter (drop unwanted data)
                  ├── Transform (enrich, rename)
                  ├── Extract metrics (create metrics from logs)
                  └── Route (send to specific bucket)

In Gen2, you configured log processing rules in Settings. In Gen3, you configure OpenPipeline rules — same concept, more powerful, and it works for all data types.

Data Access Permissions

Grail introduces field-level and record-level access control:

• Bucket-level — who can read/write which buckets
• Table-level — who can query which tables (logs, spans, etc.)
• Record-level — filter data based on attributes (e.g., only see logs from your team's services)
• Field-level — hide sensitive fields (e.g., mask PII in log content)

This is a massive upgrade from Gen2 where data access was all-or-nothing per environment.