Executive Summary
Observability is the engineering practice of inferring internal system state from external telemetry — logs, metrics, and traces — without shipping new code for each question. Compliance converts telemetry from an operational convenience into a legal/control objective: log completeness, tamper-evidence, time synchronization, access control, retention, and auditability must be demonstrable to an auditor or regulator (NIST SP 800-92, publication ID NIST SP 800-92, September 2006; accessed 2026-04-27).
The technology value chain spans:
- Instrumentation/collection: OpenTelemetry SDKs, eBPF probes, Prometheus exporters.
- Transport/processing: OpenTelemetry Collector receivers, processors, and exporters for filtering, redaction, sampling, and routing.
- Storage/indexing: time-series engines for metrics, full-text search for logs, trace-native stores.
- Stream/compute correctness: watermark- and trigger-based out-of-order event processing.
- Compliance controls: immutable retention, access control, time sync, audit trails.
The thesis is that observability and compliance are converging into a single data-pipeline and governance problem. Vendors monetize this convergence with per-host or usage-based SaaS pricing, while open-source instrumentation standards such as OpenTelemetry reduce switching costs and force vendors to differentiate on analytics, compliance controls, retention economics, and AI-assisted root-cause analysis. This is not a "pure open-source wins" thesis: the storage, compliance, and query layers remain expensive and sticky.
Definitions and Technology Stack
2.1 The three signal types
OpenTelemetry, the dominant open instrumentation project under the Cloud Native Computing Foundation, defines the moving parts as:
- Metrics: numeric measurements with timestamps and attributes.
- Logs: timestamped structured or unstructured records.
- Traces: directed acyclic graphs of spans that represent causal relationships across process boundaries.
(OpenTelemetry Observability Primer, accessed 2026-04-27)
This triad is now the lingua franca of vendor-neutral observability. The OpenTelemetry Collector supports receiving, processing, and exporting all three signal types (OpenTelemetry Collector docs, accessed 2026-04-27). The signal-triad claim is a load-bearing structural claim; it is verifiable directly from the OpenTelemetry project documentation but is [SINGLE-SOURCE] for the specific normative definitions because no independent standards-body adoption was retrieved.
2.2 Compliance layer
Compliance is not merely "keep logs." NIST Special Publication 800-92, Guide to Computer Security Log Management, states that organizations should establish policies for log generation, retention, protection, and disposal, and that log management infrastructure should protect confidentiality, integrity, and availability of log data (NIST SP 800-92, September 2006; accessed 2026-04-27). NIST SP 800-137, Information Security Continuous Monitoring, extends the requirement from static log review to continuous monitoring and assessment (NIST SP 800-137, September 2011; accessed 2026-04-27).
In an observability platform, compliance-specific technical requirements include:
- Complete or policy-governed event capture: not every trace must be retained, but security-relevant audit events generally should not be silently sampled away.
- Tamper-evidence and write-once/immutable storage: log integrity is a control objective.
- Time synchronization: log correlation assumes trustworthy timestamps.
- Access control and audit of the observability platform itself: the monitor must itself be monitored.
- Retention policies tied to legal/contractual obligations: e.g., SOX, PCI DSS, HIPAA, GDPR, or internal policy.
These requirements are not generic engineering preferences; they are grounded in the control language of NIST SP 800-92 and NIST SP 800-137.
2.3 eBPF as an instrumentation surface
eBPF is a kernel-level execution mechanism that allows sandboxed programs to run when events occur, such as network packets, syscalls, or scheduler events. It is used for low-overhead, always-on data collection without requiring application source changes (eBPF Foundation “What is eBPF?”, accessed 2026-04-27). This is architecturally important because it can observe the kernel and system-call surface even for legacy applications.
The specific claim that eBPF enables low-overhead kernel-level telemetry collection is [SINGLE-SOURCE] from the eBPF Foundation documentation in this note. It is widely used in production, but this note did not retrieve an independent peer-reviewed benchmark quantifying the overhead in a controlled setting. A production deployment context should be revalidated against vendor benchmark reports or peer-reviewed systems papers.
2.4 Stream processing and out-of-order data
Real-world telemetry arrives out of order, late, and unevenly. Akidau et al., "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing," Proceedings of the VLDB Endowment, Vol. 8, No. 12, pp. 1793–1804, 2015, DOI 10.14778/2824032.2824076 (DOI link, published 2015-08-27; accessed 2026-04-27), formalizes concepts such as event-time vs. processing-time, watermarks, and triggers. This is directly relevant to observability pipelines: a compliant event stream cannot simply drop late events; the pipeline must issue correct retractions or late-data updates.
2.5 Tail-latency amplification and why tracing exists
Dean and Barroso, "The Tail at Scale," Communications of the ACM, Vol. 56, No. 2, pp. 74–80, 2013, DOI 10.1145/2408776.2408794 (DOI link, published 2013-02-01; accessed 2026-04-27), explains why one slow leaf in a fan-out request graph can degrade a disproportionate share of user-facing requests. The paper is peer-reviewed and is the conceptual basis for modern distributed tracing: without trace context, you cannot localize the slow leaf; you only see aggregate latency.
Patent Evidence
A patent is evidence of recorded legal claims, not proof of performance or commercial viability. This note identifies one granted patent in the log/machine-data observability space:
- US 8,694,474 B2
Document identifier: US8694474B2
Status: granted, not application
Grant date: 2014-04-08
URL: [patents.google.com
Access date: 2026-04-27
The patent is indexed as a Splunk Inc. machine-data indexing/search patent at Google Patents. It is relevant to the full-text search layer of observability and to compliance log retrieval. The available patent page is treated as a Tier 1 primary source for the existence of the granted claims, but it is [SINGLE-SOURCE] for patent-content claims and is not evidence that a specific vendor product implements it or that the patented technique is commercially necessary. Any assertion that this patent creates a durable moat would require claim-level mapping against current products, which is beyond the present note and is not asserted here.
Peer-Reviewed Evidence and Standard Documents
4.1 Dataflow Model
Citation: Akidau, T., Bradshaw, R., Chambers, C., et al. "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing." Proceedings of the VLDB Endowment, Vol. 8, No. 12, 2015, pp. 1793–1804. DOI 10.14778/2824032.2824076. Published 2015-08-27. Accessed 2026-04-27.
Why it matters: Observability pipelines ingest unbounded event streams. The Dataflow Model is a peer-reviewed formal treatment of when to emit aggregations, how to handle late data, and how to balance completeness, latency, and cost. A compliance-adjacent pipeline can use watermarks to decide whether an audit aggregation window is "complete enough" to close, while still allowing late retractions.
4.2 The Tail at Scale
Citation: Dean, J., and Barroso, L. A. "The Tail at Scale." Communications of the ACM, Vol. 56, No. 2, 2013, pp. 74–80. DOI 10.1145/2408776.2408794. Published 2013-02-01. Accessed 2026-04-27.
Why it matters: It supplies the peer-reviewed rationale for distributed tracing. The paper shows that with fan-out, the probability of seeing at least one slow component rises quickly even when individual components are reliable. This is not a vendor marketing claim; it is a peer-reviewed latency model.
4.3 Dapper
Citation: Sigelman, B. H., Barroso, L. A., Burrows, M., et al. "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure." Google Technical Report, 2010. URL: [research.google ; accessed 2026-04-27.
Status: technical report, not peer-reviewed. It is Tier 2, not Tier 1. It is included as historical deployment context for low-overhead tracing, and because it is an independent lineage from the peer-reviewed Tail at Scale paper. The technical claim that Dapper demonstrates production tracing at Google scale is self-reported by Google and therefore is not a strong moat/feasibility proof on its own.
4.4 NIST standards
- NIST SP 800-92 — Guide to Computer Security Log Management, September 2006. URL: [nvlpubs.nist.gov
- NIST SP 800-137 — Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations, September 2011. URL: [nvlpubs.nist.gov
Both are Tier 1 standards-body documents for the compliance side of the stack. They do not endorse any vendor; they define the control objectives that observability products must meet.
Quantitative Worked Calculations
5.1 Tail-latency amplification from fan-out
Source logic: Dean and Barroso, The Tail at Scale, DOI 10.1145/2408776.2408794.
Formula:
\[
P(\text{at least one slow leaf}) = 1 - (1-p)^n
\]
Where:
- \(p\) = probability a single leaf exceeds the latency target.
- \(n\) = number of leaf requests in the fan-out.
Inputs:
- \(n = 100\)
- \(p = 0.01\) (1% slow-leaf probability)
Calculation:
\[
P = 1 - (1 - 0.01)^{100}
\]
\[
P = 1 - 0.99^{100}
\]
\[
P \approx 1 - 0.3660
\]
\[
P \approx 0.634 = 63.4\%
\]
Interpretation: If a service fans out to 100 leaves and each leaf has only a 1% chance of being slow, the aggregate request has a 63.4% chance of hitting at least one slow leaf. This is tail-latency amplification.
Sensitivity table:
| \(n\) fan-out | \(p=0.01\) slow-leaf probability | Probability of ≥1 slow leaf |
|---|---|---|
| 10 | 0.01 | \(1 - 0.99^{10} \approx 9.6\%\) |
| 50 | 0.01 | \(1 - 0.99^{50} \approx 39.5\%\) |
| 100 | 0.01 | \(1 - 0.99^{100} \approx 63.4\%\) |
| 1000 | 0.01 | \(1 - 0.99^{1000} \approx 99.996\%\) |
This calculation is directly derived from the fan-out argument in Tail at Scale. The specific \(p=0.01\) and \(n=100\) inputs are illustrative assumptions, not quoted measured values from the paper.
5.2 Daily log volume and compliance retention
Formula:
\[
\text{Events/day} = \text{hosts} \times \text{events/sec/host} \times 86{,}400
\]
\[
\text{Raw bytes/day} = \text{Events/day} \times \text{avg bytes/event}
\]
Inputs:
| Parameter | Value | Source | Tier | Measured/derived/assumed/inferred | Cross-verified |
|---|---|---|---|---|---|
| hosts | 1,000 | Assumed for scale illustration | unverified | assumed | no |
| events/sec/host | 100 | Assumed; needs primary source validation | unverified | assumed | no |
| avg event size | 500 bytes | Assumed; needs primary source validation | unverified | assumed | no |
| seconds/day | 86,400 | SI definition of day | universal | derived | yes |
| sampling retention | 10% | Assumed; not a compliance requirement | unverified | assumed | no |
| fan-out \(n\) | 100 | Tail at Scale DOI 10.1145/2408776.2408794 | Tier 1 | assumed for illustration | yes |
| slow-leaf \(p\) | 0.01 | Tail at Scale DOI 10.1145/2408776.2408794 | Tier 1 | assumed for illustration | yes |
Calculation for 1,000 hosts:
Events/day:
\[
1{,}000 \times 100 \times 86{,}400 = 8.64 \times 10^9 \text{ events/day}
\]
Raw bytes/day:
\[
8.64 \times 10^9 \times 500 = 4.32 \times 10^{12} \text{ bytes/day}
\]
\[
4.32 \times 10^{12} \text{ bytes} = 4.32 \text{ TB/day raw}
\]
If 10% retained:
\[
0.10 \times 8.64 \times 10^9 = 8.64 \times 10^8 \text{ events/day}
\]
\[
0.10 \times 4.32 \text{ TB} = 0.432 \text{ TB/day raw retained}
\]
Sensitivity: If event rate is 1,000 events/sec/host instead of 100, raw volume becomes 43.2 TB/day. If host count is 10 instead of 1,000, raw volume becomes 43.2 GB/day. This shows why observability pricing is usually per-host or per-GB and why sampling decisions are the dominant cost lever.
Compliance caveat: Sampling is not automatically compliance-valid for audit logs. NIST SP 800-92 emphasizes log integrity and availability; organizations must define which event classes are security-relevant and cannot be silently discarded. Therefore the 10% sample calculation is an engineering-cost calculation, not a compliance assertion. If a compliance regime requires unsampled capture of all security events, the compliance tier must be provisioned at the full event rate.
Unit Economics / Business Model
Observability and compliance converge into a subscription/consumption business. There are three main monetization patterns:
- SaaS observability platform: vendors charge per host/month or per ingested GB with graded tiers. Datadog’s self-reported pricing page illustrates per-host and per-event tiering (Datadog pricing, accessed 2026-04-27). This is self-reported; exact current list prices change and are not restated here as fixed financial figures.
- Hybrid open-core: Elastic licenses Elasticsearch/Kibana and sells Elastic Cloud, security, and observability managed services. Its SEC filings describe a subscription revenue model; exact current margin line items should be revalidated directly from the 10-K (Elastic EDGAR 10-K index, accessed 2026-04-27).
- Hyperscaler-native monitoring and compliance: AWS CloudWatch/CloudTrail, Microsoft Azure Monitor/Sentinel, and Google Cloud Observability monetize by consuming compute, storage, and retention usage rather than by charging a separate observability license.
Who captures value:
- Platform vendors capture the analytics and retention margin.
- Hyperscalers capture the underlying compute/storage margin and native compliance tooling.
- Open-source instrumentation projects (OpenTelemetry, eBPF Foundation) do not directly monetize, but they lower customer switching costs and force vendors to compete on stored-data value and compliance controls.
Margin note: Observability SaaS is generally high-gross-margin software, but this note does not state a specific margin percentage. Public-company gross margins are disclosed in SEC 10-K filings; any exact figure pulled from those filings should be labeled as the specific GAAP/non-GAAP line item from the exact accession. No 10-K accession was retrieved in-session at full text, so exact margin percentages are needs primary source validation rather than reproduced from memory.
Pricing pressure: Open standards reduce collection lock-in, but storage, indexing, and compliance retention are not free. A platform that cannot prove tamper-evidence, access audit, and contractual retention will lose security/compliance budgets to those that can.
Disconfirming Evidence
The convergence thesis would be weakened or falsified by any of the following:
- OpenTelemetry standard loses vendor and cloud support. If a proprietary API or hyperscaler-native closed standard becomes dominant despite OpenTelemetry’s current open-source adoption, the switching-cost reduction thesis weakens. The current adoption evidence is self-reported by the OpenTelemetry project (OpenTelemetry docs, accessed 2026-04-27).
- Compliance regimes accept sampled or aggregate data for audit purposes. If regulators/auditors accept cheap aggregated metrics instead of complete or event-level logs, the high-volume storage and indexing requirement shrinks, lowering the value of full-text log platforms.
- eBPF does not achieve portability or security acceptance. If eBPF remains Linux-specific or is limited by security policy, kernel-level collection will not be the universal instrumentation layer. The current eBPF value claim is [SINGLE-SOURCE] from the eBPF Foundation (eBPF Foundation, accessed 2026-04-27).
- The Splunk patent is invalidated, expires without renewal, or is not essential to current products. Patent invalidations would remove the cited granted-patent moat marker (US8694474B2, granted 2014-04-08; accessed 2026-04-27).
- Observability pricing collapses under hyperscaler-native bundling. If AWS/Azure/GCP bundle observability and compliance telemetry at near-zero incremental price, standalone vendors would face severe margin compression. This would show up as slowing revenue or shrinking gross margins in the vendors’ 10-K filings (SEC EDGAR index for Datadog, accessed 2026-04-27).