feat(eventdrop): add centralized telemetry API for dropped events

[jijo-OO7]

WIP: Phase 1 – Centralized telemetry API for dropped events (#8572)

Description

This PR establishes the foundation for dropped event telemetry in the Broker data plane by introducing a centralized RecordEventDropped API. The package provides OpenTelemetry instrumentation (metrics and traces) for drop sites without modifying existing handler code.

This is Phase 1 - API definition only. Handler integration happens in a follow-up PR once the design is validated.

Changes

• New pkg/eventdrop package with:
  • Reason enum capturing drop causes (TTLMissing, TTLExhausted)
  • Info struct for drop context metadata
  • RecordEventDropped() entry point for unified telemetry
  • OTel metrics (low-cardinality) and traces (high-cardinality)
  • Comprehensive test coverage

Telemetry Strategy

Following Evan's guidance in #8572:

Metrics: Low-cardinality labels only

• eventing_broker_events_dropped_total counter
• Labels: namespace, broker, trigger, reason
• EventType/EventSource excluded to avoid cardinality explosion

Traces: Full context via sampling

• event-dropped event on active spans
• Attributes: all metadata including eventType, eventSource, details
• Safe due to trace sampling

Design Notes

• Graceful degradation: OTEL init failures don't disrupt broker operation
• Flexible Info struct: Not all fields required at all call sites (e.g., Trigger empty at ingress)
• Single entry point: Ensures consistency and simplifies future extensions
• No behavior changes: Telemetry only, existing flow unchanged

Integration (Phase 1.5)

Follow-up PR will wire this into:

• pkg/broker/filter/filter_handler.go (ReasonTTLMissing)
• pkg/broker/ingress/ingress_handler.go (ReasonTTLExhausted)
  Separate PR allows design review before touching handlers.

Testing

Phase 1 validates API surface and graceful degradation. Deep OTel verification deferred until Phase 1.5 integration.

cc/ @evankanderson (per #8572 guidance) @Cali0707 @matzew @creydr - Design feedback before proceeding to handler wiring would be valuable.

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jijo-OO7
Once this PR has been reviewed and has the lgtm label, please assign pierdipi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[knative-prow]

Hi @jijo-OO7. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.