Monitoring Stream Health

To monitor any active stream for issues that may impact the quality of your stream, navigate to the Stream Health tab on the stream detail page.

During any Active stream, components on this page will display a variety of health indicators.

• Healthy indicates that all of the Health Checks are in a Healthy state.

• Unhealthy indicates that one or more of the Health Checks are in an Unhealthy state

• Idle indicates that the stream is not currently active

• Transcoding — A Healthy status indicates that video transcoding is occurring and the multiple profiles of your stream are available for playback. An Unhealthy status indicates that we have failed to generate the profiles so only the original video will be available. Check the event log below for any actionable errors or check the Status page for platform-wide issues.

• Real-time — This indicates whether our system is transcoding the stream faster than real time, in other words, if the video transcode latency is lower than the video duration. An Unhealthy status may be caused by unconventional configurations. Check your stream software configuration by following the steps in this article, and restart your stream.

• Multistreaming — This health check indicates whether all configured Multistream Targets are connected. If no targets are configured, this will show up as blank and not be considered for the global health state. If this shows up as Unhealthy you can also check the status of the individual Multistream Targets in the Overivew tab, where the targets can be either Idle, Pending, Onlineor Offline:

  • An Idle status indicates that the stream is not currently active so neither is multistreaming.

  • A Pending status indicates that our system is trying to connect to the respective Multistream Target.

  • An Online status indicates that the Multistream Target is successfully connected.

  • An Offline status means that we've received an error when trying to connect to the target. To try again, check the ingest URL and stream key of your destination service, re-configure your Multistream Target and restart your stream.

  Logs

  Logs will surface informational alerts (ex. stream has started/stopped, multistream destinations have connected/disconnected) or any fatal errors from the transcoding process. Here is an example of a fatal error:

  Transcode error from ewr-prod-livepeer-orchestrator-0 for segment 0:
  **Unsupported input pixel format**
  

If you run into this or any similar errors, check the configuration in your streaming software (e.g. OBS) and restart the stream.

Session ingest rate tells you the bitrate of the video received by Livepeer ingest servers, updated every 10 seconds. A high bitrate is suggestive that streams into the Livepeer system are high quality, your encoder is working properly, and your internet connection is good. A fluctuating or low bitrate may suggest that your encoder is configured improperly or that the streamer internet connection isn't as strong as it should be to deliver high quality streams to your viewers.

The health information about a stream can also be queried using a separate Stream Health API, which resides on a different root path /data. To use it, you need to include the same API key as the one used for the regular API, check API key documentation for more information. The API consists of the following endpoints:

• Health Status (/data/stream/{streamId}/health): Grabs the latest health status info about a given stream, referenced by its Stream ID. This API powers the dashboard healthchecks and multistream status.

• Events (/data/stream/{streamId}/events): Provides direct access to the low-level events sent by Livepeer video-processing services. It also allows subscribing via SSE to receive new events as they happen, and is useful for debugging. This API powers the log feed in the dashboard.

Now let's go into more detail about each of the mentioned API endpoints.

Request: GET origin.livepeer.com/data/stream/{id}/health

curl 'https://origin.livepeer.com/data/stream/{id}/health' \
-H 'authorization: Bearer {api-key}'

The stream ID is the same one as used in the regular API.

Response

Content-Type: application/json

{
  "id": "{id}",
  "healthy": {
    "status": true,
    "frequency": { "1m": 1, "10m": 0.9655, "60m": 0.9655 },
    "lastProbeTime": 1635993921603,
    "lastTransitionsTime": 1635993751824
  },
  "conditions": [
    {
      "type": "Active",
      "status": true,
      "extraData": {
        "nodeId": "prod-mist-blue-server-1",
        "region": "lon"
      },
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993739354,
      "lastTransitionsTime": 1635993739354
    },
    {
      "type": "Transcoding",
      "status": true,
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993920641,
      "lastTransitionsTime": 1635993751824
    },
    {
      "type": "TranscodeRealTime",
      "status": true,
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993920641,
      "lastTransitionsTime": 1635993751824
    },
    {
      "type": "TranscodeNoErrors",
      "status": true,
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993920641,
      "lastTransitionsTime": 1635993751824
    },
    {
      "type": "Multistreaming",
      "status": null,
      "lastProbeTime": null,
      "lastTransitionsTime": null
    }
  ],
  "metrics": {
    "MediaTimeMillis": [
      {
        "name": "MediaTimeMillis",
        "dimensions": { "nodeId": "prod-mist-blue-server-1" },
        "last": [1635993921603, 182528]
      }
    ],
    "TranscodeRealtimeRatio": [
      {
        "name": "TranscodeRealtimeRatio",
        "dimensions": {
          "nodeId": "prod-livepeer-broadcaster-6emu-97d8bf496-7zh9f"
        },
        "last": [1635993920641, 3.088169642857143]
      }
    ]
  }
}

In the above response payload:

• All timestamps are representend in milliseconds since the Unix epoch, which is the default representation of timestamps across all the Livepeer API.

• The healthy top-level condition is a computed state based on some specific conditions of the stream. As of writing this doc, a stream is considered healthy if it's both Active, Transcoding in realtime (TranscodeRealTime), and if multistream targets are configured that they are also healthy (Multistreaming).

• The conditions array contains more specific information about the stream health state, with the type field specifying what the condition means.

• More generally, a Condition represents a specific state of the stream health.

  • The status field represents the value of the last reading for that condition, and the lastProbeTime specifies when that was.

  • The lastTransitionsTime and the frequency fields provide some insight on how it's changing over time.

  • The lastTransitionTime is the timestamp of the last time where the condition status changed, and;

  • The frequency is the ratio of successful probes (status: true) in the specified time ranges preceding lastProbeTime.

• The metrics are still an experimental API and are not recommended for use in production systems. But the last field is basically a tuple containing the timestamp and the value of the last reading for that metric.

Request: GET origin.livepeer.com/data/stream/{id}/events

curl 'https://origin.livepeer.com/data/stream/{id}/events' \
-H 'authorization: Bearer {api-key}'

Response

Content-Type: text/event-stream

retry: 10000

id: b6e0d6a4-0718-4db4-b2ec-31bb23a9fde0
event: lp_event
data: {"type":"media_server_metrics","id":"b6e0d6a4-0718-4db4-b2ec-31bb23a9fde0","timestamp":1635994871604,"streamId":"{id}","nodeId":"prod-mist-blue-server-1","region":"lon","stats":{"mediaTimeMs":1132566},"multistream":[]}

id: 7a0e6f9e-baa1-42c4-a92f-4dfa2f50ac05
event: lp_event
data: {"type":"transcode","id":"7a0e6f9e-baa1-42c4-a92f-4dfa2f50ac05","timestamp":1635994875678,"streamId":"{id}","nodeId":"prod-livepeer-broadcaster-6emu-97d8bf496-7zh9f","segment":{"name":"","seqNo":292,"duration":8.334,"byteSize":2270100},"startTime":1635994874469,"latencyMs":1209,"success":true,"attempts":[{"orchestrator":{"address":"0x1cd98ad89a7d143847f62d2249e4005d09e10648","transcodeUri":"https://vno2-prod-livepeer-orchestrator-7.livepeer.com:443"},"latencyMs":1209,"error":null}]}

id: fd782a49-4986-4b3c-b3b4-9088ae76c152
event: lp_event
data:
{"type":"media_server_metrics","id":"fd782a49-4986-4b3c-b3b4-9088ae76c152","timestamp":1635994881604,"streamId":"{id}","nodeId":"prod-mist-blue-server-1","region":"lon","stats":{"mediaTimeMs":1142550},"multistream":[]}

...

The /events API response implements the Server-Sent Events protocol and can be consumed using any SSE client. On JavaScript, we recomend using the eventsource NPM package to be able to add an Authorization header to the API request.

The individual events sent over the SSE stream are all JSON objects, and are exactly the same low-level objects as published by the services running in the Livepeer infrastructure. Some fields are always present, specifically:

• type: the unique type of the event, which defines the structure of the rest of the JSON object.

• id: a unique identifier for the event, which is automatically used by SSE clients to recover a connection without losing events.

• timestamp: the timestamp of the event, in milliseconds since the Unix epoch.

You can also get historic events and/or within a time range, by specifying the from and to query parameters with unix millis timestamps. For example:

• /data/stream/{id}/events?from=1635437394000 to get all events since Oct 28 16:09:54Z 2021 (exclusively) and stay connected for live events.

• /data/stream/{id}/events?from=1635437394000&to=1635447394000 to get all events between Oct 28 16:09:54Z 2021 (exclusively) and Oct 28 18:56:34Z 2021 (inclusively).

Finally, we recomend using the events API mostly for debugging purposes, for example to get all available information about a live stream to find any issues. The specific events that are published and their respective schemas will keep evolving over time so are still subject to changes.