Network Working Group F. Le Faucheur Internet-Draft M. Viveganandhan Intended status: Informational K. Leung Expires: August 26, 2012 Cisco February 23, 2012 CDNI Logging Formats for HTTP and HTTP Adaptive Streaming Deliveries draft-lefaucheur-cdni-logging-delivery-00 Abstract The interconnection of Content Distribution Networks (CDNs) is motivated by several use cases. CDN Interconnection can be achieved through four CDNI interfaces, one of which is the CDNI Logging interface. This document discusses CDNI logging formats for content deliveries performed using HTTP or HTTP adaptive streaming. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on August 26, 2012. Copyright Notice Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents Le Faucheur, et al. Expires August 26, 2012 [Page 1] Internet-Draft CDNI Logging for HTTP Delivery February 2012 (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 2. CDNI Logging for regular HTTP Delivery . . . . . . . . . . . . 4 2.1. Regular HTTP Log Triggers . . . . . . . . . . . . . . . . 4 2.2. Regular HTTP Log Fields . . . . . . . . . . . . . . . . . 5 3. CDNI Logging for HTTP Adaptive Streaming . . . . . . . . . . . 6 3.1. HTTP Adaptive Logging Session . . . . . . . . . . . . . . 6 3.2. CDNI Log formats for HTTP Adaptive Streaming . . . . . . . 7 3.2.1. Segment-Based Logging . . . . . . . . . . . . . . . . 7 3.2.2. Event-Based Logging . . . . . . . . . . . . . . . . . 9 3.2.3. Summary-Based Logging . . . . . . . . . . . . . . . . 10 4. Performance Monitoring . . . . . . . . . . . . . . . . . . . . 10 5. CDNI Log Encoding and Transport . . . . . . . . . . . . . . . 11 5.1. CDNI Log Header Information . . . . . . . . . . . . . . . 11 5.2. CDNI per-Log-Record Information . . . . . . . . . . . . . 11 5.3. CDNI Log Footer Information . . . . . . . . . . . . . . . 12 6. Impact on CDNI Metadata Requirements . . . . . . . . . . . . . 12 7. Generation of CDNI Logs . . . . . . . . . . . . . . . . . . . 12 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 9. Security Considerations . . . . . . . . . . . . . . . . . . . 13 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 13 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 13 11.1. Normative References . . . . . . . . . . . . . . . . . . . 13 11.2. Informative References . . . . . . . . . . . . . . . . . . 13 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 14 Le Faucheur, et al. Expires August 26, 2012 [Page 2] Internet-Draft CDNI Logging for HTTP Delivery February 2012 1. Introduction The interconnection of Content Distribution Networks (CDNs) is motivated by several use cases, such as those described in [I-D.ietf-cdni-use-cases]. The overall problem space for CDN Interconnection (CDNI) is described in [I-D.ietf-cdni-problem-statement]. The CDN Interconnection framework is defined in [I-D.davie-cdni-framework] and the requirements for the CDN Interconnection solution are specified in [I-D.ietf-cdni-requirements]. One of the CDNI interfaces defined in these documents is the CDNI Logging interface whose description is quoted here: "o CDNI Logging interface: This interface allows the Logging system in interconnected CDNs to communicate the relevant activity logs in order to allow log consuming applications to operate in a multi-CDN environments. For example, an upstream CDN may collect delivery logs from a downstream CDN in order to perform consolidated charging of the CSP or for settlement purposes across CDNs. Similarly, an upstream CDN may collect delivery logs from a downstream CDN in order to provide consolidated reporting and monitoring to the CSP." The logging interface is also discussed in further details in [I-D.bertrand-cdni-logging]. As discussed in the CDNI framework [I-D.davie-cdni-framework] logs associated with content delivery using HTTP adaptive streaming have specific requirements in terms of summarization: "Most schemes to deliver HTTP-based adaptive bit-rate video use a large number of relatively small HTTP requests (e.g., one request per 3-second chunk of video.) It may be desirable to aggregate Logging information so that a single log entry is provided for the entire video rather than for each chunk. Note however that such aggregation requires a degree of application awareness in dCDN to recognize that the many HTTP requests correspond to a single video." The prime purpose of this document is to define CDNI Logging formats corresponding to content delivery that uses HTTP or HTTP adaptive streaming. Section 2 discusses the aspect of CDNI Logging that are specific to content delivery performed via regular HTTP. Section 3 discusses the aspect of CDNI Logging that are specific to content delivery performed via HTTP adaptive streaming. Section 5 discusses the aspect of CDNI Logging that are generic to all the CDNI Logs for delivery. Le Faucheur, et al. Expires August 26, 2012 [Page 3] Internet-Draft CDNI Logging for HTTP Delivery February 2012 CDNI Logging for other events than content delivery (e.g. failures, request routing, service monitoring,...) will be discussed in subsequent versions. 1.1. Terminology This document uses all terminology defined in [I-D.ietf-cdni-problem-statement]. In addition, we introduce the following terminology: o HTTP Adaptive Streaming Segment (or Segment): piece of a content item that is accessible as a standalone HTTP resource, and thus is discreetly accessible via its own Unique Resource Identifier (URI). o HTTP Adaptive Streaming Logging Session (or Session): state maintained by the delivering CDN allowing association, for logging purposes, of the actions (e.g. delivery of Segments) corresponding to a given content request by a given End User. Note that actions associated with a given Session may all be performed by a single Surrogate of the delivering CDN, or may be performed by different Surrogates of the delivering CDN successively (e.g. in case of surrogate fail-over) or performed by different Surrogates in parallel (e.g. in case of load-balancing of Segments delivery). o Adaptive Streaming Manifest File (or Manifest File): file that indicates to the End-User the information (e.g. content representations available, rules for mapping between a content Segment and its URI) necessary to request the appropriate Segment at the appropriate time based on the End-User adaptive logic (e.g. increase bitrate, decrease bitrate) and behavior (e.g. seek, pause, play). 2. CDNI Logging for regular HTTP Delivery This section discusses the triggers on which a CDNI log is generated for delivery of content using regular HTTP delivery, as well as the fields contained in the corresponding log. 2.1. Regular HTTP Log Triggers A CDNI Log is generated for a regular HTTP delivery on the following triggers: Le Faucheur, et al. Expires August 26, 2012 [Page 4] Internet-Draft CDNI Logging for HTTP Delivery February 2012 +-----------------+-------------------------------------------------+ | Event | Description | +-----------------+-------------------------------------------------+ | content Request | Reception and processing of a request for a | | | content | +-----------------+-------------------------------------------------+ 2.2. Regular HTTP Log Fields A CDNI Log for regular HTTP delivery contains the following fields: +---------------+------------+--------------------------------------+ | Field | Descriptio | Examples | | | n | | +---------------+------------+--------------------------------------+ | Current-Time | Time, in | [20/Feb/2012:00:29.510+0200] | | | millisecon | | | | ds, at | | | | which the | | | | request | | | | was | | | | received. | | | Time-to-Serve | Time, in | 952195 | | | microsecon | | | | ds, taken | | | | to | | | | complete | | | | the | | | | request. | | | Client-IP | IP address | 64.103.25.105 | | | of the | | | | requesting | | | | client. | | | Action | Squid | TCP_HIT, TCP_MISS, ... | | | action | | | | describing | | | | how the | | | | request | | | | was | | | | treated | | | | locally | | | | descriptio | | | | n(e.g., | | | | cache | | | | hit/miss) | | | | . | | Le Faucheur, et al. Expires August 26, 2012 [Page 5] Internet-Draft CDNI Logging for HTTP Delivery February 2012 | Status-Return | HTTP | 200, 404, ... | | ed | response | | | | code. | | | Bytes-Transfe | Bytes sent | 23567992 | | rred | to the | | | | client, | | | | including | | | | the | | | | headers. | | | Method | HTTP | GET | | | request | | | | method. | | | URI | URI of | http://cache3.cdn1.com/movie/ice/ice | | | requested | movie.mpg | | | content. | | | Content-Type | MIME-Type | video/mpeg | | | from the | | | | reply | | | | header. | | +---------------+------------+--------------------------------------+ 3. CDNI Logging for HTTP Adaptive Streaming This section discusses the CDNI logs for delivery of content using HTTP adaptive streaming. 3.1. HTTP Adaptive Logging Session There are multiple methods by which the delivering CDN can create/ maintain/delete an HTTP Adaptive Streaming Logging Session. As a first example method, the delivering CDN may create a Session on successful authorization of delivery of the Manifest File for a particular End-User content request, create a per-Session coookie and then validate the association of Segment requests with the Session through HTTP cookie mechanisms. The delivering CDN may then maintain the Session as long as Segment requests are being received for that Session. The delivering CDN may ultimately delete the Session once Segment requests for that Session are no longer received for some period of time. As another example, Session identification information included in the fragment URIs may be used by the delivering CDN to associate Segment requests with a Session and therefore for determining when a Session is to be created/maintained/deleted. Methods for tracking HTTP Adaptive Streaming Logging Sessions in a Le Faucheur, et al. Expires August 26, 2012 [Page 6] Internet-Draft CDNI Logging for HTTP Delivery February 2012 delivering CDN are implementation specific and thus outside the scope of the present document. 3.2. CDNI Log formats for HTTP Adaptive Streaming Three CDNI Logging formats are defined for delivery using HTTP adaptive streaming: o Segment-Based Logging, as specified in Section 3.2.1. o Event-Based Logging, as specified in Section 3.2.2. o Summary-based Logging, as specified in Section 3.2.3. An implementation of the CDNI Logging interface MUST support logging for delivery of content using HTTP adaptive streaming in the Segment- Based Logging format and the Event-Based Logging format, and MAY support logging in the Summary-Based Logging format. By default (i.e. unless it is explicitly requested that another format be used as discussed in Section 6), CDNI logs for delivery of content using HTTP adaptive streaming MUST be formatted as per the Event-Based format. 3.2.1. Segment-Based Logging 3.2.1.1. Segment-Based Log Triggers A Segment-Based Log is generated on the following triggers: +---------------------+---------------------------------------------+ | Event | Description | +---------------------+---------------------------------------------+ | Segment/Manifest | Reception of a request for a Segment (or | | Request | Manifest File) | +---------------------+---------------------------------------------+ 3.2.1.2. Segment-Based Log fields A Segment-based Log for HTTP adaptive streaming contains all the same fields as a CDNI Log for a regular HTTP request as well as the following additional fields: Le Faucheur, et al. Expires August 26, 2012 [Page 7] Internet-Draft CDNI Logging for HTTP Delivery February 2012 +-----------------+-------------+-----------------------------------+ | Field | Description | Examples | +-----------------+-------------+-----------------------------------+ | abr-protocol | Type of | hls,hds,hss | | | HTTP | | | | adaptive | | | | streaming. | | | Session-id | a string | 6141F5795BE774691D234A0465B9667A | | | generated | | | | by the | | | | delivering | | | | CDN and | | | | unique (to | | | | the | | | | delivering | | | | CDN) to | | | | identify | | | | the | | | | Session. | | | Representation | Characteris | 300000bps | | | tics of the | | | | content | | | | representa | | | | tion (e.g., | | | | bitrate). | | | manifest-uri | URI of the | http://cache3.cdn1.com/movie/ice/ | | | Manifest | index300k.m3u8 | | | File. | | | content-id | Identifier | http://cache3.cdn1.com/movie/ice/ | | | for the | index.m3u8 | | | content | | | | item | | | | independent | | | | ly of its | | | | representa | | | | tions. | | | Session-trackin | The method | cookie, uri_query | | g-method | used in the | | | | delivering | | | | CDN for | | | | tracking of | | | | the | | | | Session. | | +-----------------+-------------+-----------------------------------+ Le Faucheur, et al. Expires August 26, 2012 [Page 8] Internet-Draft CDNI Logging for HTTP Delivery February 2012 3.2.2. Event-Based Logging 3.2.2.1. Event-Based Log Triggers An Event-Based Log is generated on the following triggers: +----------------+--------------------------------------------------+ | Event | Description | +----------------+--------------------------------------------------+ | Representation | Reception of a request for a Segment whose | | Change/Bitrate | bitrate is higher than the bitrate of the | | Upshift. | previously requested Segment. | | Representation | Reception of a request for a Segment whose | | Change/Bitrate | bitrate is lower than the bitrate of the | | Downshift. | previously requested Segment. | | Representation | TBD | | Change/TBD. | | | Session Seek. | Reception of a request for a Segment whose | | | playtime is not sequentially following the | | | previously requested Segment. | | Session Stop. | Determination by delivering CDN that the | | | End-User has temporarily stopped requesting | | | Segments associated with the Session. This may | | | be determined using heuristics such as | | | expiration of a time-out (e.g. of 10 seconds) | | | after reception of the last Segment request. | | Session Close. | Determination by delivering CDN that the | | | End-User has definitively stopped requesting | | | Segments associated with the Session. This may | | | be determined using heuristics such as | | | expiration of a time-out after any End-USer | | | activity associated with the Session. | +----------------+--------------------------------------------------+ 3.2.2.2. Event-Based Log Fields An Event-Based Log for HTTP adaptive streaming contains all the same fields as a Segment-Based Log as well as the following additional fields: +------------+----------------------+-------------------------------+ | Field | Description | Examples | +------------+----------------------+-------------------------------+ | event-code | Event that triggered | Representation Change/Bitrate | | | the generation of | Upshift, Session Seek, | | | the log | Session Stop, ... | +------------+----------------------+-------------------------------+ Le Faucheur, et al. Expires August 26, 2012 [Page 9] Internet-Draft CDNI Logging for HTTP Delivery February 2012 3.2.3. Summary-Based Logging 3.2.3.1. Summary-Based Log Triggers A Summary-Based Log is generated on the following triggers: +--------------+----------------------------------------------------+ | Event | Description | +--------------+----------------------------------------------------+ | Session | Determination by delivering CDN that the End-User | | Close | has definitively stopped requesting Segments | | | associated with the Session. This may be | | | determined using heuristics such as expiration of | | | a time-out after any End-USer activity associated | | | with the Session. | | Intermediate | Expiration of a Logging Period Time-out since a | | Logging | previous Summary-Based Log was generated for that | | Period | Session (or since the start of the Session). | +--------------+----------------------------------------------------+ 3.2.3.2. Summary-Based Log Fields An Event-Based Log for HTTP adaptive streaming contains all the same fields as a Segment-Based Log, minus the Representation field, plus the following additional fields: [This will be specified in subsequent versions. The objective is to define fields that summarize succinctly -while retaining enough information- the fluctuations in content representation (e.g. bitrate/quality variations) as well as stop/start/seek patterns over the duration of the Session (or over the Intermediate Logging period) for the audio and video components. ] 4. Performance Monitoring The CDNI Log fields listed in the previous sections allow monitoring of essential delivery performance indicators across the CDN Mesh. For example, for a regular HTTP delivery, these fields allow tracking of the time taken to serve the requested content, tracking of content delivery failures, tracking of partial deliveries and tracking of cache hit ratios. As another example, for HTTP adaptive streaming, these fields allow tracking of the presentation (and its fluctuation over time) served to the End-User as well as the End-User random content access (e.g. Play/Stop/Seek). Subsequent versions of this document will discuss potential additional log fields for enhanced performance monitoring. Le Faucheur, et al. Expires August 26, 2012 [Page 10] Internet-Draft CDNI Logging for HTTP Delivery February 2012 5. CDNI Log Encoding and Transport Details for CDNI Log encoding and transport will be specified in subsequent versions. However, Section 5.1 identifies the information that is to be included in the header of CDNI Logs, Section 5.2 identifies information that is to be attached to every CDNI Log record and Section 5.3 identifies the information that is to be included in the footer of CDNI Logs. 5.1. CDNI Log Header Information The header of CDNI Logs contains the following fields: +----------------+-------------------+------------------------------+ | Field | Description | Examples | +----------------+-------------------+------------------------------+ | Format-Version | Version of the | v1.0 | | | CDNI Log format. | | | Log-Type | The type of CDNI | Delivery_HTTP, | | | Log. | Delivery_Adaptive_Segment, | | | | Delivery_Adaptive_Event | | Log-ID | Unique identifier | | | | for the CDNI Log | | | | (facilitates | | | | detection of | | | | duplicate Logs | | | | and tracking in | | | | case of | | | | aggregation). | | | Log-Timestamp | Time, in | [20/Feb/2012:00:29.510+0200] | | | milliseconds, the | | | | CDNI Log was | | | | generated. | | +----------------+-------------------+------------------------------+ 5.2. CDNI per-Log-Record Information In addition to the log fields discussed in previous sections, each CDNI log record contains the following fields: +----------+---------------------------------------------+----------+ | Field | Description | Examples | +----------+---------------------------------------------+----------+ | Log | Digest of the Log records (facilitates | | | Record | recovery of uncorrupted Log records inside | | | Digest | a corrupted CDNI Log) | | +----------+---------------------------------------------+----------+ Le Faucheur, et al. Expires August 26, 2012 [Page 11] Internet-Draft CDNI Logging for HTTP Delivery February 2012 5.3. CDNI Log Footer Information The footer of CDNI Logs contains the following fields: +---------+----------------------------------------------+----------+ | Field | Description | Examples | +---------+----------------------------------------------+----------+ | Log | Digest of the complete Log (facilitates | | | Digest | detection of Log corruption) | | +---------+----------------------------------------------+----------+ 6. Impact on CDNI Metadata Requirements We request that the following requirement be added in section 6 of [I-D.ietf-cdni-requirements]: " META-X [HIGH] The CDNI Metadata Distribution interface shall support an optional mechanism allowing the Upstream CDN to indicate to the Downstream CDN which CDNI Log format is to be provided for all, for specific sets of, or for specific content items delivered using HTTP adaptive streaming. ". A CDNI implementation that does not support this optional CDNI Metadata Distribution Interface mechanism MUST ignore this log format indication and generate CDNI logging format for adaptive streaming using the default CDNI Logging format for such delivery. 7. Generation of CDNI Logs Like other CDNI interfaces, the CDNI Logging interface specifies operations across CDNs and not inside a CDN. Therefore, the log formats specified in this document apply to CDNI logging information exchanged across CDNs and does not constrain the process for generating such inter-CDN logs within a given CDN. The format of the logs generated by a given CDN Surrogate is beyond the scope of the present document. We observe that a given CDN could elect to have its Surrogates natively generate logs in the same format as the one to be used for exchange with another CDN, or that the CDN could elect to have its Surrogates generate logs in any other format (as long as they include the necessary information) and have these logs then reformatted prior to exchange with another CDN. Le Faucheur, et al. Expires August 26, 2012 [Page 12] Internet-Draft CDNI Logging for HTTP Delivery February 2012 8. IANA Considerations [This will be specified in subsequent versions]. 9. Security Considerations CDNI Logs exchanged over the CDNI Logging interface can be consumed by very sensitive applications including inter-CDN accounting and billing. The associated security concerns are discussed in [I-D.davie-cdni-framework]. At this stage, we observe that the CDNI Logging interface can leverage the existing security mechanisms supported by the underlying transport protocol that will be selected for transport of CDNI Logs (e.g. to support authentication of the entities exchanging CDNI Logs through the CDNI Logging interface, to support privacy and protection against spoofing). This will be further discussed in subsequent versions of this document. 10. Acknowledgements The authors want to thank Gene Halbrooks for his input into this document. 11. References 11.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 11.2. Informative References [I-D.bertrand-cdni-logging] Gilles, B. and S. Emile, "CDNI Logging Interface", draft-bertrand-cdni-logging-00 (work in progress), February 2012. [I-D.davie-cdni-framework] Davie, B. and L. Peterson, "Framework for CDN Interconnection", draft-davie-cdni-framework-01 (work in progress), October 2011. [I-D.ietf-cdni-problem-statement] Niven-Jenkins, B., Faucheur, F., and N. Bitar, "Content Distribution Network Interconnection (CDNI) Problem Statement", draft-ietf-cdni-problem-statement-03 (work in Le Faucheur, et al. Expires August 26, 2012 [Page 13] Internet-Draft CDNI Logging for HTTP Delivery February 2012 progress), January 2012. [I-D.ietf-cdni-requirements] Leung, K. and Y. Lee, "Content Distribution Network Interconnection (CDNI) Requirements", draft-ietf-cdni-requirements-02 (work in progress), December 2011. [I-D.ietf-cdni-use-cases] Gilles, B., Emile, S., Watson, G., Burbridge, T., Eardley, P., and K. Ma, "Use Cases for Content Delivery Network Interconnection", draft-ietf-cdni-use-cases-03 (work in progress), January 2012. Authors' Addresses Francois Le Faucheur Cisco Systems Greenside, 400 Avenue de Roumanille Sophia Antipolis 06410 France Phone: +33 4 97 23 26 19 Email: flefauch@cisco.com Mahesh Viveganandhan Cisco Systems 375 East Tasman Drive San Jose 95134 USA Email: mvittal@cisco.com Kent Leung Cisco Systems 3625 Cisco Way San Jose 95134 USA Phone: +1 408 526 5030 Email: kleung@cisco.com Le Faucheur, et al. Expires August 26, 2012 [Page 14]