Format for the Session Initiation
Protocol (SIP) Common Log Format (CLF)Cisco Systems7200-12 Kit Creek RoadResearch Triangle ParkNC27709USgsalguei@cisco.comBell Labs, Alcatel-Lucent1960 Lucent LaneRm 9C-533NapervilleIL60563USvkg@bell-labs.comTekelec17210 Campbell Rd.Suite 250DallasTX75252USadam@nostrum.com
Real Time Applications and Infrastructure
SIPCLFThe SIPCLF Workgroup has defined a common log format framework for
Session Initiation Protocol (SIP) servers. This common log format mimics
the successful event logging format found in well-known web
servers like Apache and web proxies like Squid. This document proposes
an indexed text encoding format for the SIP Common Log Format (CLF) that
retains the key advantages of a text-based format, while significantly
increasing processing performance over a purely text-based
implementation. This file format adheres to the SIP CLF data model and
provides an effective encoding scheme for all mandatory and optional
fields that appear in a SIP CLF record.The extensive list of benefits and the widespread adoption of the
Apache Common Log Format (CLF) has prompted the development of a
functionally equivalent event logging mechanism for the Session
Initiation Protocol (SIP). Implementing a
logging scheme for SIP is a considerable challenge. This is due in part
to the fact that the behavior of a SIP entity is more complex as
compared to an HTTP entity. Additionally, there are shortcomings to the
purely text-based HTTP Common Log Format that need to be addressed in
order to allow for real-time inspection of SIP log files . Experience
with Apache Common Log Format has shown that dealing with large
quantities of log data can be very processor intensive, as doing so
necessarily requires reading and parsing every byte in the log file(s)
of interest.An implementation independent framework for the SIP CLF has been
defined in .
This memo describes an indexed text file format for logging SIP messages
received and sent by SIP clients, servers, and proxies that adheres to
the data model presented in Section 8 of . This document
defines a format that is no more difficult to generate by logging
entities, while being radically faster to process. In particular, the
format is optimized for both rapidly scanning through log records, as
well as quickly locating commonly accessed data fields.Further, the format proposed by this document retains the key
advantage of being human readable and able to be processed using the
various Unix text processing tools, such as sed, awk, perl, cut, and
grep.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in . defines additional terms used in this
document that are specific to the SIP domain such as "proxy";
"registrar"; "redirect server"; "user agent server" or "UAS"; "user
agent client" or "UAC"; "back-to-back user agent" or "B2BUA"; "dialog";
"transaction"; "server transaction".This document uses the term "SIP Server" that is defined to include
the following SIP entities: user agent server, registrar, redirect
server, a SIP proxy in the role of user agent server, and a B2BUA in the
role of a user agent server.The reader is expected to be familiar with the terminology and
concepts defined in .This document defines the logging syntax for the SIP CLF. This syntax
is demonstrated through the use of various examples. The formatting
described here does not permit these examples to be unambiguously
rendered due to the constraints imposed by the formatting rules for
Internet-Drafts. To avoid ambiguity and to meet the Internet-Draft
layout requirements this document uses the <allOneLine/> markup
convention established in .For the sake of clarity and completeness, the entire text defining
this markup convention from Section 2.1 of is quoted below:Several of these examples contain unfolded lines longer than 72
characters. These are captured between <allOneLine/> tags. The
single unfolded line is reconstructed by directly concatenating all
lines appearing between the tags (discarding any line feeds or
carriage returns). There will be no whitespace at the end of lines.
Any whitespace appearing at a fold-point will appear at the
beginning of a line.The following represent the same string of bits:Header-name: first value, reallylongsecondvalue, third
value<allOneLine>Header-name: first value, reallylongsecondvalue, third value</allOneLine><allOneLine>Header-name: first value, reallylongsecondvalue, third value</allOneLine>Note that this is NOT SIP header-line folding, where different
strings of bits have equivalent meaning.The IP addresses used in the examples in this document adhere to the
best practices outlined in and correspond
to the documentation address block 192.0.2.0/24 (TEST-NET-1) as
described in .The Common Log Format for the Session Initiation Protocol defines a data model
to which this logging format format adheres. Each SIP CLF record MUST
consist of all the mandatory data model elements outlined in Section 8.1
of .The format presented in is
for a single SIP CLF log entry. While there is no actual subdivision in
practice, this format can be logically subdivided into the following
three distinct components: 1. Index Pointers - The first 60-bytes of this format. This
portion is metadata, primarily composed of a list of pointers that
indicate the beginning of both the variable length mandatory and
optional fields that are logged as part of this record. These
pointers are implemented as a mechanism to improve processing of
these records and to allow a reader to expeditiously skip directly to
the desired field without unnecessarily going through the entire
record. This logical subdivision within the SIP CLF format will be
referenced in this document with the <IndexPointers> tag. A 0x0A (LF character) delimits <IndexPointers> from the next logical grouping.2. Mandatory Fields - The next logical grouping in this format is
a tab (0x09) delimited listing of the mandatory fields as described in
Section 8.1 of and in the order
listed in <IndexPointers>. This logical subdivision within the
SIP CLF format will be referenced in this document with the
<MandatoryFields> tag.3. Optional Fields - The last logical component MAY be present as
it is an OPTIONAL extension to the SIP CLF format. Its purpose is to
provide flexibility to the developer of this SIP CLF to log any
desired fields not included in <MandatoryFields>. This
includes SIP bodies and any vendor-specific extensions. This logical
subdivision within the SIP CLF format will be referenced in this
document with the <OptionalFields> tag.Note that and plus the terminating line-feed (0x0A) at the end of the SIP CLF record are different representations of the same format but are functionally equivalent. The representation of this format is a two line record where the <IndexPointers> metadata is on one line and the actual data like <MandatoryFields> and
<OptionalFields> (if present) is on another. In the following sections note that
indications of "hexadecimal encoded" indicate that the value is to be
written out in human-readable base-16 numbers using the ASCII characters
0x30 through 0x39 ('0' through '9') and 0x41 through 0x46 ('A' through
'F'). Similarly, indications of "decimal encoded" indicate that the
value is to be written out in human readable base-10 number using the
ASCII characters 0x30 through 0x39 ('0' through '9'). In both encodings,
numbers always take up the number of bytes indicated, and are padded on
the left with ASCII '0' (zero) characters to fill the entire space.The fields that make up <IndexPointers> are described
below:0x41 for this document;
hexadecimal encoded.Hexadecimal encoded total
length of this log record, including "Version", "Record Length",
"Flags" fields and terminating line-feed.Bytes 8 through 55 contain hexadecimal encoded pointers that point
to the starting location of each of the variable-length mandatory
fields. Note that there are no delimiters between these pointer values
-- they are packed together as a single, 52-character hexadecimal
encoded string. The "Pointer" fields indicate absolute byte values
within the record, and MUST be >=82. They point to the start of the
corresponding value within the <MandatoryFields> portion. A
description of each of the mandatory fields that these pointer values
point to can be found in .This final pointer
indicates the location within the SIP CLF record where the
OPTIONAL group of <OptionalFields> begin, if present. The
"Optional Fields Start Pointer" points to the ASCII Tab (0x09)
character for the first entry in the <OptionalFields>
portion. If the OPTIONAL group of <OptionalFields> are not
implemented, then the "Optional Fields Start Pointer" field MUST
point to the terminating line-feed (0x0A) at the end of the SIP
CLF record.Following the pointers in <IndexPointers>, two fixed-length
fields are encoded to specify the exact time of the log entry. As
before, all fields are completely filled, pre-pending values with '0'
characters as necessary.Date and time of the request
or response represented as the number of seconds since the Unix
epoch (i.e. seconds since midnight, January 1st, 1970, GMT). Represented in big-endian fashion with most significant octet first from zero starting at the left, or high-order, position. Decimal encoded.Fractional seconds
portion of the Timestamp field to millisecond accuracy. Represented in big-endian fashion with most significant octet first from zero starting at the left, or high-order, position. Decimal
encoded.Request/Response flagRetransmission flagSent/Received flagTransport flagEncryption flagAfter the "Timestamp", "Fractional Seconds" and the "Flags" fields
are the actual values for the mandatory fields specified in Section
8.1 of , which
are described below:The Command Sequence header field, including
the CSeq number and method name.Set to the value of the SIP
response status code for responses. Set to a single ASCII dash
(0x2D) for requests.The Request-URI in the start line (mandatory
in request), including any URI parameters.The IP address of the
downstream server, including the port number. For IPv4 addresses
the port number MUST be separated from the IP address by a single
':'. IPv6 addresses are represented using the bracket notation
detailed in Section 6 of . That is,
the IPv6 address enclosed in square brackets and separated from
the port number by a single ':').The IP address of the
upstream client, including the port number over which the SIP
message was received. For IPv4 addresses the port number MUST be
separated from the IP address by a single ':'. IPv6 addresses are
represented using the bracket notation detailed in Section 6 of
. That is, the IPv6 address enclosed
in square brackets and separated from the port number by a single
':').Value of the URI in the To header field.Value of the tag parameter (if present) in
the To header field.Value of the URI in the From header
field.Value of the tag parameter in the From
header field.Whilst one may question the value of the From URI in light of , the From URI, nonetheless, imparts some
information. For one, the From tag is important and, in the case of a
REGISTER request, the From URI can provide information on whether this
was a third-party registration or a first-party one.The value of the Call-ID header field.Server transaction identification code -
the transaction identifier associated with the server transaction.
Implementations can reuse the server transaction identifier (the
topmost branch-id of the incoming request, with or without the
magic cookie), or they could generate a unique identification
string for a server transaction (this identifier needs to be
locally unique to the server only.) This identifier is used to
correlate ACKs and CANCELs to an INVITE transaction; it is also
used to aid in forking. (See Section 9.4 of for usage.)Client transaction identification code -
this field is used to associate client transactions with a server
transaction for forking proxies or B2BUAs. Upon forking,
implementations can reuse the value they inserted into the topmost
Via header's branch parameter, or they can generate a unique
identification string for the client transaction. (See Section 9.4
of for
usage.)This data MUST appear in the order listed in <IndexPointers>,
and each field MUST be present. Fields are subject the maximum SIP CLF
field size of 4096 bytes as detailed in Section 8 of and are separated
by a single ASCII Tab character (0x09). Any Tab characters present in
the data to be written will be replaced by an ASCII space character
(0x20) prior to being logged.An element will not always have an appropriate value to provide for one of these fields, even when the field is required to appear in the SIP CLF record. In such circumstances, when a given mandatory field is not present then that empty field MUST be encoded as a single horizontal dash ("-").In the event that a field failed to parse it MUST be encoded as a
single question mark ("?"). If these characters are part of a sequence
of other characters, then there is no ambiguity. If the field being
logged contains only one character, and that character is the literal
"-", the implementation SHOULD insert an escaped %2D for that field in
the SIP CLF record. Similarly, if the field contains only one
character, and that character is the literal "?", the implementation
SHOULD insert an escaped %3F for that field in the SIP CLF record.Optional fields are those SIP message elements that are not a part
of the mandatory fields list detailed in Section 8.1 of . After the
<MandatoryFields> section, there is an OPTIONAL
<OptionalFields> group (shown in ) that MAY appear zero or more times.
This <OptionalFields> group provides extensibility to the SIP
CLF. It allows SIP CLF implementers the flexibility to extend the
logging capability of the indexed-ASCII representation beyond just the
mandatory log elements described in Section 8.1 of .Logging any optional SIP elements MUST be done according to the
format shown in . The location
of the start of <OptionalFields> within the SIP CLF record is
indicated by the "Optional Fields Start Pointer" field in
<IndexPointers>. After the initial Tab delimiter byte (0x09)
shown in , the optional field
being logged is generally represented by the notation:Tag@Vendor-ID,Length,ValueThe optional field identifier (Tag@Vendor-ID) is composed of a two
byte Tag and an eight byte Vendor-ID (both decimal encoded) separated
by an "@" character (0x40). This uniquely identifies the optional
field being logged. The format for this identifier is loosely modeled
after the private use option used by the Syslog protocol (Note: this is the second format detailed in
Section 6.3.2 of ). It makes use of the
Private Enterprise Number (PEN), which provides an identifier through
a globally unique name space . This syntax
provides the necessary extensibility to SIP CLF to allow logging of
any SIP header, body, as well as any vendor-specified SIP element.Optional fields are logged according to the following two syntax
rules:Vendor-ID = 00000000A Vendor-ID of zero is used to log the entire SIP message,
message body, Reason-Phrase or any SIP header fields that are not
a part of the mandatory fields list detailed in Section 8.1 of
. The
following Tag values are used to identify which of these optional
elements are being logged:When logging a SIP Header Field (Tag=00), the associated
"Value" field MUST be populated by the entire header field
being logged. That is, the field-name, the associated colon
(":") and the field-value. This mechanism provides the
capability to optionally log any SIP header field by
identifying the field being logged within the "Value"
field.Because the Reason-Phrase in a response is part of the
Status-Line and is not identified with a field-name, it is a
special case. In this instance, the the associated "Value"
field MUST be populated by the name "Reason-Phrase" followed
by a colon (":") and a single space (SP) between the colon and
the logged Reason-Phrase value.The corresponding "Length" field includes the length of the
entire "Value" field. This includes the field-name, the colon,
and any LWS separator.If an optional field occurs more than once in a SIP message
(e.g. Contact, Route, Record-Route, etc.) then each occurrence
MUST be logged separately with same Tag value.SIP message bodies with the following body types can be
optionally logged:Session Description Protocol (SDP) (Content-Type:
application/sdp)Extensible Markup Language (XML) payloads
(Content-Type: application/*+xml)binary (Content-Type: application/{isup,qsig})miscellaneous text content (Content-Type:
message/sipfrag, message/http, text/plain, ...)When logging a message body (Tag=01), the associated
"Value" field is populated with the Content-Type itself plus
the SIP message body separated with a linear white space (LWS)
separator. In this manner, everything about all four body
types is self-described using a single tag as compared to
enumerating a separate tag for each body type. Additionally,
the corresponding "Length" field includes the SIP message
body, the length of the embedded Content-Type, and the LWS
separator between the MIME type and the body content. Note
that binary bodies would have to be byte encoded to render
them in the ASCII file.Logging the message body (Tag=01) or the entire SIP message
(Tag=02) MUST conform to the maximum size limitation of 4096
bytes for a SIP CLF field, as detailed in Section 8 of . These can
be repeated multiple times to accommodate SIP messages or
bodies that exceed 4096 bytes in length.Vendor-ID = PENA Vendor-ID set to a vendor's own private enterprise number
from the complete current list of private enterprise numbers
maintained by IANA is used to log any
other vendor-specified optional element of a SIP header or body.
The value of the Tag is set at the discretion of the
implementer:The remaining fields in the format shown in are defined below:Indicates the length of only
the "Value" field of this optionally logged element, hexadecimal
encoded. This length does not include the header shown in .Contains the actual
value of this optional field. As with the mandatory fields, ASCII
Tab characters (0x09) are replaced with ASCII space characters
(0x20).The following are examples of optionally logged SIP elements using
the syntax described in this section. All these examples only show the
<OptionalFields> portion of the SIP CLF record. The mandatory
<IndexPointers> and <MandatoryFields> portions of the SIP
CLF are intentionally omitted for the sake of brevity. Note that all
of these examples of optionally logged fields begin with a leading Tab
delimiter byte (0x09) that is not apparent here.Contact header field logged as an optional field: The Contact header field would
be logged as an optional field in the following manner: Reason-Phrase logged as an optional field: SDP body to be logged as an optional field: This body has a Content-Type of application/sdp and is
of length of 123 bytes including all the line-feeds. When logging
this body the "Value" field is composed of the Content-Type and
the body separated by a LWS, which gives it a combined length of
139 (0x008B) bytes. This SIP body would be logged as an optional
field in the following manner: Note that the body is actually logged on a single line
and are thus captured between <allOneLine/> tags. The
line-feeds are escaped using \r\n to delimit the various lines in
the message body.Codec information from the SDP body logged as an optional
field: A vendor may choose to log a SIP message element such
as the codec information from the SDP body. This Vendor-specified
SIP element would be logged as an optional field in the following
manner: N-th message received from a particular peer logged as an
optional field: Which would signify that this is
the 1,877th message from the peering partner example.com. Note
that the previous two examples showing an optionally logged
Vendor-specified SIP element use a Vendor-ID with a Private
Enterprise Number of 32473. This value has been reserved by IANA
to be used as an example PEN in documentation according to .The following SIP message is an INVITE request sent by a SIP client:
Shown below is approximately how this message would appear
as a single record in a SIP CLF logging file if encoded according to the
syntax described in this document. Due to internet-draft conventions,
this log entry has been split into five lines, instead of the two lines
that actually appear in a log file; and the tab characters have been
padded out using spaces to simulate their appearance in a text terminal.
A Base64 encoded version of this log entry (without the changes required to format it for an Internet-Draft) is shown below.
To recover the unencoded file, the Base64 text above may be passed as input to the following perl script (the output should be redirected to a file).
This format has been designed to allow text tools to easily process
logs without needing to understand the indexing format. Index lines may
be rapidly discarded by checking the first character of the line: index
lines will always start with an alphabetical character, while field
lines will start with a numerical character.Within a field line, script tools can quickly split fields at the tab
characters. The first 12 fields are positional, and the meaning of any
subsequent fields can be determined by checking the first four
characters of the field. Alternately, these non-positional fields can be
located using a regular expression. For example, the "Contact value" in
a request can be found by searching for the perl regex
/\t0000,....,([^\t]*)/.This document does not introduce any new security considerations
beyond those discussed in .SIP CLF log files will take up substantive amount of disk space
depending on traffic volume at a processing entity and the amount of
information being logged. As such, any enterprise using SIP CLF should
establish operational procedures for file rollovers as appropriate to
the needs of the organization.Listing such operational guidelines in this document is out of scope
for this work.This document does not require any considerations from IANA.The authors of this document would like to acknowledge and thank
Peter Musgrave for his support, guidance, and continued invaluable
feedback.This work benefited from the discussions and invaluable input by the
various members of the SIPCLF working group. These include Brian
Trammell, Eric Burger, Cullen Jennings, Benoit Claise, Saverio
Niccolini, Dan Burnett. Special thanks to Hadriel Kaplan, Chris Lonvick,
Paul E. Jones, John Elwell for their constructive comments, suggestions,
and reviews that were critical to the formulation and refinement of this
draft.Thanks to Anders Nygren for his early implementation, insight, and
reviews of the SIP CLF format.Private Enterprise NumbersIANA