ISPPlanet Primer: Analyzing Cache Logs

Our primer offers an overview of Common and Squid log formats and the tools you may use to work with them.

by Lisa Phifer VP Core Competence, Inc. [April 19, 2000]

During our ISP-Planet cache review series, we pored through dozens of access logs generated by six different caches. The products we tested generated access logs in two popular formats: the Common Log Format and the native Squid log format. Cache access logs can provide a wealth of information. Here, we offer a quick beginner's primer on cache log analysis.

Common Log Format
The NCSA common log format was long ago designed for the CERN HTTPd and is commonly used by web servers like Apache. Some caches, including NICS, CacheFlow, and squid, can generate access logs in this format. Each entry contains basic information about one HTTP request and its result. Records are composed of the following space-delimited fields, with a single dash placeholder for each absent value:
 

host	Fully-qualified domain name or IP address of the requesting client
ident	The RFC 931 identity of the client (if returned by identd)
authuser	The username (for authenticated requests on password-protected domains)
timestamp	Time of request [DD/MMM/YYYY:hh:mm:ss + zzzz], surrounded by brackets
request	The HTTP request sent by the client, surrounded by double quotes
status	The HTTP status code returned by the server
bytes	The number of content bytes returned to client, without HTTP header

Record Format

host ident authuser [timestamp] "request" status bytes

Examples

192.168.1.142 — [22/Dec/1999:19:41:30 -0500] "GET http://www.corecom.com/ HTTP/1.0" 200 1722

192.168.1.142 — [22/Dec/1999:19:42:45 -0500] "GET http://www.corecom.com/ HTTP/1.0" 304 177

This example illustrates two HTTP GET requests made on http://www.corecom.com by a client browser running on 192.168.1.142. For the first request, the cache successfully retrieved a 1722 byte object from the origin server and returned it to the client. When this same object was requested again "if modified since", it was found to be in the cache and fresh. For further discussion, see HTTP status codes (below).

Some web servers and caches append other fields to the common log record. The extended log format used by some web servers adds referrer and user-agent fields. Squid v2.3 adds its own response and hierarchy codes (e.g., TCP_MISS:DIRECT) when emulating the common log format.

Introduction/Common Log Format

>Squid Log Format

HTTP Status Codes

Log Analysis Tools