Analyze The SafeSquid Logs
Overview
Logs provide a timeline of events for the Linux operating system, applications and system, and are very useful and valuable troubleshooting tool when you encounter any problem. When any issue/problem occurs then system administrator will analyze the log files first.
Most of the log files are stored in plain text form and are stored in the /var/log directory path.
There are 2 type of logs which are essential with SafeSquid installed on your server. One is SafeSquid application log and other is safesquid activity log which are written in system logs.
To view any log file you can use any one of the following command:
# less /var/log/logfilename
# more -f /var/log/logfilename
# cat /var/log/logfilename
# tail -f /var/log/logfilename
# grep -i error /var/log/logfilename
SafeSquid Application Logs
SafeSquid application produces these logs. Logs contain all kinds of error messages, warnings or other events written by the SafeSquid. These messages can provide logical, high level information that are connected to specific use cases. Each log message needs to provide useful information that helps you understand the behavior of SafeSquid.
SafeSquid application generates six different log formats.
We traditionally name them as safesquid.log (Native logs), extended.log (NCSA / Extended logs), config.log (Config logs) and performance.log (Performance logs).
Two more log formats were added subsequently as bypass.log and privacy.log.
The path to these log files are as follows:
| Log File | Path |
| safesquid.log | /var/log/safesquid/native/ |
| extended.log | /var/log/safesquid/extended/ |
| config.log | /var/log/safesquid/config/ |
| performance.log | /var/log/safesquid/performance/ |
| bypass.log | /var/log/safesquid/bypass/ |
| privacy.log | /var/log/safesquid/privacy/ |
You can see the Native Log and Config Log details from SafeSquid Interface.
Native logs
It records various functional aspects like REQUESTS, SECURITY, REDIRECT etc. that are effected by the various features and their configuration.
You can control the verbosity of the Native log by specifying LOGLEVEL, as shown in the table below.
The LOGLEVEL parameter affects only the SafeSquid's Native log.
| Value | Process logged | Value | Process logged |
| 1 | Requests | 16384 | Forwarding |
| 2 | Network | 32768 | Sync |
| 4 | LDAP | 65536 | Antivirus |
| 8 | Header filtering | 131072 | External parser |
| 16 | Interface Request | 262144 | ICAP |
| 32 | Cookie filtering | 524288 | SSL |
| 64 | Redirect | 1048576 | Category |
| 128 | Template | 2097152 | URL command |
| 256 | Text Analyzer | 4194304 | Modules |
| 512 | Rewriting | 8388608 | Security |
| 1024 | Limits | 16777216 | Warning |
| 2048 | Caching | 33554432 | Error |
| 4096 | Prefetching | 67108864 | Profiles |
| 8192 | ICP | 134217728 | Debug |
So, if you wish to record only the requests set LOGLEVEL to 1, if you wish to record only caching related activities set LOGLEVEL to 2048.
If you wish to record all the three activities of rewriting, limits and forwarding, you would simply set LOGLEVEL to 512 + 1024 + 16384 i.e. 17920.
Similarly, if you wished to view absolutely everything (and run the risk of generating a very huge log file in a very short time!), you could set LOGLEVEL to a total of all the values in the table, i.e. 134217727 which is also the default LOGLEVEL if you simply comment the LOGLEVEL specification!.
If you wished to produce just debug logs you should set the LOGLEVEL to 134217728.
If you wished to record all activities and debug information, you should set the LOGLEVEL to 268435455.
NOTE: Adjusting this value requires a restart of SafeSquid service.
This file stored all data related to every request and response processed by the SafeSquid. These logs will be useful for debugging purpose.
Access the SafeSquid User Interface
On top right of Safesquid Interface view Reports >> Dashboard
Click on "Native logs" to see the run time native logs.
- You can see each request and response processed through SafeSquid.
- You can also search for particular string by using search bar. '"Resume" button will help you to stop real time streaming of logs.
 |
Extended logs
The extended.log (NCSA / Extended log format) records maximum details of each request handled by the proxy application. Extended logs will be helpful for generation of reports to analyze the user activities.
FORMAT / LEGEND:
"record_id" "client_id" "request_id" "date_time" "elapsed_time" "status" "size" "upload" "download" "bypassed" "client_ip" "username" "method" "url" "http_referer" "useragent" "mime" "filter_name" "filtering_reason" "interface" "cachecode" "peercode" "peer" "request_host" "request_tld" "referer_host" "referer_tld" "range" "time_profiles" "user_groups" "request_profiles" "application_signatures" "categories" "response_profiles" "upload_content_types" "download_content_types" "profiles"
Example Log Line1:
"1531492103912WfkgX" "91" "2" "13/Jul/2018:19:58:26" "3663" "200" "626" "0" "626" "FALSE" "192.168.0.24" "anonymous@192.168.0.24" "GET" "
" "https://accounts.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0" "image/png" "-" "-" "192.168.24.74:8080" "TCP_MISS" "DIRECT" "ssl.gstatic.com" "ssl.gstatic.com" "gstatic.com" "accounts.google.com" "google.com" "100-1K" "" "ADMINS" "" "Unidentified Web2.0,Firefox,Internet Browser" "Search Engines & Portals" "POTENTIAL MALWARE THREATS,SMALL DOWNLOADS" "-" "image/png" "READ ONLY,ANTIVIRUS"
Example Log Line2:
"153157359815951WfkgX" "1595" "1" "14/Jul/2018:18:36:38" "106" "403" "1517132" "1517132" "0" "FALSE" "192.168.0.24" "anonymous@192.168.0.24" "POST" "http://www.csm-testcenter.org:80/test" "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0" "text/html" "DLP" "application/vnd.ms-excel" "192.168.24.74:8080" "TCP_DENIED" "DIRECT" "safesquid" "www.csm-testcenter.org" "csm-testcenter.org" "www.csm-testcenter.org" "csm-testcenter.org" "1M-10M" "" "ADMINS" "MEDIUM UPLOADS" "Unidentified Web2.0,All Posts,All Uploads,Uploads,Firefox,Internet Browser" "Unknown" "" "application/vnd.ms-excel,text/plain" "-" "READ ONLY,ANTIVIRUS,BLOCK UPLOADS"
The details of the fields in extended.log are as follows:
| Field | Explanation |
| record_id | A unique record identifier, to prevent duplication of records when imported into SQL databases. Example1> 1531492103912WfkgX |
| client_id | The internal SafeSquid ID associated with the present/ongoing connection. Example1> 91 |
| request_id | A unique ID for every request created in the present/ongoing connection to the web-server. Example1> 2 |
| date_time | The date and time when the HTTP request was sent. The fields in the date/time field are [DD/MMM/YYYY:hh:mm:ss], where the fields are defined as follows: DD is the day of the month, MMM is the month, YYYY is the year, hh is the hour, mm is the minute, ss is the seconds. Example1> 13/Jul/2018:19:58:26 |
| elapsed_time | Length of time in milliseconds that current HTTP request used to complete the transaction. Example1> 3663 |
| status | Numeric code indicating the success or failure of the HTTP request. This code is a server response to a browser's request. Example1> 200 |
| size | Numeric field indicating the data transferred in number of bytes as part of the HTTP request, not including the HTTP header. Example1> 626 |
| upload | Numeric field indicating the data transferred from webserver to client. Example2> 1517132 |
| download | Numeric field indicating the data transferred from client to webserver. Example1> 626 |
| bypassed | Boolean(TRUE/FALSE) value which indicates whether the current request was bypassed or not after blocked by SafeSquid filter. Example1> FALSE |
| client_ip | The IP address of the requesting client. Example1> 192.168.0.24 |
| username | The username@client_ip, (or user ID) used by the client for authentication. If no value is present, "anonymous" is substituted. Example1> anonymous@192.168.0.24 |
| method | A method is part of HTTP request sent by client to the server. Example1> GET |
| url | An HTTP url is a reference to a web resource that specifies its location on a computer network and a mechanism for retrieving it. Example1>
|
| http_referer | An HTTP header field that identifies the address of the webpage that linked to the resource being requested. "-" is substituted when there is no referrer in HTTP header field. Example1> https://accounts.google.com/ |
| useragent | User agent are the client which initiates a request. Useraagent are often browsers, editors, spiders(web-traversing robots), or other end user tools. Example1> Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0 |
| mime | Mime-type(media type) are used to identify the format of a file and format contents transmitted on the Internet. Example1> image/png |
| filter_name | Filter name due to which the request was blocked. "-" is substituted when request is allowed. Example2> DLP |
| filtering_reason | If the request get blocked, then predefined reason for blocked request will be set. "-" is substituted for unknown reasons and allowed requests. Example2> application/vnd.ms-excel |
| interface | Interface IP:PORT that received the request. This field is important when SafeSquid is listening on multiple IPs or Ports. Example1> 192.168.24.74:8080 |
| cachecode | Cachecodes are access error code set by SafeSquid during connection with TCP/UDP request. Example2> TCP_DENIED |
| peercode | Peercode entry represents a code that explains how the request was handled, for example, by forwarding it to a peer, or returning the request to the source. Example1> DIRECT |
| peer | Peer represents the name of the host from which the object was requested. This host may be the origin site, a parent, or any other peer. Example1> ssl.gstatic.com |
| request_host | Fully qualified domain name(FQDN) of the requested web-server. Example1> ssl.gstatic.com |
| request_tld | Top-level domain name of the requested web-server. Example1> gstatic.com |
| referer_host | The referrer is the URL of the HTTP resource that referred the user to the resource requested. "-" is substituted when there are no referrers. Example1> accounts.google.com |
| referer_tld | Top-level domain name of the referrer URL that referred the user to the resource requested. "-" is substituted when there are no referrers. Example1> google.com |
| range | Data transferred in number of bytes as part of the HTTP request which comes under SafeSquid predefined range. Example2> 1M-10M |
| time_profiles | Comma separated list of Time profiles that categorize, websites based on the time ranges. "" is substituted when no profiles are applied. |
| user_groups | Comma separated list of User Group(s) to which this entry is applicable. Example1> ADMINS |
| request_profiles | Comma separated list of Request profiles that were applied to the request. "" is substituted when no request profiles are applied. Example2> MEDIUM UPLOADS |
| application_signatures | Comma separated list of application signatures that were applied to the request. Example2> Unidentified Web2.0,All Posts,All Uploads,Uploads,Firefox,Internet Browser |
| categories | Comma separated list of categories that were applied to the request. "" is substituted when no profiles are applied. Example2> Search Engines & Portals |
| response_profiles | Comma separated list of Response profiles that were applied to the request. "" is substituted when no response profiles are applied. Example2> POTENTIAL MALWARE THREATS,SMALL DOWNLOADS |
| upload_content_types | Comma separated list of Upload Content Type. "-" is substituted when no profiles are applied or content types are unknown. Example2> application/vnd.ms-excel,text/plain |
| download_content_types | Comma separated list of Download Content Type. "-" is substituted when no profiles are applied or content types are unknown. Example1> image/png |
| profiles | Comma separated list of profiles that were applied to the request. "" is substituted when no profiles are applied. Example2> READ ONLY,ANTIVIRUS,BLOCK UPLOADS |
On top right of Safesquid Interface view Reports >> Dashboard
Click on "Detailed logs" to see the run time extended logs.
Run time detailed logs displayed are Time, Username, Client ID, Method, URL, Status, Size, Referer, Useragent, Mime, Filter Name, Filter Reason, Profiles, Request profiles, Response profiles, Categories, Applications, Upload content type, Download content type from Extended Logs.
- You can use search option to find information related to specific user or specific response code...etc. It's a common search option for filtering.
- Show/Hide Column button is for hiding any option from above list.
- You can also search individually from above list.
 |
Config logs
Config logs contain the details related to the SafeSquid user interface activities like configuration change. You can find the users information who made the configuration changes and when.
FORMAT / LEGEND:
"ACCESS_TIME" "SAFESQUID_INTERFACE" "USERNAME@IP" "PAGE" "SECTION" "ACTION" "HTTP_METHOD" "URL" "REFERER" "ARGUMENTS" "CONFIG_FILE" "REASON"
Example Log Line1:
"01/Aug/2018:01:30:19" "192.168.24.74:8080" "anonymous@192.168.0.27" "" "dlp" "edit" "POST" "http://safesquid.cfg:80/" "http://safesquid.cfg/" "F0=TRUE&F1=Block the uploading of PDF and Microsoft Word Documents. &F2=BLOCK-ALL-UPLOADS&F3=application/octet-stream,application/x-msdownload|application/x-dosexec&F4=\.bat$|\.exe$,\.exe$&F5=MIME_OR_FILE&F6=DO_NOT_BYPASS&handler=configuration§ion=dlp&subsection=DLP&id=3&action=edit" "" ""
Example Log Line2:
"01/Aug/2018:18:29:38" "192.168.24.74:8080" "anonymous@192.168.0.27" "" "" "" "POST" "http://safesquid.cfg:80/" "http://safesquid.cfg/" "handler=save&filename=config.xml" "/usr/local/safesquid/security/policies/config_anonymous@192.168.0.27_2018-08-27-18-29-38.xml" ""
The details of the fields in config.log are as follows:
| Field | Explanation |
| ACCESS_TIME | The date and time when the request for modification in SafeSquid configuration was sent. The fields in the date/time field are [DD/MMM/YYYY:hh:mm:ss], where the fields are defined as follows: DD is the day of the month, MMM is the month, YYYY is the year, hh is the hour, mm is the minute, ss is the seconds. Example1> 01/Aug/2018:01:30:19 |
| SAFESQUID_INTERFACE | Interface IP:PORT that received the request for modification in SafeSquid configuration. This field is important when SafeSquid is listening on multiple IPs or Ports. Example1> 192.168.24.74:8080 |
| USERNAME@IP | The USERNAME@IP, (or user ID) of the client who modified the SafeSquid configuration. If username is unknown then, "anonymous" is substituted. Example1> anonymous@192.168.0.24 |
| PAGE | |
| SECTION | SafeSquid section where rule is either created or modified or deleted. Example1> DLP |
| ACTION | Action taken on any SafeSquid section to add a new policy or edit/ clone/ delete an existing policy. Example1> edit |
| HTTP_METHOD | HTTP method to request sent by client to the server. Example1> POST |
| URL | SafeSquid interface URL is a reference to open WebGUI to View/change configuration. Example> http://safesquid.cfg:80/ |
| REFERER | Referer address of the SafeSquid interface URL. Example1> http://safesquid.cfg/ |
| ARGUMENTS | This field displays the SafeSquid policy details which was modified. Where F# shows the field values of the all fields. section=# displays section name. id=# determines the policy number from top. action=# shows the action performed on that policy. Example1> F0=TRUE&F1=Block the uploading of PDF and Microsoft Word Documents. &F2=BLOCK UPLOADS&F3=application/octet-stream,application/x-msdownload|application/x-dosexec&F4=\.bat$|\.exe$,\.exe$&F5=MIME_OR_FILE&F6=DO_NOT_BYPASS&handler=configuration§ion=dlp&subsection=DLP&id=3&action=edit |
| CONFIG_FILE | A backup config.xml file along with the path which can be restored at any time by the administrator. Example2>/usr/local/safesquid/security/policies/config_anonymous@192.168.0.27_2018-08-27-18-29-38.xml |
| REASON |
On top right of Safesquid Interface view Reports >> Dashboard
Click on "Config logs"
- You can use search option to find information related to specific Time, Interface, Username, Section, Action, Arguments, Config File.
 |
Performance logs
Performance logs will be helpful to analyze the performance of SafeSquid.
SafeSquid Performance logs provide performance metrics to identify any outage due to resource shortfall, or failure in Internet Connectivity, or surge in web-traffic, etc.
SafeSquid performance log has been extended to make it easier for analysis with third-party software such as GNU Plot that analyses records on a progressive per line basis.
FORMAT / LEGEND:
Time Stamp (YYYYMMDDhhmmss),Elapsed Time,Client Connections Handled,Client Connections Closed,Client Transactions Handled,Client Connections in Pool,Spare Client Threads,Client Threads in Use,Client Threads in Waiting,Threads Starting up,Threads Reserved for Prefetching,Threading Errors,Outbound Connections created,Outbound Connections Failed,Outbound Connection Pool Reused,Outbound Connections in Pool,Bytes in (KBytes),Bytes Out (KBytes),Caching Objects Created in Memory,Caching Objects Removed from Memory,DNS Queries Reused,New DNS Queries,DNS Query failures,Total System Memory (KBytes),Free System Memory (KBytes),SafeSquid Virtual Memory (KBytes),SafeSquid Resident Memory (KBytes),SafeSquid Shared Memory (KBytes),SafeSquid Code Memory (KBytes),SafeSquid Data Memory (KBytes),SafeSquid Library Memory (KBytes),Connections Handled Delta,Connections Closed Delta,Transactions Handled Delta,Client Pool Delta,Spare Threads Delta,Active Threads Delta,Threads Waiting Delta,Threads Starting up Delta,Threads Prefetching Delta,Threading Errors Delta,Outbound Connections created Delta,Outbound Connections Failed Delta,Outbound Connection Pool Reused Delta,Outbound Connections in Pool Delta,Bytes in (KBytes) Delta,Bytes Out (KBytes) Delta,Caching Objects Created in Memory Delta,Caching Objects Removed from Memory Delta,DNS Queries Reused Delta,New DNS Queries Delta,DNS Query failures Delta,load avg.(1 min),load avg.(5 min),load avg.(15 min),Running Processes,Waiting Processes,User Time,System Time,Total (user + system) Time,User Time Delta,System Time Delta,Total Time Delta
Example Log Line1:
20180905172137,1603,21151,20941,44316,119,8016,91,68,17,0,0,13640,975,20650,139,1853191,2012552,0,0,11317,3185,41,5976008,2389076,596224,98868,4204,2472,0,
472172,13,32,52,11,0,-30,30,0,0,0,5,3,27,0,1698,884,0,0,5,2,0,0.25,0.21,0.49,1,562,60.000000,56.000000,116.000000,0.000000,0.000000,0.000000
Example Log Line2:
20180905172139,1605,21170,20972,44346,132,8016,66,93,17,0,0,13654,976,20660,139,1854126,2014492,0,0,11328,3186,41,5976008,2387184,596224,98888,4204,2472,0,
472172,19,31,30,13,0,-25,25,0,0,0,14,1,10,0,935,1940,0,0,11,1,0,0.23,0.21,0.49,1,562,60.000000,56.000000,116.000000,0.000000,0.000000,0.000000
The details of the fields in performance.log are as follows:
| Field | Explanation |
| Time Stamp (YYYYMMDDhhmmss) | Time when SafeSquid generated this record and stored in performance log. SafeSquid begins printing of each record on a new line with the first column as the TimeStamp in YYYYMMDDhhmmss format. Example> Log line1 was loged at 2018 September, 5th at 17:21:37 hence it displays 20180905172137 in YYYYMMDDhhmmss format. |
| Elapsed Time | Time in seconds since the SafeSquid process started. This is the time in seconds since the process started. Example> Log line1 reports that SafeSquid process running from last 1603 seconds. |
| Client Connections Handled | Total Client Connections handled by SafeSquid since the start of process. Example> Log line1 reports that since SafeSquid process running(from 1603 seconds), process handled 21151 incoming connections. |
| Client Connections Closed | Total Client Connections closed by SafeSquid since the start of process. Example> Log line1 reports that since the start of the process, SafeSquid had closed 20941 incoming connections. Thus we can interpret that at 2018-05-09 17:21:37 there were 21151 - 20941 = 210 concurrent incoming client connections. So while the 3rd counter and 4th counter are not very significant by themselves, their differential value is very important as it gives a an idea about concurrent connections being handled at any point. |
| Client Transactions Handled | Total Number of transactions handled between clients and SafeSquid since the start of the process. Within each connection a client may send multiple requests, to service these requests SafeSquid may have to transact multiple HTTP transactions. Example> Log line1 reports that since start of the process, SafeSquid had handled 44316 transactions. |
| Client Connections in Pool | Connections in local pool of SafeSquid. After request handling each connection is been kept in client connection pool for persistence. SafeSquid may not forcibly close a client connection after performing a transaction within, and even wait for sometime allowing the client to reuse a connection after a few seconds. Such connections are maintained in a "Client Pool". Example> Log line1 reports that there were 119 client connections that were idling in the client pool. Thus out of the 210 concurrent connections (derived above), 119 were in the client pool and the remaining 91 were either actively transacting, or maybe in the various stages of closure. |
| Spare Client Threads | Additional number of Threads that SafeSquid may produce to service client requests when SafeSquid printed this record. Example> Log line1 reports that SafeSquid can still produce 8016 client requests. |
| Client Threads in Use | Concurrent Requests being serviced by SafeSquid when SafeSquid printed this record. Example> Log line1 reports that SafeSquid had used 91 threads to fulfill client requests. |
| Client Threads in Waiting | Threads just released by client requests and available for instantly servicing a client request when SafeSquid printed this record. Example> Log line1 reports that at present 68 client threads are in waiting state. |
| Threads Starting up | Some Threads already assigned for some sections when SafeSquid starts, these are special threads of SafeSquid. Example> Log line1 reports that at present 17 threads are used by SafeSquid for internal processes. |
| Threads Reserved for Prefetching | Threads reserved by SafeSquid, to service pre-fetching demands. Example> Log line1 reports that at present 0 thread is used for pre-fetching. |
| Threading Errors | Total number of events when SafeSquid could not produce a thread due to system failure or MAXTHREADS limitation. Example> Log line1 reports that since start of the process 0 threading errors occured. |
| Outbound Connections created | Total New connections created by SafeSquid to various remote webservers to service a client request. Example> Log line1 shows total 13640 new connections created by SafeSquid. |
| Outbound Connections Failed | Total Number of Events since the start of the process, when SafeSquid failed to create a connection to various remote webservers to service a client request. Example> Log line1 shows total 975 failed outbound connections. |
| Outbound Connection Pool Reused | After handling request from client, connections are kept/stored in connection pool in waiting state. These connections are been reused so that additional burden on server of creating too many connections on server be lower down. Total reused connection till current time. Example> Log line1 shows 20650 outbound connections are been reused from connection pool from last 1603 seconds. |
| Outbound Connections in Pool | Total number of Outbound connections idling in Connection Pool. Example> Log line1 shows 139 outbound connections idling in pool. |
| Bytes in (KBytes) | Total number of KiloBytes received by SafeSquid since the start of the process. Example> Log line1 shows 1853191 Kilobytes has been received from server since the start of the process. |
| Bytes Out (KBytes) | Total number of KiloBytes sent by SafeSquid since the start of the process. Example> Log line1 shows 2012552 Kilobytes has been sent by SafeSquid since start of the process. |
| Caching Objects Created in Memory | Total number of Cached Objects created by SafeSquid in memory since the start of the process. Example> Log line1 shows 0 caching objects created in memory. |
| Caching Objects Removed from Memory | Total number of Cached Objects removed by SafeSquid from memory since the start of the process. Example> Log line1 shows 0 caching objects removed from memory. |
| DNS Queries Reused | Total number of DNS queries to fulfil client request that are previously used and saved in DNS cache. Example> Log line1 shows 11317 DNS queries are reused from DNS cache since start of the process. |
| New DNS Queries | Total number of successful DNS queries to fulfill client request. Example> Log line1 shows 3185 DNS queries are reused from DNS cache since start of the process. |
| DNS Query failures | Total number of events when SafeSquid successfully failed to do a DNS lookup since the start of the process. Example> Log line1 shows 41 DNS query fails since start of the process. |
| Total System Memory (KBytes) | Total system memory of server where SafeSquid is installed. Example> Log line1 indicates server has 5976008 KB(i.e. 6 GB) of RAM. |
| Free System Memory (KBytes) | Current Free system memory of server where SafeSquid is installed. Example> Log line1 indicates free system memory when log captured was 2389076 KB(i.e. 2.38 GB). |
| SafeSquid Virtual Memory (KBytes) | Current Virtual memory of server where SafeSquid is installed. Example> Log line1 shows server has 596224 KB virtual memory. |
| SafeSquid Resident Memory (KBytes) | Current Resident memory of server where SafeSquid is installed. Or The size of the SafeSquid process resident in physical memory. Example> Log line1 shows server has 98868 KB resident memory. |
| SafeSquid Shared Memory (KBytes) | Current shared memory of server where SafeSquid is installed. The memory shared with other processes other than SafeSquid i.e. memory mapped both by SafeSquid process and other process(such as shared libraries or other helping executables and libraries). Example> Log line1 indicates 4204 KB shared memory. |
| SafeSquid Code Memory (KBytes) | Total SafeSquid Code Memory in the text format. Example> Log line1 indicates 2472 KB of code is written in SafeSquid. |
| SafeSquid Data Memory (KBytes) | Memory that is been used by SafeSquid in the server. The size of the SafeSquid process i.e. the size of loaded executable code of SafeSquid. Example> Log line1 shows 0 bytes of data memory used. |
| SafeSquid Library Memory (KBytes) | Total memory used by SafeSquid helping Libraries. Example> Log line1 shows 472172 Kbytes of memory used by SafeSquid libraries. |
| Connections Handled Delta | Difference between two consecutive connection handled entries i.e. [present number of connection handled - previous entry connection handled]. Example> Log line2 shows Connections Handled Delta = 21170 - 21151 => 19 |
| Connections Closed Delta | Difference between two consecutive connection closed entries i.e. [Number of connection closed - previous entry connection closed]. Example> Log line2 shows Connections Closed Delta = 20972 - 20941 => 31 |
| Transactions Handled Delta | Difference between two consecutive total transactions handled this can be determined by client ID i.e. [current entry transaction handled - previous entry transaction handled]. Example> Log line2 shows Transactions Handled Delta = 44346 - 44316 => 30 |
| Client Pool Delta | Difference between two consecutive client pool entries i.e. [present connections in pool waiting state - previous entry pool having connection waiting]. Example> Log line2 shows Client Pool Delta = 132 - 119 => 13 |
| Spare Threads Delta | Difference between two consecutive unused threads entry i.e. [current unused threads - previous entry unused threads]. Example> Log line2 shows Spare Threads Delta = 8016 - 8016 => 0 |
| Active Threads Delta | Difference between two consecutive entries active threads i.e. [current active threads - previous entry active threads]. Example> Log line2 shows Active Threads Delta = 66 - 91 => -25 |
| Threads Waiting Delta | Difference between two consecutive entries waiting threads i.e. [current waiting threads - previous entry waiting threads]. Example> Log line2 shows Threads Waiting Delta = 93 - 68 => 25 |
| Threads Starting up Delta | Difference between two consecutive entries startup/special threads i.e. [current special threads running - previous entry special threads running]. Example> Log line2 shows Threads Starting up Delta = 17 - 17 => 0 |
| Threads Prefetching Delta | Difference between two consecutive entries prefetching threads running i.e. [present number of threads used for prefetching - previous entry prefetch threads used]. Example> Log line2 shows Threads Prefetching Delta => 0 |
| Threading Errors Delta | Difference between two consecutive entries count of threads having error i.e. [total threads having error reported - previous entry error reported thread]. Example> Log line2 shows Threading Errors Delta => 0 |
| Outbound Connections created Delta | Difference between two consecutive entries where new connections are been created i.e. [count of new connections created - previous entry having count of new connections created]. Example> Log line2 shows Outbound Connections created Delta = 13654 - 13650 => 14 |
| Outbound Connections Failed Delta | Difference between two consecutive entries where connections failed to complete the request i.e. [Failed connections count - failed connections count]. Example> Log line2 shows Outbound Connections Failed Delta = 976 - 975 => 1 |
| Outbound Connection Pool Reused Delta | Difference between two consecutive entries where pooled connections are been reused i.e. [connections reused from pool now - connections reused from pool 2 seconds before]. Example> Log line2 shows Outbound Connection Pool Reused Delta = 20660 - 20650 => 10 |
| Outbound Connections in Pool Delta | Difference between two consecutive entries total number of pooled connections i.e. [connections in pool waiting at present - connections in pool before 2 seconds waiting]. Example> Log line2 shows Outbound Connections in Pool Delta = 139 - 139 => 0 |
| Bytes in (KBytes) Delta | Difference between two consecutive entries total data received in Kilobytes from server i.e. [Data[KB] received from server - Data[KB] received from server 2 seconds before]. Example> Log line2 shows Bytes in Delta = 1854126 - 1853191 => 935 KBytes |
| Bytes Out (KBytes) Delta | Difference between two consecutive entries total data send in Kilobytes from client i.e. [Data[KB] send to server - Data[KB] send to server 2 seconds before]. Example> Log line2 shows Bytes Out Delta = 2014492 - 2012552 => 1940 |
| Caching Objects Created in Memory Delta | Difference between two consecutive entries total caching objects created in memory i.e. [total caching objects created - total caching objects created 2 seconds before]. Example> Log line2 shows Caching Objects Created in Memory Delta => 0 |
| Caching Objects Removed from Memory Delta | Difference between two consecutive entries total caching objects removed from memory i.e. [total caching objects removed at present - total caching objects removed 2 seconds before]. Example> Log line2 shows Caching Objects Removed from Memory Delta => 0 |
| DNS Queries Reused Delta | Difference between two consecutive entries total DNS queries used from DNS cache i.e. [total DNS queries used from cache at present - total DNS queries used from cache 2 seconds before]. Example> Log line2 shows DNS Queries Reused Delta = 11328 - 11317 => 11 |
| New DNS Queries Delta | Difference between two consecutive entries new DNS queries fired by SafeSquid i.e. [new DNS queries fired at present - new DNS queries 2 seconds before]. Example> Log line2 shows New DNS Queries Delta = 3186 - 3185 => 1 |
| DNS Query failures Delta | Difference between two consecutive entries of the DNS failure count i.e. [DNS queries failure count at present - DNS queries failure count 2 seconds before]. Example> Log line2 shows DNS Query failures Delta => 0 |
| load avg.(1 min) | 1 Minute System load average when SafeSquid printed this record. Example> Log line2 shows 0.23 of 1 minute load average |
| load avg.(5 min) | 5 Minute System load average when SafeSquid printed this record. Example> Log line2 shows 0.21 of 5 minute load average |
| load avg.(15 min) | 15 Minute System load average when SafeSquid printed this record. Example> Log line2 shows 0.49 of 15 minute load average |
| Running Processes | Total of all SafeSquid Running Processes using CPU time when SafeSquid printed this record. Example> Log line2 shows only 1 running process |
| Waiting Processes | Total of all SafeSquid Waiting Processes waiting for CPU time when SafeSquid printed this record. Example> Log line2 shows 562 Waiting Processes |
| User Time | User time is the amount of CPU time spent in user-mode code(outside the kernel) within the process. i.e. actual CPU time consumed by SafeSquid processes. Example> Log line2 shows Cumulated CPU time consumed by SafeSquid processes is 60.000000 |
| System Time | System time is the amount of CPU time spent in the kernel within the process. This means executing CPU time spent in system calls within the kernel, as opposed to library code, which is still running in user-space. Like 'user', this is only CPU time used by the process. Example> Log line2 shows system time consumed by SafeSquid processes is 56.000000 |
| Total (user + system) Time | User + System will tell you how much actual CPU time your process used. Note that this is across all CPUs, so if the process has multiple threads it could potentially exceed the wall clock time reported by Real. Note that in the output these figures include the User and System time of all child processes as well, although the underlying system calls return the statistics for the process and its children separately. Example> Log line2 shows Cumulated Total of CPU Time consumed due to SafeSquid processes is 116.000000 |
| User Time Delta | Difference between two consecutive entries of total user time i.e. [CPU time used by SafeSquid at present - CPU time used by SafeSquid before 2 seconds]. Example> Log line2 shows User Time Delta = 0.000000 |
| System Time Delta | Difference between two consecutive entries of total system time i.e. [CPU time used by SafeSquid at present - CPU time used by SafeSquid before 2 seconds]. Example> Log line2 shows System Time Delta = 0.000000 |
| Total Time Delta | Difference between two consecutive entries of total (system + user) time i.e. [Present (user+system) time used by SafeSquid - (user+system) time used by SafeSquid before 2 seconds]. Example> Log line2 shows Total Time Delta = 0.000000 |
Performance Plot
From the interface go to support page, there you will see an option Performance Plot, Click on that tab and select two time intervals it will give you the performance plot between the selected time.
See More about How to generate the Performance Plot
Bypass logs
Bypass logs contain the details related to the execution of bypass privilege granted to any user. When users with bypass privilege, execute their privilege to access a web-site that is not explicitly allowed, it is recorded in the bypass logs. It also records the users opinions about the site, and the URLs that were additionally bypassed, to present a seamless experience.
FORMAT / LEGEND:
"TimeStamp" "Action" "User "Referrer.Domain" "Requested.Domain" "From/Referral/URL" "Method" "Requested/URL" "Categories,Applied" "Suggested,Categories"
Example Log Line1:
"2018.09.12:21:07:11" "NOT BYPASSED" "anonymous@192.168.0.24" "microsoft.com" "optimizely.com" "https://powerbi.microsoft.com/" "POST" "https://logx.optimizely.com:443/v1/events" "business,computersandsoftware" "-"
Example Log Line2:
"2018.09.12:21:07:51" "HOST ADDED" "anonymous@192.168.0.24" "microsoft.com" "optimizely.com" "https://powerbi.microsoft.com/" "POST" "https://logx.optimizely.com:443/v1/events" "business,computersandsoftware" "test"
The details of the fields in bypass.log are as follows:
| Field | Explanation |
| TimeStamp | The date and time when the user with bypass privilege accessed a web-site that is not explicitly allowed. The fields in the date/time field are [YYYY.MM.DD:hh:mm:ss], where the fields are defined as follows: DD is the day of the month, MMM is the month, YYYY is the year, hh is the hour, mm is the minute, ss is the seconds. Example> 2018.09.12:21:07:11 |
| Action | Action taken by User to either block or bypass the cookie for current URL(either host or referer). User can also re-categorize the current URL into predefined categories. Example> NOT BYPASSED |
| User | The User is annotates as username@clientip used by the client for authentication. If no value is present, "anonymous" is substituted. Example> anonymous@192.168.0.24 |
| Referrer.Domain | Domain name of the referrer URL that referred the user to the resource requested. "-" is substituted when there are no referrers. Example> microsoft.com |
| Requested.Domain | Domain name of the requested web-server. Example> optimizely.com |
| From/Referral/URL | From/Referrer/URL is the URL of the HTTP resource that referred the user to the resource requested. "-" is substituted when there are no referrer URL. Example> https://powerbi.microsoft.com/ |
| Method | A method is part of HTTP request sent by client to the server. Example> POST |
| Requested/URL | Requested/URL is reference to a web resource that specifies its location on a computer network and a mechanism for retrieving it. Example> https://logx.optimizely.com:443/v1/events |
| Categories,Applied | Comma separated list of categories that were applied to the request. "_" is substituted when no categories are applied. Example> business,computersandsoftware |
| Suggested,Categories | Comma separated list of categories that were suggested to the request which you want to access. "_" is substituted when no categories are suggested. Log line2 is printed after the requested URL is given access with suggestion of re-categorization. Example> test |
Privacy logs
Privacy Logs are used to record incidences of cookies, and queries going to third-party web sites.
Privacy log captures incidences of third party web-sites when a user accesses a web-site, and action taken by elevated privacy.
FORMAT / LEGEND:
YYYY/mm/DD:HH:MM:SS <ClientID> username@IP HTTP-REFERRER METHOD URL [USER_AGENT] action
The details of the fields in privacy.log are as follows:
| Field | Explanation |
| YYYY/mm/DD:HH:MM:SS | Time when SafeSquid generated this record and stored in privacy log. SafeSquid begins printing of each record on a new line with the first column as the TimeStamp in YYYY/mm/DD:HH:MM:SS format. |
| <ClientID> | The internal SafeSquid ID associated with the present/ongoing connection. |
| username@IP | The username@IP(or user ID) used by the client for authentication. If no value is present, "anonymous" is substituted. |
| HTTP-REFERRER | An HTTP header field that identifies the address of the webpage that linked to the resource being requested. "-" is substituted when there is no referrer in HTTP header field. |
| METHOD | A method is part of HTTP request sent by client to the server. |
| URL | An HTTP url is a reference to a web resource that specifies its location on a computer network and a mechanism for retrieving it. |
| [USER_AGENT] | User agent are the client which initiates a request. Useraagent are often browsers, editors, spiders(web-traversing robots), or other end user tools. |
| action | Action is the privacy level applied to the request. |
SafeSquid activity log written in system logs
The path to these log files are as follows:
| Log File | Path |
| Monit Log | /var/log/monit.log |
| Syslog | /var/log/syslog |
Monit Log
SafeSquid Uses Monit service for managing and monitoring its process. Monit service is particularly useful for monitoring daemon processes, such as those started at system boot time. While Monit service is running and if SafeSquid process is terminated either manually or automatically then SafeSquid will be restarted. Also Monit service plays a crucial role when user want to modify any startup parameter or upgrade the SafeSquid version.
FORMAT:
[date] priority : message
Example Log Lines:
[IST Jan 20 13:08:37] info : 'safesquid' trying to restart
[IST Jan 20 13:08:37] info : 'safesquid' start: /etc/init.d/safesquid
[IST Jan 20 13:08:40] error : 'safesquid.dns.conf' timestamp was changed for /usr/local/safesquid/security/dns/safesquid.dns.conf
[IST Jan 20 13:08:40] info : 'safesquid.dns.conf' exec: /bin/bash
[IST Jan 20 13:08:40] error : 'upgrade' timestamp was changed for /tmp/safesquid/upgrade
[IST Jan 20 13:08:40] info : 'upgrade' exec: /bin/bash
[IST Jan 20 13:08:40] error : 'safesquid.ldb' file doesn't exist
[IST Jan 20 13:08:40] info : 'safesquid.ldb' trying to restart
[IST Jan 20 13:08:43] error : 'Logs' space usage 86.9% matches resource limit [space usage>5.0%]
[IST Jan 20 13:08:43] info : 'safesquid' process is running with pid 1815
[IST Jan 20 13:08:43] error : 'named' process PID changed from 1170 to 1891
[IST Jan 20 13:08:43] info : 'safesquid.dns.conf' timestamp was not changed for /usr/local/safesquid/security/dns/safesquid.dns.conf
[IST Jan 20 13:08:43] info : 'upgrade' timestamp was not changed for /tmp/safesquid/upgrade
Syslog
Linux’s syslog service provide a highly configurable logging system. Some messages triggered by SafeSquid application are been stored in syslog. Syslog is a standard log for sending and receiving notification messages–in a particular format–from various network devices. The messages include time stamps, event messages, severity, host IP addresses, diagnostics and more.
Syslog was designed to monitor network devices and systems to send out notification messages if there are any issues with functioning–it also sends out alerts for pre-notified events and monitors suspicious activity via the change log/event log of participating network devices.
You will find log lines of SafeSquid application in the syslog file.
Example Log Lines:
2020 01 20 14:25:08 [-1] init: Startup Log: /var/log/syslog
2020 01 20 14:25:08 [-1] init: safesquid: start called
2020 01 20 14:25:08 [-1] init: USER: ssquid exists
2020 01 20 14:25:08 [-1] init: ssquid: checking for group membership: root
2020 01 20 14:25:08 [-1] init: ssquid: has group memberships: root
2020 01 20 14:25:08 [-1] init: ssquid: is member of group: root
2020 01 20 14:25:08 [-1] init: export MALLOC_CHECK_=2
2020 01 20 14:25:08 [-1] init: debug: invoking tcp_tune
2020 01 20 14:25:08 [-1] init: RAMDEVICE /dev/ram1 does not exist, and therefore cannot be used
2020 01 20 14:25:08 [-1] init: LIBS_DIR: /opt/safesquid/libs no preload libraries found
2020 01 20 14:25:08 [-1] init: validating: OPT_DIR:/opt/safesquid with permissions ug=rwX,o=r
2020 01 20 14:25:08 [-1] init: already exists: OPT_DIR:/opt/safesquid
2020 01 20 14:25:08 [-1] init: SET_PERMISSION: OPT_DIR:/opt/safesquid ug=rwX,o=r
2020 01 20 14:25:08 [-1] init: validating: USR_LOCAL_DIR:/usr/local/safesquid with permissions ug=rwX,o=r
2020 01 20 14:25:08 [-1] init: already exists: USR_LOCAL_DIR:/usr/local/safesquid
2020 01 20 14:25:08 [-1] init: SET_PERMISSION: USR_LOCAL_DIR:/usr/local/safesquid ug=rwX,o=r
2020 01 20 14:25:08 [-1] init: validating: TMP_DIR:/tmp/safesquid with permissions ug=rwX,o=r
2020 01 20 14:25:08 [-1] init: already exists: TMP_DIR:/tmp/safesquid
2020 01 20 14:25:08 [-1] init: SET_PERMISSION: TMP_DIR:/tmp/safesquid ug=rwX,o=r