Generating Performance Plot & Automating the Process

Overview

SafeSquid is a powerful networking tool that operates at the application layer of the OSI model.

As it inspects and filters every connection made by users, it requires adequate computing resources to ensure a smooth user experience.

The amount of resources required will depend on the number of users and their usage patterns.

However, as user traffic increases, so does the CPU and memory consumption of SafeSquid, potentially affecting the browsing sessions of end-users.

To prevent this, it is crucial to ensure that the deployed SafeSquid instance has enough computing power to handle all user requests without any bottlenecks.

To aid in this process, SafeSquid offers a performance plot generation feature, which helps to identify the connections handled by SafeSquid and the corresponding CPU and memory consumption.

The performance plot illustrates key performance indicators such as response time, throughput, concurrent connections and error rate on the y-axis and the progression of time on the x-axis.

These performance plots provide valuable insights into the proxy server's performance, allowing for the identification of trends, patterns, and potential bottlenecks.

By analyzing the performance plot, administrators can make informed decisions and take necessary actions to optimize the proxy server's performance, ensuring smooth and efficient operations for all users.

The performance plot is a powerful tool that allows administrators to proactively monitor and manage the performance of their SafeSquid instance.

Performance Plot Explained

Time Range.

Based on the time selected while generating the performance plot.

Below plot is generate is of 1 hour.

To Be Used When You Want to :

Pin point the time range anomaly occurred.

Virtual memory and process age.

The purple line in the graph represents SafeSquid's virtual memory usage in Megabytes over the last hour.

The green line represents the age of the SafeSquid process.

Spikes in the green line indicate when the SafeSquid service has been restarted.

This graph can be used to monitor SafeSquid's virtual memory usage and process age over time, as well as identify instances of the proxy service being restarted.

Total CPU Use Trend.

This graph displays the total CPU usage trend over a selected duration.

The purple line represents the total CPU usage, which helps to identify overall CPU consumption.

This plot can be used to understand the pattern of CPU usage over a specific period of time, such as the last hour, day, or week.

It can reveal patterns such as consistently high usage during certain times of day or spikes in usage at specific moments.

This graph can be used to analyze CPU usage trends, which can provide insight into how the system is being used and identify potential performance issues or bottlenecks.

Total CPU use Delta, System time and user time.

The purple line in the graph represents the Total CPU usage in Delta.

The Total CPU usage Delta is a value that is calculated as the difference between two points in time.

This measurement is used to determine how much the CPU's usage has changed over a period of time.

A higher delta value indicates a greater change in usage and provides insight into the amount of resources consumed by applications and services on the system.

The green line represents the user time trend, which is the amount of CPU time spent in user-mode code (outside the kernel) within the process.

This is only actual CPU time used in executing the process.

The blue line represents the system time trend, which is the amount of CPU time spent in the kernel within the process.

This includes executing CPU time spent in system calls within the kernel.

This graph can be used to analyze system resource usage over a period of time.

Running and waiting process.

The purple line in the graph represents the running process.

It shows all the processes that are currently running on your SafeSquid proxy server.

The green line represents the waiting process.

It shows all the processes that are waiting to be executed.

This graph can be used to check the total number of processes running and the number of processes waiting to be executed on your server.

This can be helpful in identifying potential issues with process management and system resource usage.

System Load Average.

This graph displays the load average for a given period of time.

The purple line represents the load average for the last 1 minute, the green line represents the load average for the last 5 minutes, and the blue line represents the load average for the last 15 minutes.

This plot can be used to estimate the system utilization over a period of time.

For example, if you have a system with 8 CPU cores, a load average of 1 minute of 8.0 or higher would suggest that the system is overwhelmed and unable to keep up with demand.

This can lead to poor performance and long wait times for users.

On the other hand, if the load average for the last 1 minute is 2.0, the last 5 minutes is 3.5 and the last 15 minutes is 4.0, it would suggest that the system is being utilized but not overwhelmed, and it may be able to handle more workloads before reaching capacity.

DNS Query Failure, outbound connections failed and thread error.

The purple line in the graph represents DNS failure.

The green line represents outbound connection failure.

The blue line represents threading errors.

This plot will help you understand the total DNS failures, outbound connection failures, and threading errors.

This plot can be referred to when wanting to know the number of DNS failures, connection failures, and threading errors.

This graph can be used to validate if your proxy server is experiencing any DNS or outbound connection failures, which can help you troubleshoot and identify any issues that may be affecting the performance of your server.

SafeSquid memory Utilization.

This graph displays the various types of memory usage of SafeSquid proxy server.

The purple line represents SafeSquid's virtual memory usage, the green line represents SafeSquid's library memory usage, the dark blue line represents the delta of memory usage, the yellow line represents the code memory usage, the orange line represents the shared memory usage, and the sky blue line represents the resident memory usage.

This graph can be used to monitor the different types of memory usage of SafeSquid proxy server, which can help identify any potential memory issues or bottlenecks that may be affecting the performance of the server. This can be useful for troubleshooting and optimization of the server.

Total and Free system memory.

The purple line in the graph represents the total memory of your proxy server.

The green line represents the free memory of your proxy server.

This plot can be used to monitor the total and free memory of your server over a period of time.

By comparing the total memory to the free memory, you can determine how much memory is being used and how much is available.

This information can be useful for troubleshooting and optimization of the server's memory usage.

This graph can be referred to when wanting to know the total and free memory the proxy server has.

Spare, Waiting, Use client threads.

The purple line in the graph represents spare client threads, which are unused and can be used to create a new connection.

The green line represents threads which are currently in use.

The blue line represents client threads which are waiting for a thread to complete.

This plot can be used to monitor the total number of available threads, the number of threads in use, and the number of threads that are waiting over a period of time.

The blue line representing the waiting thread indicates that users' connection is on hold and waiting for the thread to be completed, this can help you understand the users' experience while using the service.

It can help you understand the overall thread usage of your server and identify any potential issues or bottlenecks that may be affecting the performance of your server.

Total DNS queries.

The purple line in the graph represents the total DNS queries made, the green line represents the total reused DNS queries.

This plot can be used to monitor the total DNS queries made and total DNS queries reused over a period of time.

By comparing the total number of DNS queries made to the total number of reused DNS queries, you can determine the efficiency of the DNS caching system.

This information can be useful for troubleshooting and optimizing the performance of the server's DNS resolution.

This graph can be referred to when wanting to check the total DNS queries made and total DNS queries reused.

Caching Object.

The purple line in the graph represents the caching objects that are currently stored in memory.

The green line represents the number of caching objects that have been removed from memory.

The blue line represents the number of caching objects that have been added to memory.

This plot can be used to monitor the status of your cached objects over a period of time.

You can use this information to understand the effectiveness of your caching system and identify any potential issues or bottlenecks that may be affecting the performance of your proxy server.

This graph can be referred to when wanting to know the status of your cached objects.

Total Bytes In and Out.

Purple line indicates Total bytes in.

Green line indicates total bytes out.

You can check the data bytes sent In and Out of the proxy server to analyze the bandwidth consumption.

To Be Used When You Want to :

Monitor the bandwidth consumption when you are on a limited bandwidth.

TCP connections.

Purple line indicates total TCP connections handled by SafeSquid.

Green Line indicates Idle TCP connections.

This plot will help you identify.

Total TCP connections are the connections which has established a secure connection between the web serves and the data is transmitted securely.

Idle TCP connections are those connections which has established a secure connection between the web serves but no data is transmitted.

To Be Used When You Want to :

Know number of TCP connections established and how many connections are idle.

Outbound Connections Pool.

Purple line indicates outbound connections in pool which are reused.

Green line indicates total outbound connections.

This plot will help you understand total outbound connection which are in pool and the outbound connection reused from the pool.

To Be Used When You Want to :

Check the number of outbound connections that are kept in pool, out of which how many are reused.

Client transaction handled and outbound connection demanded.

Purple line indicates all client transactions handled.

Green line indicates all outbound connections demanded.

Total client transaction handled by SafeSquid can be used to understand number of transactions handled by SafeSquid and number of outbound connections demanded.

To Be Used When You Want to :

Validate the total number of connections demanded by client.

New Incoming connections and client connection in pool.

Purple line indicates all New incoming connections

Green line indicates all client’s connection in pool

To Be Used When You Want to :

Get an estimate of total number of requests made to proxy server and number of client connections are store in pool for reuse.

Concurrent Connections.

Purple line indicates all concurrent client connections.

Green line indicates all concurrent active requests.

Total number of concurrent connections made to proxy server and total number of active concurrent connections.

To Be Used When You Want to :

Know total concurrent connections handled by proxy server and active concurrent connection at the time.

Default Process of Creating Performance Plot

The SafeSquid web interface has an option to generate performance plots, but it can be slow and time-consuming, especially if there is a lot of user log data.

Additionally, it is not possible to automate plot generation on set intervals, such as every hour or every day. A better option for generating performance plots is using the SafeSquid CLI, as it is faster and more efficient.

However, using the CLI script requires providing start and end time as arguments, which can be difficult to retrieve as the timestamp used in performance logs is not easily understandable.

Creating Performance Plot using genPlot.sh Script

The genPlot.sh script allows for automation of performance plot generation based on set time intervals.

The script generates the plots and stores them in the appropriate folder.

Additionally, Monit can be used to further automate the process and generate plots on a custom frequency.

For example, you can use the script to generate performance plots every hour, day, week, fortnight, month, or year.

The script typically takes 7-8 minutes to generate a performance plot for the past 1 hour.

You can also use the genPlot.sh script as a standalone tool to generate reports as needed.

Automating the Process of Performance Plot Creation

To automate the process of plot creation on custom time follow below steps:

Installation

Download the genPlot.tar.gz file

 wget https://<Download Location> -O /tmp/ ; tar -xzvf /tmp/genPlot.tar.gz -C /usr/local/src/

Edit plot.monit file and comment the time frequency which will not be used to generate plot

After updating copy plot.monit file to /etc/monit/conf.d/

 cp /usr/local/src/plot.monit /etc/monit/conf.d/

Copy genPlot.sh script to /usr/local/bin/

 cp /usr/local/src/genPlot.sh /usr/local/bin/

Add execute permissions

 chmod 755 /usr/local/bin/genPlot.sh

Check the Monit control file and reload Monit

 monit -t && monit reload

Performance plot will be generated every hour, day and week as per our plot.monit file.

Validating

To view the logs for plot creation check your /var/log/monit.log file

Note: Do not check for logs immediately after setting up the scripts, wait for at least couple of hours and then check for logs

 grep -E "PERFORMANCE_PLOT_EVERY_(HOUR|DAY|WEEK)" /var/log/monit.log

How to view generated plots

Using a Web-Server

To view generate plot in your browser install Apache web server.

To install Apache web server run below command

 apt install apache2

edit the /etc/apache2/sites-enabled/000-default.conf

 vim /etc/apache2/sites-enabled/000-default.conf

Update the document root from /var/www/html to /var/www/safesquid

Now reload the site configurations using below command

 a2dissite 000-default.conf && systemctl reload apache2 && a2ensite 000-default.conf && systemctl reload apache2

Now open your browser access the webserver using servers IP address

On a local Machine

For users without a webserver, you can access the generated performance plot from /var/www/safesquid/performance_plot location.

To view all plots created run below command.

tree -af /var/www/safesquid

Based on the frequency set you can access the folder and view your performance log.

For example: users who have set Monit to configure plot every hour will find the performance plot to be located inside folder Every_Hour

Copy the files to your location machine and using any image view you can view your performance plots.

Generate plot as required

Usage

To generate report as required run command

 genPlot.sh <options>
 Options: Hour, Day, Week, Fortnight, Month, Year
 Example: genPlot.sh Hour

Example command will generate performance plot of last 1 hour