PC System Utilities Software – Software Testing Weekly #1

Benchmark Tests / / 6 minutes of reading

Introduction

We design and test PC System Utilities for more than 18 years. However, we never showed our customers much of our thought process, except for rare occasions on our pc system utilities forum.

With a new build of jv16 PowerTools in our test lab, while getting ready to release the next product update, we decided to start sharing some of the details about “how the sausage gets made”.

By reading this post, you will learn more about how jv16 PowerTools is being tested for some specific test cases.

Simple, but sinister!

Test cases

In order to ensure compatibility of our release candidate software version with the most popular 3rd party software products, we usually conduct tests that answer questions similar to the ones described below:

Test Case #01: Do most popular software still start correctly after using jv16 PowerTools > Clean and SpeedUp My Computer?

This test case is designed to identify any false positives that may result from our recent development work on the latest version of the Clean and SpeedUp My Computer tool, part of jv16 PowerTools.

Test Case #02: What is the improvement in system startup time after using jv16 PowerTools > Startup Optimizer?

This test case is designed to measure the effectiveness of the Startup Optimizer tool on an operating system with  lots of bloatware installed, some of which automatically start with the operating system.

Test Environment

The current test cases were performed using an Oracle VirtualBox – Version 5.1.22 r115126 (Qt5.7.1) virtual environment running Windows 10 with 4 CPU cores and 8 GB of RAM assigned.

The virtual machine was run on a host computer running Ubuntu Linux 17.04 LTS (64 bit) with an Intel-i5 processor, 20 GB RAM and a SSD system disk.

Verifiability

All of the past and current test results are accompanied by screenshots and/or video recordings which have been saved during various testing phases in order to address some of the most common test result credibility concerns for this type of studies.

Test Case #01: Passed!

During the first test case, the following steps were taken:

Step 1) On a freshly installed Windows 10, some of the most popular 3rd party software were installed. To ease this process, a free mass software installer automation by Ninite was used.

Step 2) To make sure that there is some cleaning work to do for our PC System Utilities software, we wanted each of the 3rd party installed software to get a chance at setting their own settings and initial configuration data in the system’s registry.

Also, we wanted to make sure that each of them actually starts correctly and without errors, before touching the system with jv16 PowerTools.

Step 3) In order to ensure that all 3rd party software have been properly installed, a system reboot was carried out. After which, crap.reg was imported into the system’s registry.

This Windows registry key samples database contains intentionally orphaned leftover traces of random 3rd party software. Meaning, registry keys that are disconnected from their local files.

This test file was obtained during previous benchmark test reports and is currently used in our software testing lab as a high quality source of unwanted registry keys.

We use such data to test the effectiveness of our registry cleaner tool as well as our competitors.

Step 4) After the unwanted registry key samples were imported, each software previously installed on Windows was once again run in order to ensure that it still starts correctly.

Conclusion: 100% of the installed 3rd party software still started correctly after the software leftover traces were imported.

Step 5) jv16 PowerTools 2017 version 4.1.0.1724 (release candidate) was installed and the initial setup performed (mapping and diagnosing the system’s registry and file system in order to determine initial system health scores).

Step 6) Before running Clean and SpeedUp My Computer using its default settings, we took note of the Registry Information tool, Home > System Startup Times and Startup Timer > Startups list.

This should help us to evaluate the effectiveness of jv16 PowerTools for this use case scenario as we will be comparing the same information after cleaning and optimizing the system.

Step 7) jv16 PowerTools 2017 > Clean and SpeedUp My Computer was used in order to identify as many system errors as possible.

Using its default scanner settings, the registry cleaner component was able to identify the following software leftovers. Some of them have been imported via the crap.reg file described earlier and some of them are the result of improper software package management by Windows.

Step 8) After the Clean and SpeedUp My Computer tool was used, the system was rebooted once in order to ensure that all settings take effect.

After the system was restarted, each software previously installed from Ninite was once again opened in order to ensure that it is still starting correctly (no false positive symptoms after using jv16 PowerTools).

Test Case #1 conclusion: Test passed! 100% of the installed 3rd party software still started correctly after the imported software leftover (for testing purposes) were cleaned by the Clean and SpeedUp My Computer tool.

Test Case #02: Passed!

During the second test case, the following steps were taken:

Step 9) After cleaning and optimizing the system using the Clean and SpeedUp My Computer tool, we recorded the information provided by the Registry Information tool as well as the Startup Timer which shows the effectiveness of the Startup Optimizer.

Before doing so, we rebooted the system at least five times in order to make sure that the Startup Timer tool has sufficient data to calculate the percent in startup time improvement based on data from before and after using the Clean and SpeedUp My Computer tool.

Step 10) Took a screenshot of the Startup Timer – Startups logging panel, a screenshot of Home > System Startup Times as well as a screenshot of the Registry Information > Details (full registry) panel. You can see all of them below.

Test Case #2 conclusion: Test passed! An average startup time improvement of more than 5% was achieved after using the Startup Optimizer tool.

Test Cases Summary:

Test Case #1 (summary) answers the following question for jv16 PowerTools – release candidate 4.1.0.1723:

Question: Are any of the most popular software affected by the normal operation of the Clean and SpeedUp My Computer tool?

Answer: None of the most popular software installed using Ninite’s automatic software installed seemed to manifest any false positives with jv16 PowerTools 2017 – version 4.1.0.1723.Test Case #2 (summary) answers the following question for jv16 PowerTools – release candidate 4.1.0.1723:

Question: What is the improvement in system startup time after using jv16 PowerTools > Startup Optimizer?

Answer: On average, the test system configuration started 6.94% faster. The result was measured 5 reboots after using the Clean and SpeedUp My Computer tool.

Related weekly benchmark test reports:

Other historical benchmark test reports:

Since February of 2009 (when the first test of this kind was done in our lab), Macecraft Software has published a total of seven registry cleaner product comparison studies: