It still allows the pages to redirect if you are logged in or visiting the pages from a mobile device.
Additionally, WP Super Cache does NOT clear it’s caching rules from your htaccess file upon deactivation.
This will continue to cause issues for Simple Page Tester split tests even after you have deactivated their plugin.
What this means:
If you had caching activated in the past on your site with WP Super Cache it will remain active until you remove the rules from your htaccess file manually regardless of whether you have the plugin marked as “disabled” in settings or not.