How your site’s appearance would be influenced if you switch from HTTP to HTTPS?
If by design HTTPS (SSL) is slower than HTTP this wouldn’t hurt your appearance in Google even if the Search machine focuses on the speed. HTTPS is a secure encrypted version of HTTP and is good thing for users if you provide payment system on your site. Google is working on improving the appearance speed when HTTPS.
You don’t need to worry about being penalized because of the speed when HTTPS.
If you want to have a secure site from the beginning it is a good time to switch to HTTPS.