XMLHttpRequest with self-signed certificate too slow

I am seeing that on Internet Explorer 11, XMLHttpRequests would take a long time to finish (over 20s) when hitting a site that is served over HTTPS using self-signed certificate.  The same request comes back within a second on FireFox, Chrome, Safari and even Microsoft Edge.  The issue on internet explorer does not happen if I use plain HTTP.  I have already imported the self-signed certificate into the trusted root authorities.

Are there any fixes or workarounds for this issue, to make the requests go faster with self-signed certificates?

Version: Internet Explorer 11.1884.14393.0 on Windows Server 2016 Standard.