Requests silently bypass SOCKS proxy when using HttpClientExecutor (Java 11+ default)

[Nnnes]

To reproduce:

ssh has built-in SOCKS server functionality with -D:

$ ssh -NvD 1080 hostname

Test the SOCKS server:

$ curl --socks5 localhost:1080 https://eth0.me

If the proxy is working, this prints hostname's public IP address.

Java:

static void test() throws Exception {
    Proxy proxy = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress("localhost", 1080));

    URLConnection urlConnection = URI.create("https://eth0.me").toURL().openConnection(proxy);
    System.out.println(new String(urlConnection.getInputStream().readAllBytes()));

    Connection jsoupConnection = Jsoup.connect("https://eth0.me").proxy(proxy);
    System.out.println(jsoupConnection.get().text());

    System.setProperty("jsoup.useHttpClient", "false");
    Connection jsoupConnection2 = Jsoup.connect("https://eth0.me").proxy(proxy);
    System.out.println(jsoupConnection2.get().text());
}

This prints hostname's public IP address, then yours, then hostname's again.

HttpClient's lack of support for SOCKS proxies is apparently poorly documented intended behavior (see https://stackoverflow.com/a/70011047 and https://bugs.openjdk.org/browse/JDK-8214516).

I think this can be fixed easily by adding a line to RequestDispatch#get(...):

if (request.proxy() != null && request.proxy().type() != Proxy.Type.HTTP) useHttpClient = false;

In the meantime, affected users (perhaps only me!) can use System.setProperty("jsoup.useHttpClient", "false"); as in my example as a workaround.

[jhy]

Thanks for the report, I'll take a deeper look at it. I haven't really thought about SOCKS proxies for a long time and we don't have any tests related to it currently. Should add at least a minimal test for it on the old path and make sure we auto downgrade.

It's surprising to me that Java is silently ignoring the configured proxy when using the HttpClient.

BTW, congratulations on the cool bug number!

[jhy]

Thanks again for the clear report.

I've updated jsoup so that if a per-request SOCKS proxy is configured, it will avoid the JDK HttpClient path and fall back to HttpURLConnection instead. That avoids the JDK HttpClient behavior of silently ignoring the SOCKS proxy and attempting a direct connection.

Can you please test against a snapshot install of jsoup, and verify that it now works the way you want?

Also refactored somewhat the POM and tests for better and more reliable execution of the MR-JAR + HttpClient combo.

[Nnnes]

Yes, it's working as expected now (tested at 8951f55). Thanks!

[jhy]

Great, thanks for confirming.