Simulating network timeouts with toxiproxy

Goal: Simulate how a Node.js application reacts to timeouts.

Solution: Use toxiproxy and its timeout "toxic" with the value of 0, i.e. the connection won't close, and data will be delayed until the toxic is removed.

The steps:

1. Start toxiproxy, exposing the port 6666 that we intend to use as localhost:6666:

docker pull shopify/toxiproxy
docker run --name=toxiproxy --rm --expose 6666 -p 6666:6666 -it shopify/toxiproxy

(If I was on Linux and not OSX then I could use --net=host and wouldn't need to expose and/or map the port.)

2. Tell toxiproxy to serve request att 6666  via an upstream service:

docker exec -it toxiproxy /bin/sh
/ # cd /go/bin/
/go/bin # ./toxiproxy-cli create upstream -l 0.0.0.0:6666 -u google.com:443

3. Modify your code to access the local port 6666 and test that everything works.

Since we want to access Google via HTTPS, we would get a certificate error when accessing it via localhost:6666 (e.g. "SSLHandshakeException: PKIX path building failed: [..] unable to find valid certification path to requested target" in Java or (much better) "(51) SSL: no alternative certificate subject name matches target host name 'localhost'" in curl) so we will add an alias to our local s /etc/hosts:

127.0.0.1 proxied.google.com

and use
https://proxied.google.com:6666 in our connecting code (instead of the https://google.com:443 we had there before). Verify that it works and the code gets a response as expected.

Note: google.com is likely a bad choice here since it will return 404 as you must specify the header "Host: www.google.com" to get 200 OK back; without it you will get 404.

4. Tell toxiproxy to have an infinite timeout for this service

Continuing our toxiproxy configuration from step 2:

./toxiproxy-cli toxic add -t timeout -a timeout=0 upstream

(Alternatively, e.g. timeout=100; then the connection will be closed after 100 ms.)

5. Trigger your code again. You should get a timeout now.

Tip: You can simulate the service being down via disabling the proxy:

./toxiproxy-cli toggle upstream

Aside: Challenges when proxying through Toxiproxy

The host header

Servers (e.g. google.com, example.com) don't like it when the Host header (derived normally from the URL) differs from what they expect. So you either need to make it possible to access localhost:<toxiproxy port> via the upstream server's hostname by adding it as an alias to /etc/hosts (but how do you then access the actual service?) or you need to override the host header. In curl that is easy with -H "Host: www.google.com" but not so in Java.

In Java (openjdk 11.0.1 2018-10-16) you need to pass -Dsun.net.http.allowRestrictedHeaders=true to the JVM at startup to enable overriding the Host header (Oracle JVM might allow to do that at runtime) and then:

(doto ^HttpURLConnection (.openConnection (URL. "https://proxied.google.com:6666/"))
  (.setRequestProperty "Host" "www.google.com")
(.getInputStream)

SSL certificate issues

As described above, when talking to HTTPS via Toxiproxy, you need to ensure that the hostname you use in your request is covered by the server's certificate, otherwise you will get SSL errors. To apply the solution described here, i.e. adding e.g. proxied.<server name, e.g. google.com> to your /etc/hosts works, provided the certificate is valid also for subdomains, i.e. is issued for <server> and *.<server>, which is not always the case.

Alternatively, you can disable certificate validation - trivial in curl with -k but much more typing in Java.