If you are familiar with ELB/ALB you will know that there are slight idiosyncrasies between the two. For example, ELB allows you to health check a back end server by TCP port. Basically allowing the user to check if a back end comes up and is listening on a specified port. ALB is slightly different in its method for health checking. ALB uses HTTP checks (layer 7) to ensure back end instances are up and listening.
This becomes a problem in Rancher, when you have multiple stacks in a single environment that are fronted by the Rancher HAProxy load balancer. By default, the HAProxy config does not have a health check endpoint configured, so ALB is never able to know if the back end server is actually up and listening for requests.
A colleague and I recently discovered a neat trick for solving this problem if you are fronting your environment with an ALB. The solution to this conundrum is to sprinkle a little bit of custom configuration to the Rancher HAProxy config.
In Rancher, you can modify the live settings without downtime. Click on the load balancer that sits behind the ALB and navigate to the Custom haproxy.cfg tab.
Modify the HAProxy config by adding the following:
# Use to report haproxy's status defaults mode http monitor-uri /_ping
Click the “Edit” button to apply these changes and you should be all set.
Next, find the health check configuration for the associated ALB in the AWS console and add a check the the /_ping path on port 80 (or whichever port you are exposing/plan to listen on). It should look similar to the following example.
Below is an example that maps a DNS name to an internal Nginx container that is listening for requests on port 80.
The check in ALB ensures that the HAProxy load balancer in Rancher is up and running before allowing traffic to be routed to it. You can verify that your Rancher load balancer is working if the instances behind your ALB start showing a status of healthy in the AWS console.
NOTE: If you don’t have any apps initially behind the Rancher load balancer (or that are listening on the port specified in the health check) the AWS instances behind ALB will remain unhealthy until you add configuration in Rancher for the stacks to be exposed, as pictured above.
After setting up HAProxy, publicly accessible services in private Rancher environments can easily be managed by updating the HAProxy config. Just add a dns name and a service to link to and HAProxy is able to figure out how and where to route requests to. To map other services that aren’t listening on port 80, the process is very similar. Use the above as a guideline and simply update the target port to whichever port the app is listening on internally.
