Let's Encrypt Fun (NOT) - a workaround

Note: this article expresses my opinions, however incorrect they may be. They don't reflect the opinions of anyone else.

Update: 10/4/2021 - I wanted to point a couple things out.

1. Special thanks to u/pabechan for reminding me that the diag test app wad 99 was only for proxy based inspection. If you need to reset the cache in flow based inspection, I have updated the configs below.
   

2. Take this with a grain of salt, there are articles out there that recommend this configuration:

config system dns-database
    edit "1"
        set domain "identrust.com"
        config dns-entry
            edit 1
                set hostname "apps"
                set ip 127.0.0.1
            next
        end
    next
end

Be mindful that setting the domain to "identrust.com" and not configuring authoritative to disable (set authoritative disable) will blackhole all of identrust.com. This could lead to unwanted behavior. You can either issue "set authoritative disable" as in the config below, or get very specific with the domain name and still set authoritative disable. This will prevent the FortiGate from acting authoritatively against anything other than "apps.identrust.com".

If you're a Fortinet customer, and you use anything more than certificate-inspection in flow-based policy inspection. You've felt the effects of the Let's Encrypt DST Root CA X3 Expiration. If you're wondering what a Root CA certificate is, it's in essence a trusted authority that all major browsers/operating systems/devices have agreed to "Trust" and will believe that any certificates that are issued by these CAs are valid, not unlike you trusting the DMV to issue your ID card or Driver's License by looking for certain attributes on it (Hologram, the Gold Star, etc..).

Normally a CA expires, notification is sent out long before hand, and all server admins/app admins, device manufacturers, etc. provide an update to the Trusted Root CAs well in advance of expiry, and the internet continues on happily. In this case, Let's Encrypt was trying to be proactive and allow older android devices to continue operating normally, more details here. Huzzah! We're good! Except as we all found out, we were not.

What Let's Encrypt did, was allow devices/browsers/etc. is allow for the lookup of two certificate paths:

Image Source: Let's Encrypt website

So in theory, if an older Android device with an older certificate store encounters a new cert, it should follow the "New Default Chain". Other devices would be presented the new chain as well, as the alternate chain, and the world goes on. Unfortunately - I'm not putting this on Let's Encrypt or anyone else for that matter - what Let's Encrypt could not account for was how device vendors will handle this, specifically security appliance vendors, more specifically Fortinet.

Fortinet FortiGate devices, when operating in Flow Based + Certificate-inspection (or even no-inspection) mode allow your browsers or devices to make the decision to trust the certificate, allowing traffic to pass. If you operate in Deep-inspection or proxy mode, FortiGate takes on more of a role by inspecting the default certificate chain. You know, the one that points back to the expired Root CA, as that "workaround" per Let's Encrypt is only going to work on Android devices. FortiGate will not look for any alternate chains, and this explains the headaches we all felt over the last few days.

The argument can be made "well it's more secure that way." and I would agree. But security unfortunately isn't the only thing we need to balance, one of my instructors in college said something that I still say to this day "You need to balance security with productivity, if you're too secure, you will impact productivity". So in this case, it may be overbearing as it's impact has been felt across the world. I am sure the folks at Fortinet are trying to figure out the most secure way (Read: without impacting production) to solve this, I don't know how that's done, I'm just a madman who rambles on.

"Blah blah blah! Get to the part where I fix my stuff!" The common workarounds have been to "put it in flow-based certificate-inspection" or "allow expired certs". I hate both of these approaches, as they either have an operational impact (number of policies needing touched for the first one, lesser security on the second). A third workaround, that I still don't particularly like but seems feasible is to disable the lookup of the X3 cert by creating a DNS blackhole for apps.identrust.com. If you choose to go this route, do so at your own peril, this workaround comes with no warranty or guarantees, like I said, I'm just a madman.

So with the rambling done here it is:

config system dns-database
    edit "apps.identrust.com"
        set domain "apps.identrust.com"
        set authoritative disable
        config dns-entry
            edit 1
                set hostname "@"
                set ip 127.0.0.1
            next
        end
    next
end
#
# restart wad to clear cache - If in proxy:
diag test application wad 99
# restart ips cert cache - if in flow based:
diag ips share clear cert_verify_cache

And that should do it! Keep in mind this is a WORKAROUND, at some point we will need to come back and disable this. That point is probably when Fortinet decides how it will address chains like this in the future.

I hope this helps. Thank you for reading!