fix(ingress): CAA correctness for Route53, wildcards and the dnsguide check

custom-domain/dstack-ingress/DNS_PROVIDERS.md

@@ -108,7 +108,9 @@ PolicyDocument:
 
 **Important Notes for Route53:**
 - The certbot plugin uses the format `certbot-dns-route53` package
-- CAA will merge AWS & Let's Encrypt CA domains to existing records if they exist
+- CAA values are merged into the existing record set at the name: only values with the same tag are replaced, so `iodef` and other tags are kept
+- Where a CAA record shares the name (`DELEGATION_ZONE`, or `SET_CAA=true` on a non-wildcard domain), the gateway pointer is an A record to the resolved gateway IP instead of a CNAME, replacing any CNAME in the same change. It is refreshed every `RENEW_INTERVAL`
+- Wildcard record names are listed by Route53 as `\052.<domain>`; they are matched as `*.<domain>`
 - It is essential that the AWS service account used can only assume the limited role. See cloudformation example.
 
 ## Docker Compose Examples

custom-domain/dstack-ingress/README.md

@@ -29,7 +29,9 @@ dstack-ingress is a HAProxy-based L4 (TCP) proxy that provides:
 You can use a wildcard domain (e.g. `*.myapp.com`) to route all subdomains to a single dstack application:
 
 - The TXT record is automatically set as `_dstack-app-address-wildcard.myapp.com` (instead of `_dstack-app-address.*.myapp.com`)
-- CAA records use the `issuewild` tag on the base domain
+- CAA records use both the `issuewild` and `issue` tags on the base domain: `issuewild` alone would leave other CAs free to issue for non-wildcard subdomains
+- If the base domain is a zone apex (`*.myapp.com` served off the `myapp.com` zone), that `issue` record replaces any existing `issue` values there and governs every name in the zone without its own CAA record, so other CAs or ACME accounts can no longer issue for them. The container warns about this; give those names their own CAA records, or set `SET_CAA=false`. On Route53 this also removes the Amazon `issue` values that older versions wrongly added at the apex
+- Two deployments with different ACME accounts, one serving `myapp.com` and one `*.myapp.com`, both write the `issue` record at `myapp.com` and overwrite each other's on every pass. Serve both names from one deployment, or set `SET_CAA=false` on one of them
 - Requires dstack-gateway with wildcard TXT resolution support ([dstack#545](https://github.com/Dstack-TEE/dstack/pull/545))
 
 ```yaml
@@ -173,7 +175,7 @@ otherwise arbitrary — the gateway answers every name under its domain.
 | `PORT` | `443` | HAProxy listen port |
 | `DOMAINS` | | Multiple domains, one per line or one comma-separated line. Without `ROUTING_MAP`, all of them go to `TARGET_ENDPOINT` |
 | `ROUTING_MAP` | | Multi-domain routing: `domain=host:port` per line. A wildcard entry (`*.example.com=host:port`) matches one label, and exact entries take precedence over it. Unmatched names go to the first entry |
-| `SET_CAA` | `false` | Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS) |
+| `SET_CAA` | `false` | Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS). Replaces existing values with the same tag; for a wildcard whose base is a zone apex this restricts the whole zone (see [Wildcard Domain Support](#wildcard-domain-support)). Except on Cloudflare, a non-wildcard domain is then published as an A record to the gateway's current address instead of a CNAME, because a CNAME would hide the CAA beside it; the address is refreshed every `RENEW_INTERVAL`. Turning `SET_CAA` off later leaves the CAA record in place, so the A record stays until you delete the CAA |
 | `TXT_PREFIX` | `_dstack-app-address` | DNS TXT record prefix |
 | `ACME_EMAIL` | | ACME contact address, in either mode. `CERTBOT_EMAIL` is the historical name and still works. See [below](#the-acme-contact-address-is-optional-and-public) — it is optional, and published |
 | `ACME_STAGING` | `false` | Use Let's Encrypt staging, in either mode. `CERTBOT_STAGING` is the historical name and still works |
@@ -238,22 +240,23 @@ _acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.dele
 dstack-ingress publishes what they point at, in the delegation zone:
 
 ```
-svc.example.com.deleg.example.net                      CNAME  <GATEWAY_DOMAIN>
+svc.example.com.deleg.example.net                      A      <address of GATEWAY_DOMAIN>
 svc.example.com.deleg.example.net                      CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
 _dstack-app-address.svc.example.com.deleg.example.net  TXT    "<app-id>:<port>"
 _acme-challenge.svc.example.com.deleg.example.net      TXT    <challenge>   (transient)
 ```
 
-The gateway pointer and the CAA share a name, which DNS does not allow for a
-CNAME. Which form works is a property of the provider — Cloudflare tolerates the
-pair, Linode publishes an address record instead — and `set_alias_record` in the
-provider layer is where that choice is made. Delegation asks for an alias and
-takes what it gets. A provider that needs a form other than CNAME overrides that
-method, as Linode does.
+The gateway pointer and the CAA share a name. Cloudflare answers a CAA query
+with the CAA beside a CNAME, so the pointer stays a CNAME there. Elsewhere it is
+an A record: by the DNS rules resolvers follow a CNAME without looking beside it,
+and Route53 and Linode refuse the pair. The address
+is the one `GATEWAY_DOMAIN` resolves to when the container writes it, and it is
+refreshed every `RENEW_INTERVAL`, so a gateway move reaches DNS only then, and
+only one of the gateway's addresses is published.
 
 **A wildcard needs a fourth CNAME.** RFC 8659 evaluates `*.app.example.com` at
-`app.example.com`, so the base gets its own alias and the container publishes an
-`issuewild` CAA behind it:
+`app.example.com`, so the base gets its own alias and the container publishes
+`issuewild` and `issue` CAA records behind it:
 
 ```
 *.app.example.com                            CNAME  app.example.com.deleg.example.net
@@ -577,6 +580,10 @@ by both, so use it: reject envelopes more than a few minutes old, and remember,
 the last one you applied. Reject anything older. A failed POST is retried with the identical envelope, so treat a
 byte-identical `payload` as a retry of the one already applied rather than a new request.
 
+`records` can hold several records with the same type and name: a wildcard asks
+for both an `issue` and an `issuewild` CAA at its base. Create each one; do not
+replace by type and name alone.
+
 ### Why lego and not certbot
 
 certbot cannot do tls-alpn-01. Its standalone plugin is HTTP-01 only, the

custom-domain/dstack-ingress/TESTING.md

@@ -251,7 +251,7 @@ authoritative nameserver — it should be gone).
 The CAA comes last and needs nothing from you: it names the ACME account, so the
 account has to exist first, and the container publishes it into the delegation
 zone itself once it does. Expect `Adding CAA record for <domain>.<zone> with tag
-issue` on the first pass — `issuewild` for a wildcard, on the base name. Confirm
+issue` on the first pass — `issue` and `issuewild` for a wildcard, on the base name. Confirm
 it at the authoritative nameserver, and confirm it resolves *through* the
 operator's CNAME, since that is what the CA follows:
 
@@ -351,6 +351,7 @@ this before suspecting the proxy.
 
 ```bash
 python3 scripts/tests/test_dnsguide.py     # DNS/CAA parsing and record building
+python3 scripts/tests/test_route53.py      # Route53 record names and changes
 bash    scripts/tests/test_sanitizers.sh   # env var validation
 ```
 

custom-domain/dstack-ingress/scripts/acme-dns-alias-hook.sh

@@ -35,7 +35,9 @@ case "$action" in
     auth)
         : "${CERTBOT_VALIDATION:?CERTBOT_VALIDATION not set}"
         echo "acme-dns-alias-hook: writing challenge TXT to delegated record $record"
-        dnsman.py set_txt --domain "$record" --content "$CERTBOT_VALIDATION"
+        # A stale challenge value that cannot be deleted is harmless: the CA
+        # accepts the record set as long as one value matches.
+        dnsman.py set_txt --tolerate-stale --domain "$record" --content "$CERTBOT_VALIDATION"
         # certbot asks Let's Encrypt to validate immediately after this hook
         # returns, so wait for the record to propagate before returning.
         #

custom-domain/dstack-ingress/scripts/dns01.sh

@@ -93,9 +93,13 @@ EOF
 }
 
 set_alias_record() {
-    local domain="$1"
+    local domain="$1" with_caa=()
+    # A plain name gets its CAA beside this record.
+    if [ "$SET_CAA" = "true" ] && [[ "$domain" != \*.* ]]; then
+        with_caa=(--with-caa)
+    fi
     echo "Setting alias record for $domain"
-    dnsman.py set_alias \
+    dnsman.py set_alias "${with_caa[@]}" \
         --domain "$domain" \
         --content "$GATEWAY_DOMAIN"
 
@@ -130,10 +134,9 @@ set_txt_record() {
 # never needs touching again -- not when the app id changes with the compose,
 # not when the ACME account is recreated, not when the gateway moves.
 #
-# The gateway pointer and the CAA end up on one name, which DNS does not allow
-# for a CNAME. Which form works is a provider property -- Cloudflare tolerates
-# the pair, Linode publishes an address record instead -- and set_alias_record
-# is where that knowledge lives, so ask for an alias and let the provider pick.
+# The gateway pointer and the CAA end up on one name. Which form of pointer
+# lets the CAA take effect is a provider property, so tell the provider and let
+# it pick (an address record unless its servers answer CAA beside a CNAME).
 delegated_name() {
     local domain="${1#\*.}"
     echo "${domain}.${DELEGATION_ZONE}"
@@ -143,7 +146,7 @@ delegation_publish() {
     local domain="$1" target txt_name
     target=$(delegated_name "$domain")
 
-    dnsman.py set_alias --domain "$target" --content "$GATEWAY_DOMAIN" || return 1
+    dnsman.py set_alias --with-caa --domain "$target" --content "$GATEWAY_DOMAIN" || return 1
 
     txt_name="$(txt_record_name "$domain").${DELEGATION_ZONE}"
     dnsman.py set_txt --domain "$txt_name" --content "$(txt_record_value)" || return 1
@@ -152,18 +155,21 @@ delegation_publish() {
 # The accounturi CAA goes on the delegated name, beside the gateway pointer.
 #
 # A wildcard is evaluated at its base (RFC 8659), so it needs the `issuewild`
-# tag and it needs the operator to have aliased the base as well -- dnsguide
+# tag (beside `issue`) and it needs the operator to have aliased the base as well -- dnsguide
 # asks for that fourth CNAME. A base that is a zone apex cannot be aliased at
 # all, so such a domain is not a fit for delegation; the CAA check reports the
 # record missing rather than the container pretending to have set it.
 delegation_publish_caa() {
-    local domain="$1" account_file account_uri
+    local domain="$1" account_file account_uri tag
     account_file=$(get_letsencrypt_account_file) || return 1
     account_uri=$(jq -j '.uri' "$account_file")
-    dnsman.py set_caa \
-        --domain "$(delegated_name "$domain")" \
-        --caa-tag "$(caa_tag_for "$domain")" \
-        --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$account_uri"
+    for tag in $(caa_tags_for "$domain"); do
+        dnsman.py set_caa \
+            --domain "$(delegated_name "$domain")" \
+            --caa-tag "$tag" \
+            --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$account_uri" ||
+            return 1
+    done
 }
 
 # Challenge delegation: this container holds no token for the served domain's
@@ -194,6 +200,16 @@ delegation_guide() {
         "$@"
 }
 
+# Only a zone apex answers SOA for itself. Best effort: a failed lookup says no.
+is_zone_apex() {
+    local resolvers="${DOH_RESOLVERS:-${DOH_RESOLVER:-https://dns.google/resolve}}"
+    curl -fsS --max-time 10 -H 'accept: application/dns-json' \
+        "${resolvers%%,*}?name=$1&type=SOA" 2>/dev/null |
+        jq -e --arg name "${1,,}" \
+            'any(.Answer[]?; .type == 6 and (.name | ascii_downcase | rtrimstr(".")) == $name)' \
+            >/dev/null
+}
+
 set_caa_record() {
     local domain="$1"
 
@@ -224,20 +240,26 @@ set_caa_record() {
 
     local caa_domain caa_tag
     caa_domain="${domain#\*.}"
-    caa_tag=$(caa_tag_for "$domain")
-
-    ACCOUNT_URI=$(jq -j '.uri' "$account_file")
-    echo "Adding CAA record ($caa_tag) for $caa_domain, accounturi=$ACCOUNT_URI"
-    dnsman.py set_caa \
-        --domain "$caa_domain" \
-        --caa-tag "$caa_tag" \
-        --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$ACCOUNT_URI"
 
-    if [ $? -ne 0 ]; then
-        echo "Warning: Failed to set CAA record for $domain"
-        echo "This is not critical - certificates can still be issued without CAA records"
-        echo "Consider disabling CAA records by setting SET_CAA=false if this continues to fail"
+    if [ "$FIRST_PASS" = "true" ] && [[ "$domain" == \*.* ]] && is_zone_apex "$caa_domain"; then
+        echo "Warning: $caa_domain is a zone apex. The CAA issue record published there for $domain"
+        echo "replaces every existing issue value and governs every name in the zone that has no CAA"
+        echo "record of its own, so other CAs and ACME accounts can no longer issue for them."
+        echo "Give those names their own CAA records, or set SET_CAA=false."
     fi
+
+    ACCOUNT_URI=$(jq -j '.uri' "$account_file")
+    for caa_tag in $(caa_tags_for "$domain"); do
+        echo "Adding CAA record ($caa_tag) for $caa_domain, accounturi=$ACCOUNT_URI"
+        if ! dnsman.py set_caa \
+            --domain "$caa_domain" \
+            --caa-tag "$caa_tag" \
+            --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$ACCOUNT_URI"; then
+            echo "Warning: Failed to set CAA record for $domain"
+            echo "This is not critical - certificates can still be issued without CAA records"
+            echo "Consider disabling CAA records by setting SET_CAA=false if this continues to fail"
+        fi
+    done
 }
 
 process_domain() {