For the avoidance of mystery: Google performs measurements of IPv6 connectivity and latency on an ongoing basis. The Google DNS servers do not return AAAA records to DNS resolvers if our measurements indicate that for users of those resolvers, HTTP/HTTPS access to dual-stack Google services is substantially worse than to equivalent IPv4-only services. “Worse” covers both reliability (e.g., failure to load a URL) and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over an ocean). The resolvers must also have a minimum query volume, which is fairly low.
I suggest checking if any of your affected users have broken 6to4 setups,
and that you are applying the relevant mitigations in RFC 6343.
MTU size issues and high latency have also both been mentioned as
possible reasons for the mysterious AAAA blacklist.