mirror of
https://github.com/hanzoai/email-verifier.git
synced 2026-10-30 15:04:29 +00:00
Jason MaandGitHub
7d177cc1ee
Running update.sh would have published 9543 free domains instead of 4479, over
5000 of them throwaway addresses -- whole machine-generated families such as
aws-mail-free-<n>.dynv6.net, msn-mail-free-<n>.dynv6.net and
gpt-mail-free-<n>.dynv6.net.
The primary free source had stopped being a free source.
Kikobeats/free-email-domains rebuilds domains.json in a daily cron by merging
three lists, two of which are disposable blocklists. Attributing all 13788
entries it publishes accounts for every one:
HubSpot free-domains-2.csv 4768 free
its own hand-maintained DOMAINS array 199 free
castle/disposable-email-domains 1000 disposable
disposable-email-domains/disposable_email_blocklist.conf 8740 disposable
---
present only in the two disposable blocklists 8821
update.sh subtracted one disposable source, tompec, which overlaps those two by
less than half, so the rest came through as free.
Subtracting the other two blocklists as well would have worked, but only for as
long as that list of three stayed accurate -- it would tie this repo's free list
to knowledge of another project's build script. Fetch the free data from where
Kikobeats gets it instead. HubSpot's list plus the tbrianjones gist already in
free_domain_sources.txt is the whole of the free content, and none of the
contamination follows.
Kikobeats' hand-maintained array is the one part with no upstream, and it is
worth keeping: docomo.ne.jp, so-net.ne.jp, ocn.ne.jp, biglobe.ne.jp,
plala.or.jp, softbank.ne.jp, kakao.com, chollian.net, aol.co.uk, chello.no and
the rest of the JP/KR/NO/DE carrier mailboxes, plus proton.me, pm.me and
protonmail.ch, which no other source lists. Vendored into
free_domains_extra.txt rather than scraped out of their .mjs at build time.
tempmail.net and spambox.xyz are left out; both are throwaway services rather
than providers.
Measured with castle+disposable-email-domains held out as an evaluation set --
fetched but never subtracted, so they only score the result:
entries still disposable .dynv6.net
Kikobeats + gist - tompec (before) 9543 4904 338
this change 4582 8 0
committed free.txt, for comparison 4479 10 0
So it is cleaner than the file in the repo today while adding 108 real
providers, and it needs no disposable source beyond the one already used.
Two more problems surfaced on the way.
free.txt has shipped "atlanticbb.net " for years. One entry in the
tbrianjones gist has a no-break space attached, and under LC_ALL=C
[[:space:]] does not match U+00A0, so every trim in this script walked past it
and IsFreeDomain("atlanticbb.net") returned false. normalise now folds U+00A0
to a space first -- through $'...' rather than a sed escape, which BSD sed does
not understand -- and a validation step drops anything still not shaped like a
hostname, reporting what it dropped rather than shrinking the list in silence.
That also catches "housefancom", a typo for housefan.com.
The same validation would drop twelve entries from the disposable list, all
IDNs spelled in Unicode instead of punycode, and all dead keys for the same
reason: IsDisposable converts its argument with domainToASCII before the
lookup. Converting them is the fix, not dropping them, and it belongs in a
change to the disposable pipeline, so that list is left alone here.
Adding a second URL to free_domain_sources.txt is also what would have exposed
the missing newline between appended sources, since the tbrianjones gist does
not end with one: the first attempt at this change produced "zzom.co.uk# free
mailbox providers that neither upstream source lists." as a domain. The parent
commit terminates each source explicitly, so it does not happen here.
Also: MAX_FREE. The floors added in the parent commit catch a source that
returns nothing; the ceiling catches a source that starts returning something
else, which is the failure that actually happened here. 9543 entries would have
been refused rather than published.
Two tests assert what the pipeline provides, both confirmed to fail when the
property is broken:
TestFreeAndDisposableDomainsAreDisjoint - the subtraction still runs
TestFreeDomainsAreWellFormedHostnames - no keys a lookup cannot match
disposable.txt also moves, from 123324 to 133602 entries. That is tompec's own
drift since the list was last regenerated; no source changed on that side.
README credited ivolo/disposable-email-domains and willwhite/freemail, neither
of which this repo has used for some time. Corrected to the sources actually
fetched, and to Kikobeats for the vendored list, which is MIT.
3 lines
182 B
Plaintext
3 lines
182 B
Plaintext
https://f.hubspotusercontent40.net/hubfs/2832391/Marketing/Lead-Capture/free-domains-2.csv
|
|
https://gist.githubusercontent.com/tbrianjones/5992856/raw/free_email_provider_domains.txt
|