HomeData report

Israeli websites and AI crawlers: data from 752 co.il sites, October 2026

· LEADZ team

These are basic conditions for a site to be read, not a measure of visibility.

In short: On 16 October 2026 we read the homepage of 752 websites on the co.il domain, drawn at random from a public web archive. On 6.5% of them the homepage is almost empty when JavaScript is not run. 68.2% have structured data (JSON-LD) on the homepage, but only 11.4% include the business details in it. Only 2 of 750 sites block an AI search crawler in robots.txt. 22.4% have an llms.txt file, including almost every Wix site in the sample (63 of 64).

The report describes what a crawler sees when it reads a site from the outside. It does not check whether these sites appear in AI answers, and it gives no score. The full report is in Hebrew.

Findings

FindingShareSites95% interval
Homepage almost empty without JavaScript (under 50 words)6.5%49 of 7525%–8.5%
Structured data (JSON-LD) on the homepage68.2%513 of 75264.8%–71.4%
Business details in the structured data (name, and phone or address)11.4%86 of 7529.4%–13.9%
robots.txt blocks at least one AI search crawler0.3%2 of 7500.1%–1%
robots.txt blocks at least one AI training crawler0.8%6 of 7500.4%–1.7%
No robots.txt (404 or 410)10.4%78 of 7508.4%–12.8%
robots.txt address returns a web page instead of a file8.4%63 of 7506.6%–10.6%
No sitemap (404 or 410)9.4%70 of 7417.5%–11.8%
Sitemap address returns something that is not a sitemap4.7%35 of 7413.4%–6.5%
llms.txt file present22.4%166 of 74219.5%–25.5%
Homepage served over HTTPS95.5%718 of 75293.7%–96.7%
H1 heading on the homepage65%489 of 75261.5%–68.4%
Meta description on the homepage69.1%520 of 75265.8%–72.3%
Page language declared (lang)93.6%704 of 75291.6%–95.2%

Why the base changes: 752 sites were counted. When a file could not be read unambiguously (for example, the server did not answer in time), the site was left out of that row only: 2 sites in the robots.txt rows, 11 in the sitemap rows and 10 in the llms.txt row.

AI crawlers blocked in robots.txt

CrawlerTypeSites that block it95% interval
OAI-SearchBotSearch2 of 750 (0.3%)0.1%–1%
Claude-SearchBotSearch2 of 750 (0.3%)0.1%–1%
PerplexityBotSearch2 of 750 (0.3%)0.1%–1%
GPTBotTraining5 of 750 (0.7%)0.3%–1.6%
ClaudeBotTraining6 of 750 (0.8%)0.4%–1.7%
Google-ExtendedTraining5 of 750 (0.7%)0.3%–1.6%

A search crawler reads pages to answer a question now. A training crawler collects text to train models. Blocking a company's training crawler does not block its search crawler. The table shows only what a site declares in robots.txt; blocking at a firewall is not visible from the outside and was not counted.

By website platform, in brief

The platform was detected from signs in the homepage code. Groups differ in size and in the kind of sites they hold, so these figures describe what was found and do not compare platforms.

  • Homepage almost empty without JavaScript: 2.3% of 481 WordPress sites (including Elementor), and 18.7% of the 171 sites whose platform was not detected.
  • llms.txt: 63 of 64 Wix sites (98.4%, interval 91.7%–99.7%). According to Wix's help center, Wix generates and maintains this file automatically (read on 16 October 2026). We did not check whether any engine reads the file.
Source

The full breakdown by platform is in the Hebrew report and in the data file.

What the data does not say

These are basic conditions for a site to be read, not a measure of visibility. The report does not check whether a site appears in the answers of ChatGPT, Gemini, Perplexity or any other engine, and it gives no score. A site that meets every row in the table is not guaranteed to appear in an answer, and a site that meets none of them may still appear.

How we checked

  • Where the sites came from: the public Common Crawl archive, crawl CC-MAIN-2026-39 (September 2026). We took every registered co.il domain with at least one page the archive fetched successfully: 67,293 sites.
  • The draw: 800 sites at random, with a fixed seed (20261016), so a repeat run returns the same list.
  • The reading: on the morning of 16 October 2026. For each site we read the homepage, robots.txt, llms.txt and the sitemap once, without running JavaScript. The crawler identified itself by its own name (LeadzAIScan), made no requests under an AI crawler's name, and sent no more than six requests per site.
  • What was excluded: 48 of 800. 19 did not answer, 17 returned an error, 10 redirected to another domain, and 2 were parked or for-sale pages. 752 sites were counted.
  • Definitions: "almost empty" = under 50 words in the homepage HTML. "Structured data" = at least one valid JSON-LD block. "Business details" = a business entity with a name, and a phone or address. "Blocks" = what robots.txt says about the homepage. "No file" = the server answered 404 or 410.
  • Interval: 95% Wilson interval, in percent.
  • Second check: 20 sites from the sample were read again the same day by a different method. 179 of 180 comparisons matched; the one difference was in the definition of a sitemap, not a reading error.
  • Privacy: the list of sites and the result for each site are not published. Only totals are.

Limitations

  • co.il only. Israeli businesses on .com, .il or other domains were not included. co.il is meant for commercial bodies, but we did not check that every site in the sample belongs to an active business.
  • Only what the archive knows. A site no one links to is probably not in the archive, so it could not be drawn.
  • Homepage only. Inner pages were not checked.
  • No JavaScript. Some crawlers do run JavaScript. The "almost empty" row describes only a crawler that does not.
  • Only what is declared. Blocking at a firewall or a protection service is not visible from the outside.
  • Structured data not checked for accuracy. We counted whether it exists, not whether the details in it are correct.
  • Partial platform detection. No platform was detected on 171 sites; they are shown as a group of their own.
  • One reading, one morning. A site that was down at that hour was excluded, or left out of that row.

Data and citation

The totals are available as a CSV file (no domains). Column finding_id matches the rows above; the labels in the file are in Hebrew. Full report in Hebrew: "מה מצאנו ב-752 אתרים ישראליים".

How to cite: LEADZ, "Israeli websites and AI crawlers: data from 752 co.il sites", October 2026 edition. 752 co.il sites, read on 16 October 2026, drawn from Common Crawl CC-MAIN-2026-39.

Download the totals (CSV)