Algolia reindex issue after upgrade to v3.5.2 ?

[peterpeterparker]

I upgraded Docusaurus v3.4.0 to v3.5.2 last week and only few cosmetic changes, did not implemented anything else.

Today the weekly Algolia re-indexing ended in error because it noticed more than 10% less entries:

[juno] Blocking error:
        The difference between the number of records:
          from : 2.22k
          to   : 1.99k
        is too large (10 %), this limit can be modified in the Editor (currently 10 %)

Before digging deeper, as it somehow feels related to upgrading Docusaurus, I was wondering if anyone else has faced such an issue too?

[slorber]

I've seen that kind of error occasionally for unexpected reasons. Try to reschedule a crawl and it might fix it. Or try to browse the Algolia pages to see if they tell you which records have disappeared.

Also, make sure that you don't delete docs versions while upgrading because your site will get less pages if you delete them.

If you compare your sitemap.xml before/after the upgrade, do you see any difference in terms of entries? Can you post a diff here? That might help you figure out what has changed.
(I mean a sitemap built using our sitemap plugin, for example https://docusaurus.io/sitemap.xml)

[slorber]

I never faced the issue before, that was a first. Not sure how to reschedule a crawl, I don't find that option in Algolia dashboard.

It's a bit confusing and I never remember how to access this page from their dashboard UI, but you can access crawlers dashboard here: https://crawler.algolia.com/admin/crawlers

CC @shortcuts didn't you make this easier to find? I still don't know how to navigate from one dashboard to another 😅

Edit: just found out you can access it this way:

Also, make sure that you don't delete docs version

Not sure what do you mean with "docs version"?

If your docs are versioned (v1 + v2) for example and you delete/archive v1 in the upgrade process, then /docs/v1 pages will not be online anymore so it's normal Algolia crawlers detect less entries.

If you compare your sitemap.xml before/after the upgrade

As far as I checked, sitemap.xml, before and after upgrade, are equals.

So that looks good to me.

@shortcuts considering the Docusaurus pages did not change, is there a way to know why the Algolia crawler see a decrease of the number of entries? Can we see details on entries that were previously found and are now absent?

Thanks

[shortcuts]

Hey there

Edit: just found out you can access it this way:

Yup that's the correct way for now, or bookmark the URL you shared is the way to go ahah it's a bit complicated to merge those two UIs together but there's an ongoing discussion!

@shortcuts considering the Docusaurus pages did not change, is there a way to know why the Algolia crawler see a decrease of the number of entries? Can we see details on entries that were previously found and are now absent?

The monitoring section of the crawler is the page will allow you to compare the last two crawls (cf screenshot), click the compare with previous crawl button, then view in monitoring, you'll see the diff of URLs crawled.

Sadly this doesn't include records diff, but when the Crawler prompts you with a Blocking error, your production index and the new crawl one are both available from from the Algolia dashboard, so you can browse it from there

[slorber]

Thanks @shortcuts

Yes comparing the 2 crawls stats won't be super helpful to troubleshoot the problem here:

It's not clear to me on the Algolia dashboard how I can browse both indices. Maybe I can only do so when the crawler is blocked?

[peterpeterparker]

Thanks a lot for the explanations, screenshots, and help, @slorber and @shortcuts! 🙏

After you pointed me to the "crawler" page of the indices, I restarted the indexing, and... it went through. So Sébastien, you were right all along. It seems this happens from time to time—this was a first for me—and it has nothing to do with the latest update of Docusaurus.

All good! The index on juno.build is up-to-date. ✅

[slorber]

Great to know thanks 👍