~tsileo/microblog.pub#120: Feature Request: Configurable robots.txt (or block /api)

I get way too many bot requests trying to access the Mastodon API endpoints.

A lot of those seem like well behaved bots, as they always request robots.txt first.

Would it make sense to add the /api endpoint to the robots.txt as disallowed? Or make the robots.txt easily configurable?

(I know I can change the code to block this endpoint, but that's not a very elegant solution)

Status: REPORTED
Submitter: João Costa
Assigned to: No-one
Submitted: 9 months ago
Updated: 9 months ago
Labels: No labels applied.

~tsileo 9 months ago

Hey,

Can you try it first by updating the robots.txt locally to see how effective it would be (and report back)?

Thanks!

João Costa 9 months ago · edit

Sure, I'll do that now and check next week.

On 03/02/2023 21:11, ~tsileo wrote:

Hey,

Can you try it first by updating the robots.txt locally to see how effective it would be (and report back)?

Thanks!

João Costa 9 months ago · edit

Indeed, it doesn't seem to make much of a difference :(

On 03/02/2023 21:22, João Costa wrote:

Sure, I'll do that now and check next week.

On 03/02/2023 21:11, ~tsileo wrote:

Hey,

Can you try it first by updating the robots.txt locally to see how effective it would be (and report back)?

Thanks!