~tsileo/microblog.pub#120: 
Feature Request: Configurable robots.txt (or block /api)

I get way too many bot requests trying to access the Mastodon API endpoints.

A lot of those seem like well behaved bots, as they always request robots.txt first.

Would it make sense to add the /api endpoint to the robots.txt as disallowed? Or make the robots.txt easily configurable?

(I know I can change the code to block this endpoint, but that's not a very elegant solution)

Status
REPORTED
Submitter
João Costa
Assigned to
No-one
Submitted
9 months ago
Updated
9 months ago
Labels
No labels applied.

~tsileo 9 months ago

Hey,

Can you try it first by updating the robots.txt locally to see how effective it would be (and report back)?

Thanks!

João Costa 9 months ago · edit

Sure, I'll do that now and check next week.

On 03/02/2023 21:11, ~tsileo wrote:

Hey,

Can you try it first by updating the robots.txt locally to see how effective it would be (and report back)?

Thanks!

João Costa 9 months ago · edit

Indeed, it doesn't seem to make much of a difference :(

On 03/02/2023 21:22, João Costa wrote:

Sure, I'll do that now and check next week.

On 03/02/2023 21:11, ~tsileo wrote:

Hey,

Can you try it first by updating the robots.txt locally to see how effective it would be (and report back)?

Thanks!

Register here or Log in to comment, or comment via email.