[GH-ISSUE #21] Maybe send a notification if robots.txt file doesn’t exist? #14

New issue

Closed

opened 2026-02-27 09:12:35 +03:00 by kerem · 4 comments

kerem commented

2026-02-27 09:12:35 +03:00

Owner

Originally created by @andrewminion-luminfire on GitHub (Jan 30, 2020).
Original GitHub issue: https://github.com/maelstrom-cms/odin/issues/21

We’re using this internally to monitor some client sites and noticed that those without a robots.txt result in failed horizon jobs:

GuzzleHttp\Exception\ClientException: Client error: `GET https://clientname.com/robots.txt` resulted in a `404 Not Found` response:

I would think the ideal handling is to catch that exception and send a notification that the file doesn’t exist. For now I can just disable robots monitoring on those sites.

Originally created by @andrewminion-luminfire on GitHub (Jan 30, 2020). Original GitHub issue: https://github.com/maelstrom-cms/odin/issues/21 We’re using this internally to monitor some client sites and noticed that those without a robots.txt result in failed horizon jobs: ```GuzzleHttp\Exception\ClientException: Client error: `GET https://clientname.com/robots.txt` resulted in a `404 Not Found` response:``` I would think the ideal handling is to catch that exception and send a notification that the file doesn’t exist. For now I can just disable robots monitoring on those sites.

kerem closed this issue

2026-02-27 09:12:35 +03:00

kerem commented

2026-02-27 09:12:36 +03:00

Author

Owner

@OwenMelbz commented on GitHub (Jan 30, 2020):

Do you definitely have robots.txt monitoring turned off in the settings??

@OwenMelbz commented on GitHub (Jan 30, 2020): Do you definitely have robots.txt monitoring turned off in the settings??

kerem commented

2026-02-27 09:12:36 +03:00

Author

Owner

@andrewminion-luminfire commented on GitHub (Jan 30, 2020):

No, robots.txt monitor was turned on in the settings for those sites, but I didn’t realize they don’t have an existing robots.txt file.

@andrewminion-luminfire commented on GitHub (Jan 30, 2020): No, robots.txt monitor was turned **on** in the settings for those sites, but I didn’t realize they don’t have an existing robots.txt file.

kerem commented

2026-02-27 09:12:36 +03:00

Author

Owner

@OwenMelbz commented on GitHub (Jan 31, 2020):

Okay, so to confirm - is there a problem?

If you do not have a robots.txt to monitor, then you should keep the setting turned off, this is only needed if you have a robots.txt to track

@OwenMelbz commented on GitHub (Jan 31, 2020): Okay, so to confirm - is there a problem? If you do not have a `robots.txt` to monitor, then you should keep the setting turned off, this is only needed if you have a `robots.txt` to track

kerem commented

2026-02-27 09:12:36 +03:00

Author

Owner

@andrewminion-luminfire commented on GitHub (Jan 31, 2020):

When you put it that way …no, there’s not really a problem 😊
On Jan 30, 2020, 6:41 PM -0600, Owen Melbourne notifications@github.com, wrote:

Okay, so to confirm - is there a problem?
If you do not have a robots.txt to monitor, then you should keep the setting turned off, this is only needed if you have a robots.txt to track
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

@andrewminion-luminfire commented on GitHub (Jan 31, 2020): When you put it that way …no, there’s not really a problem 😊 On Jan 30, 2020, 6:41 PM -0600, Owen Melbourne <notifications@github.com>, wrote: > Okay, so to confirm - is there a problem? > If you do not have a robots.txt to monitor, then you should keep the setting turned off, this is only needed if you have a robots.txt to track > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub, or unsubscribe.