KarlHeinzSchwuke@feddit.org to Technology@lemmy.worldEnglish · 2 months agoI was wrong about robots.txtevgeniipendragon.comexternal-linkmessage-square22linkfedilinkarrow-up192arrow-down116
arrow-up176arrow-down1external-linkI was wrong about robots.txtevgeniipendragon.comKarlHeinzSchwuke@feddit.org to Technology@lemmy.worldEnglish · 2 months agomessage-square22linkfedilink
minus-squareArchr@lemmy.worldlinkfedilinkEnglisharrow-up17arrow-down1·edit-22 months agoI feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article. Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.
minus-squareGeneral_Effort@lemmy.worldlinkfedilinkEnglisharrow-up7·2 months ago that is not how general news media has been talking about robots.txt. Ahh, yes. I think there is a lesson there.
I feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article.
Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.
Ahh, yes. I think there is a lesson there.