Select - Your Community
Select
Get Mobile App

Stream of Goodies

avatar

Techmeme

shared a link post in group #Stream of Goodies

Feed Image

www.techmeme.com

A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers

By David Pierce / The Verge. View the full context on Techmeme.

Comment here to discuss with all recipients or tap a user's profile image to discuss privately.

Embed post to a webpage :
<div data-postid="qnknxkd" [...] </div>
A group of likeminded people in Stream of Goodies are talking about this.