robots.txt

ZDNet Resources

• sort by:
• Relevance
• Date
• Popularity

How to evade Google search

How to evade Google searchGeez -- Web development 101Someone over at Dell should read the 'Webmaster Tips' page at Google. It couldn't be any clearer. Better luck next time, boys!Security basicsIf you don't want it accessed from the web, then fer-cryin-out-loud don't put it where it's accesable from the web....

Tags: Channel management, Laziness, Google Inc., Web, Robots.txt, Google Search

Discussion threads 2006-02-01

Robots.txt Editor 2.0 (Windows)

Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the robots.txt files required to instruct search engine spiders, which are parts of a Web site are not to be indexed and made searchable by the general...

Tags: Web, Search Engine, Spider, Microsoft Windows, NetPromoter, Robots.txt, Search, Web Site Development, Channel Management, Internet, Marketing

Software downloads 2005-08-01

Additional Resources

iGooMap 1.7.3 (Mac)

With iGooMap, you can automatically create XML Sitemaps conforming to the sitemap protocol or import and edit existings sitemaps visually. You can upload the sitemap to your server and notify all search engines supporting sitemaps. Easy handling, just enter the URL of your web site. You can scan...

Tags: Web, Apple Macintosh, Search Engine, Server, Web Site, PointWorks Software Development, Web Site Development, XML, Web Technology, Channel Management, Search, Internet, Software/Web Development, Web Development, Marketing

Software downloads 2009-06-14

A1 Sitemap Generator 1.8.6 (Windows)

Help search engines crawl and index your Web site with HTML, RSS, and XML sitemaps as defined by Google XML sitemaps protocol including video, mobile and news XML sitemaps. Near-endless Web site scanning options. Configure amount of simultaneous connections to use. Supports crawler filters, nofollow, robots.txt, custom connection and read...

Tags: Web, Microsoft Windows, Web Site, Micro-Sys, Web Site Development, XML, Web Technology, CSS, Channel Management, Internet, Software/Web Development, Web Development, Scripting Languages, Marketing

Software downloads 2009-04-13

Everyone has a secret life but will Google find it?

Everyone has a secret life but will Google find it?Better still ?"I have even better advice: be very active on the Internet ? anything bad will be diluted into the long tail of you."I should prefer "never do anything bad and publish only high quality items online" ...... especially good...

Tags: SEARCH, Google Inc., secret life

Discussion threads 2009-03-04

RTSoftwares Sitemap Generator 1.0 (Windows)

Free Sitemap Generator tool to generate HTML/TXT sitemap. Sitemap generated by sitemap generator is robots.txt compatible. Easy-to-use sitemap generator for Web masters. One click sitemap generation with many configurable options. Download your copy of free sitemap generator and create rich html sitemap for your Web site with a simple button...

Tags: Generator, RTSoftwares, Channel Management, Web Site Development, Microsoft Windows, XML, Productivity, Marketing, Internet, Operating Systems, Software, Software/Web Development, Web Development

Software downloads 2009-01-25

RedX Meta Tag Builder 1.5 (Windows)

A freeware meta-tag generator which allows you to quickly build HTML meta tags for you websites, including (title, description, keywords, robots, categories, author, e-mail, expires, copyright, generator, language, rating, and the optional the refresh tag). This easy to use software allows you to build meta tags and then to save...

Tags: Microsoft Windows, Tag, RedXsoftware, Robots, Emerging Technologies

Software downloads 2009-01-08

Chilkat Spider ActiveX 2.2.2 (Windows)

Crawl a Web site with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Automatically handles HTTPS URLs. Includes safety measures for avoiding large / infinite size Web pages or infinitely growing-in-length URLs. Includes a wind-down features for crawling a maximum number of pages per...

Tags: Web, Microsoft Windows, ActiveX, Chilkat Software, ActiveX/COM/COM+/DCOM, Middleware, Channel Management, Software Development, Software/Web Development, Enterprise Software, Software, Marketing

Software downloads 2008-12-15

Google opens up your profile to search engines

Google has been letting people create profiles for a while now -- you can see mine here, and you can create yours here. Until today, your public profile that you created has remained hidden from search engines. Just about a half hour ago, Google added a...

Tags: Profile, Google Inc., Search Engine, Search, Garett Rogers

Blog posts 2008-10-15

Google PPOB? What is it?

Google PPOB? What is it?Classified Ads aggregator....Looks like a classified ad aggregator...in russianhttp://google.ru/ppob Google Classifieds BetaRE: Google PPOB? What is it?Garret, this was discussed here (the 1st result for [google ppob robots.txt] in Google):http://blogoscoped.com/archive/2008-10-08-n14.htmlGet with the program dude! ;-)

Tags: Google Inc., Google PPOB

Discussion threads 2008-10-09

Google PPOB? What is it?

When you are looking for things that Google is about to do, the best place to keep your eye on is Google's robots.txt file -- a document they keep updated with a list of websites they don't want search engines to index. A week ago, they added "/search2001" which...

Tags: Google Inc., Internet, Web Site Development, Search, Web Technology, Garett Rogers

Blog posts 2008-10-09

Google to go back in time?

Warning, this post is speculative, but I think all signs point to what I am about to describe. Google may be about to launch a search tool that uses the same algorithms used way back in 2001. Why would they do this? I think it's probably the...

Tags: Google Inc., Search Engine, Entry, Search, Robots, Emerging Technologies, Garett Rogers

Blog posts 2008-09-30

Is Cuil killing the Internet?

Is Cuil killing the Internet?Brute force indexing?I have an inkling Cuil might not be everything it's touted to be. The only reason for brute force indexing is to fill in the gaps where you've come up short in your harvesting/indexing algorithms.RE: Is Cuil killing the Internet?I used Cuil to...

Tags: Cuil, Internet

Discussion threads 2008-09-02

Linking The Web 2 (Windows)

Linking The Web is a free utility that regularly monitors all backlinks to your sites. If some of the links don't exist, contain wrong anchor, are covered by a "nofollow" flag or hidden by robots.txt then you will be immediately notified by email. Also the program can keep track of...

Tags: Web, Link, S5 Development, Microsoft Windows, Channel Management, E-mail, Operating Systems, Software, Marketing, Online Communications

Software downloads 2008-07-14

New Google Mobile feature lets you search without typing

Google added a new entry into their robots.txt file -- one way you can keep tabs on what Google doesn't want to see. The new entry that forbids crawlers from seeing http://www.google.com/m/lcb made me naturally curious. The new feature, that from what I can tell is...

Tags: Google Inc., Mobile, Feature, Advertising & Promotion, Marketing, Garett Rogers

Blog posts 2008-03-22

SpidEye 1 (Windows)

SpidEye is like a spider's eye, showing what a Web Crawler would see while navigating the Web. From each page visited it collects and summaries relevant information that are meant to assist Web masters on the design of their Web pages for bots. This freeware provides full HTML navigation, displays...

Tags: Web, Microsoft Windows, Software Atelier, SpidEye, Channel Management, Marketing

Software downloads 2007-12-20

Google will soon support "Unavailable After" META tag

Barry Schwartz found an interesting article from highrankings.com that gives us some juicy details about what it takes to get a good rank on Google and what tools we can look for soon. Dan Crow, director of crawl systems at Google, attended the annual SEMNE (Search Engine Marketing New...

Tags: Google Webmasters

Blog posts 2007-07-14

Newspapers want Google News' quarter

Newspapers want Google News' quarterNewspapers want Google News' quarterGoogle short changes the public again. When are people going to wake up and realize Google doesn't run the world. I'm sure these newspapers are going to want back pay as well as current royalties severely affecting Google's bottom line....

Tags: Google News, Google Only, Google Inc.

Discussion threads 2007-05-23

Google News Flash: Google.com SERPs get News, and images!

Google News Flash: Google.com SERPs get News, and images!Old - KindaThis was covered by Chris Sherman on Apr. 19, 2007 at http://searchengineland.com/070419-150004.phpI then posted examples on April 26, 2007 at http://www.seroundtable.com/archives/013264.htmlThis is getting to be a really tiring one-sided debate...People expect Google to be a search engine. Normal search...

Tags: SEARCH, Google Inc., search engine, Internet, Web page, Web, Google News

Discussion threads 2007-05-09

Google cash-free caching gets boost with 'Webmaster Tools'

Google cash-free caching gets boost with 'Webmaster Tools'Machine generated content is CheapAt last Google, you are allowing to download one of my data SearchHistoryhttp://www.google.com/robots.txt"User-agent: *Allow: /searchhistory/"Why Google disallowing all other data? For Google, User data is free, media content is free. but their Index data is costly? Not really. Machine...

Tags: Channel management, Google Inc.

Discussion threads 2007-04-18