robots.txt

ZDNet Resources

• sort by:
• Relevance
• Date
• Popularity

How to evade Google search

How to evade Google searchGeez -- Web development 101Someone over at Dell should read the 'Webmaster Tips' page at Google. It couldn't be any clearer. Better luck next time, boys!Security basicsIf you don't want it accessed from the web, then fer-cryin-out-loud don't put it where it's accesable from the web....

Tags: Channel management, Laziness, Google Inc., Web, Robots.txt, Google Search

Discussion threads 2006-02-01

Robots.txt Editor 2.0 (Windows)

Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the robots.txt files required to instruct search engine spiders, which are parts of a Web site are not to be indexed and made searchable by the general...

Tags: Web, Search Engine, Spider, Microsoft Windows, NetPromoter, Robots.txt, Search, Web Site Development, Channel Management, Internet, Marketing

Software downloads 2005-08-01

Additional Resources

iGooMap 1.8 (Mac)

With iGooMap, you can automatically create XML Sitemaps conforming to the sitemap protocol or import and edit existings sitemaps visually. You can upload the sitemap to your server and notify all search engines supporting sitemaps. Easy handling, just enter the URL of your web site. You can scan...

Tags: Web, Apple Macintosh, Search Engine, Server, Web Site, PointWorks Software Development, Web Site Development, XML, Web Technology, Channel Management, Internet, Software/Web Development, Web Development, Marketing

Software downloads 2009-10-27

mySitemap2 2.10 (Mac)

mySitemap2 allows you to easily create HTML and XML sitemaps Google Sitemaps by scanning websites either locally or already on the internet. Once scanned, mySitemap2 will allow you to view, edit and export the sitemap for inclusion on your website. Creates both HTML and XML sitemaps Scan websites locally or...

Tags: HTML, Apple Macintosh, Web Site, Tangled Software, mySitemap2, Web Site Development, XML, Web Technology, Internet, Software/Web Development, Web Development

Software downloads 2009-10-26

WSJ chief criticizes news aggregators - creators carry cost burden

More important...than net neutrality? You are out of your mind. You probably think you are a journalist as well. What a joke. Do you even proof read your own dribble? It's obvious you don't. What kind of journalist would send out content with grammar and spelling errors? it has...

Tags: Advertising & Promotion, advertisement, Google Inc., revenue

Discussion threads 2009-10-23

Newsmakers: Be careful what you wish for

Without Google they get no trafficIt does not take a rocket scientist to know that without Google, most of this news sites would probably have little to now traffic at all.So how is Google stealing from them? Since when is sending customers to your door a bad thing?Without the...

Tags: Web site development, Operational accounting, Google Inc., Google News

Discussion threads 2009-10-11

Newsmakers: Be careful what you wish for

At the World Media Summit, Tom Curley and Rupert Murdoch took turns bashing Google over their klepto antics and ruthless news stealing behavior. They have been stealing their content for too long, and it's about time they paid up. The truth is, they can prevent Google...

Tags: Google Inc., Garett Rogers

Blog posts 2009-10-11

RedX Meta Tag Builder 1.7 (Windows)

A freeware meta-tag generator which allows you to quickly build HTML meta tags for you websites, including (title, description, keywords, robots, categories, author, e-mail, expires, copyright, generator, language, rating, and the optional the refresh tag). This easy to use software allows you to build meta tags and then to save...

Tags: Generator, HTML, Microsoft Windows, RedXsoftware, Robots, Emerging Technologies

Software downloads 2009-09-22

Everyone has a secret life but will Google find it?

Everyone has a secret life but will Google find it?Better still ?"I have even better advice: be very active on the Internet ? anything bad will be diluted into the long tail of you."I should prefer "never do anything bad and publish only high quality items online" ...... especially good...

Tags: SEARCH, Google Inc., secret life

Discussion threads 2009-03-04

RTSoftwares Sitemap Generator 1.0 (Windows)

Free Sitemap Generator tool to generate HTML/TXT sitemap. Sitemap generated by sitemap generator is robots.txt compatible. Easy-to-use sitemap generator for Web masters. One click sitemap generation with many configurable options. Download your copy of free sitemap generator and create rich html sitemap for your Web site with a simple button...

Tags: Generator, RTSoftwares, Channel Management, Web Site Development, Microsoft Windows, XML, Productivity, Marketing, Internet, Operating Systems, Software, Software/Web Development, Web Development

Software downloads 2009-01-25

Chilkat Spider ActiveX 2.2.2 (Windows)

Crawl a Web site with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Automatically handles HTTPS URLs. Includes safety measures for avoiding large / infinite size Web pages or infinitely growing-in-length URLs. Includes a wind-down features for crawling a maximum number of pages per...

Tags: Web, Microsoft Windows, ActiveX, Chilkat Software, ActiveX/COM/COM+/DCOM, Middleware, Channel Management, Software Development, Software/Web Development, Enterprise Software, Software, Marketing

Software downloads 2008-12-15

Google opens up your profile to search engines

Google has been letting people create profiles for a while now -- you can see mine here, and you can create yours here. Until today, your public profile that you created has remained hidden from search engines. Just about a half hour ago, Google added a...

Tags: Profile, Google Inc., Search Engine, Search, Garett Rogers

Blog posts 2008-10-15

Google PPOB? What is it?

Google PPOB? What is it?Classified Ads aggregator....Looks like a classified ad aggregator...in russianhttp://google.ru/ppob Google Classifieds BetaRE: Google PPOB? What is it?Garret, this was discussed here (the 1st result for [google ppob robots.txt] in Google):http://blogoscoped.com/archive/2008-10-08-n14.htmlGet with the program dude! ;-)

Tags: Google Inc., Google PPOB

Discussion threads 2008-10-09

Google PPOB? What is it?

When you are looking for things that Google is about to do, the best place to keep your eye on is Google's robots.txt file -- a document they keep updated with a list of websites they don't want search engines to index. A week ago, they added "/search2001" which...

Tags: Google Inc., Internet, Web Site Development, Search, Web Technology, Garett Rogers

Blog posts 2008-10-09

Google to go back in time?

Warning, this post is speculative, but I think all signs point to what I am about to describe. Google may be about to launch a search tool that uses the same algorithms used way back in 2001. Why would they do this? I think it's probably the...

Tags: Google Inc., Search Engine, Entry, Search, Robots, Emerging Technologies, Garett Rogers

Blog posts 2008-09-30

Is Cuil killing the Internet?

Is Cuil killing the Internet?Brute force indexing?I have an inkling Cuil might not be everything it's touted to be. The only reason for brute force indexing is to fill in the gaps where you've come up short in your harvesting/indexing algorithms.RE: Is Cuil killing the Internet?I used Cuil to...

Tags: Cuil, Internet

Discussion threads 2008-09-02

Linking The Web 2 (Windows)

Linking The Web is a free utility that regularly monitors all backlinks to your sites. If some of the links don't exist, contain wrong anchor, are covered by a "nofollow" flag or hidden by robots.txt then you will be immediately notified by email. Also the program can keep track of...

Tags: Web, Link, S5 Development, Microsoft Windows, Channel Management, E-mail, Operating Systems, Software, Marketing, Online Communications

Software downloads 2008-07-14

New Google Mobile feature lets you search without typing

Google added a new entry into their robots.txt file -- one way you can keep tabs on what Google doesn't want to see. The new entry that forbids crawlers from seeing http://www.google.com/m/lcb made me naturally curious. The new feature, that from what I can tell is...

Tags: Google Inc., Mobile, Feature, Advertising & Promotion, Marketing, Garett Rogers

Blog posts 2008-03-22

SpidEye 1 (Windows)

SpidEye is like a spider's eye, showing what a Web Crawler would see while navigating the Web. From each page visited it collects and summaries relevant information that are meant to assist Web masters on the design of their Web pages for bots. This freeware provides full HTML navigation, displays...

Tags: Web, Microsoft Windows, Software Atelier, SpidEye, Channel Management, Marketing

Software downloads 2007-12-20

Google will soon support "Unavailable After" META tag

Barry Schwartz found an interesting article from highrankings.com that gives us some juicy details about what it takes to get a good rank on Google and what tools we can look for soon. Dan Crow, director of crawl systems at Google, attended the annual SEMNE (Search Engine Marketing New...

Tags: Google Webmasters

Blog posts 2007-07-14