Google Now Uses RSS/Atom Feeds to Discover New URLs

Google Webmaster Central has blogged that they are now using RSS/Atom to discover new URLs or webpages.  In short Google is now indexing and crawling not only contents found on websites and blogs but contents that are syndicated from them through RSS/Atom feeds. Whereas before, Google relies mainly on the links provided on website and blog contents, now your site can be found by Google through RSS/Atom feeds that are published on online RSS Feed reader, such as Google Reader.

To those who still don’t know what an RSS/Atom is.

RSS (most commonly translated as “Really Simple Syndication” but sometimes “Rich Site Summary“) is a family of web feed formats used to publish frequently updated works—such as blog entries, news headlines, audio, and video—in a standardized format. An RSS document (which is called a “feed”, “web feed”,or “channel”) includes full or summarized text, plus metadata such as publishing dates and authorship.

Web feeds benefit publishers by letting them syndicate content automatically. They benefit readers who want to subscribe to timely updates from favored websites or to aggregate feeds from many sites into one place. RSS feeds can be read using software called an “RSS reader”, “feed reader”, or “aggregator”, which can be web-based, desktop-based, or mobile-device-based. A standardized XML file format allows the information to be published once and viewed by many different programs.

The user subscribes to a feed by entering into the reader the feed’s URI or by clicking an RSS icon in a web browser that initiates the subscription process. The RSS reader checks the user’s subscribed feeds regularly for new work, downloads any updates that it finds, and provides a user interface to monitor and read the feeds.

So in order for Google to discover your webpages through  RSS/Atom, it is important that you allow crawling of robots.txt

To find out if Googlebot can crawl your feeds and find your pages as fast as possible, check your Webmaster Tools account.  The Test robots.txt tool will show you if your robots.txt file is blocking Googlebot from a file or directory on your site. The toolbox can be find on Crawler Access under Site Configuration.

Google Caffeine’s Pros and Cons

Google Caffeine is the new version of Google search. Google officially calls this the “next generation architecture for Google’s web search”.

Google caffeine as what Matt Cutts says, is a rewriting of their indexing system to be able to crawl, index and serve results faster.

“For the last several months, a large team of Googlers has been working on a secret project: a next-generation architecture for Google’s web search. It’s the first step in a process that will let us push the envelope on size, indexing speed, accuracy, comprehensiveness and other dimensions. The new infrastructure sits “under the hood” of Google’s search engine, which means that most users won’t notice a difference in search results. But web developers and power searchers might notice a few differences, so we’re opening up a web developer preview to collect feedback.”

Read more

New tool from Google alarms sites