Latest updates from Cleonix Technologies
Everything you need to know about robots.txt

A web surface is an open place. Almost all the websites on the surface can be accessed by several search engines e.g. if we search something in Google, a vast number of results can be obtained from it. But, what if the web designers create something on their website and don’t want Google or other search engines to access it? This is where we need Robots.txt. Robots.txt is a text file which you put on your site to tell search robots which pages you would like them not to visit. It is not HTML. Robots.txt is not so compulsory for search engines but generally, search engines fulfill what they are asked not to do.

Need of robots.txt file
The most important reason for this is to keep the entire sections of a website private so that no robots can access it. It also helps to forbid search engines from indexing certain files. Moreover, it also specifies the location of the sitemap.

Structure of a Robots.txt File
The structure of a robots.txt is very simple – it is a continuous list of user agents and disallowed files and directories. Basically, the syntax is as follows:



“User-agent” are search engines crawlers. Disallow section lists the files and directories to be excluded from indexing. In addition to “User-agent:” and “disallow:” entries, you can add comment lines – just put the # sign at the beginning of the line:

# All user agents are prevented to see the /temp directory.

User-agent: *

Disallow: /temp/

Reason to use the robots.txt file
It blocks the contents from search engines.
It tunes access to the site from reputable robots.
It is used in currently developing the website, which need not show in search engines.
It is used to make contents available to specific search engines.

About the author


Leave a Reply

Your email address will not be published. Required fields are marked *

Latest Post
digital marketing

Basic overview of digital marketing everyone needs to know

Posted on 15 Jan 2021

Top trending java frameworks for 2021

Posted on 11 Jan 2021

web design

Find the best web designing company in Kolkata to build your own website

Posted on 15 Dec 2020

app development Appreciation from UNICEF basics of digital marketing Bengali New Year best web design company in saltlake best web designing company in kolkata Bhai dooj blogging blogging tips blogging tutorial Bootstrap cleonix technologies Cross-Browser Compatibility Css3 Dhanteras digital marketing digital marketing tutorial Diwali Durga Puja E-Commerce Eid Mubarak favicon Gandhi Jayanti Ganesh Chaturthi GIF Graphics design Happy Holi Happy Teacher's Day HTML5 Independece day information technology iOS iOS app development iOS benefits IT blogs Janmasthami java framework java frameworks 2021 java learning java tutorial JPEG learn blogging Logo Design Magento magento 2 magento with google shopping meta tags Mother's Day Nationbuilder off page seo off-page seo techniques online marketing osCommerce PHP PNG ppc Pray for Chennai professional web design Raksha Bandhan Responsive Website Design SEO SEO tips in 2020 software tools Subho Noboborsho SVG UI UX virtual assistant virtual assistant monitoring web design web design in kolkata Web Development Website Design Website Development Woo Commerce Wordpress