As of 1 July 2019, indexing prioritising mobile content is activated by default for all new websites (new to the Web or previously unknown to Google Search): this is just one of several points on the timeline of Google’s mobile-first index, which is precisely the method by which the search engine launches Googlebot to discover new online pages, first announced in 2016 and then officially released at the end of March 2018. Understanding why Google has decided to prioritise web scanning through the smartphone user agent is not difficult, given that most of the world’s users now access the Net through a smartphone, but perhaps not everyone knows what this ‘revolution’ means for sites: let’s try to shed some light on all the aspects linked to Google’s mobile-first indexing, with a particular look at the official optimisation tips for our pages and content and the mistakes to avoid.
What is Google’s mobile-first index?
Today, in 2022, mobile-first indexing is the default system by which Google discovers and indexes new sites, as well as the one enabled for most of the sites currently being crawled; in particular, according to the latest estimates made official by the company, more than 70 per cent of the web is indexed in this way, but the complete switch-off has been postponed several times, and is currently without a ‘final’ date, as we said in the analysis of the end of 2021.
This means, therefore, that it is more likely that the SERPs we see are based on how Google crawled and indexed content based on the mobile version of that page, rather than the old desktop system, which was based on the version of the page designed and developed for users from desktop connections, precisely.
The novelty of Google’s mobile first index is precisely this: the search engine primarily launches Googlebot in smartphone agent mode to evaluate the mobile version of content for indexing and ranking purposes, because most users access Google Search via mobile device.
How the mobile indexing works for sites
There is one overriding aspect to be made quite clear: there is no separate index for mobile devices and desktop sites, and Google Search continues to use only one index. Specifically, Google’s system continues to show the URL that is most appropriate for users, be it a desktop or mobile URL.
As mentioned, to date, not all websites have yet been enabled for mobile-first indexing, nor is there any way of knowing exactly when a website will be moved or anything we can do to make the process faster. Moreover, just as we cannot act to force a website into this index, neither can we request its subsequent exclusion.
To find out if a site is ready for mobile-first indexing (or if it has switched to the new system) we can check the Search Console and in particular the URL Inspection Tool, which allows us to examine each individual URL on the site and find out how it has been crawled and indexed (but usually it is sufficient to check the home page). This only applies to sites older than July 2019, for which Google continues to monitor and evaluate pages for their readiness for mobile first index and sends a notification via GSC when they are deemed ready to switch.
As we said before, this rule does not apply to new sites – meaning all sites published online or recognised for the first time by Googlebot after 1 July 2019 – that have now been included in the mobile index by default for three years, and in these cases, webmasters do not receive notification via Search Console of their inclusion.
What is the mobile first indexing for
On the occasion of the official release of this indexing system, in March 2018, Google Software Engineer Fan Zhang had described the innovative features of the mobile first index, which, ‘after a year and a half of careful experimentation and testing’, was just about to go live with the migration of the first sites that followed the relevant best practices.
In the same post, the novelty was further clarified: until then, Google’s crawling, indexing, and ranking systems ‘had generally used the desktop version of a page’, which, however, ‘could cause problems for mobile users when that version differs greatly from the mobile version’. A new mobile-first indexing was therefore necessary, based precisely on the ‘mobile version of the page for indexing and ranking’, with the aim of ‘better helping our users find what they are looking for’.
Another aspect that is immediately emphasised is that this new index ‘is not separate from our main index‘: that is, Google will continue to have one and the same index to offer search results (no ‘double Google index’, as was speculated some time ago), and if in recent years the desktop version has been used for ranking, from now on and ‘more and more often we will use mobile versions of content’.
The prioritised indexing of mobile content only allows Googlebot to scan and index pages with the smartphone agent first, and there are no alleged ranking advantages either, since this is just the way Google scans and indexes content.
For users, there is no difference whatsoever, because they will continue to display in the search results <strong>the most appropriate URL</strong> according to Google, whether it is a desktop or mobile URL.
What changes with mobile indexing
According to Google, the switch can have a number of effects: for instance, site owners will see a significantly faster crawl speed from the Googlebot smartphone, and the search engine will show the mobile version of pages in search results and in Google’s cached pages. Another important and official note, for sites that ‘have AMP and non-AMP pages Google will prefer to index the mobile version of the non-AMP page’ (a reference that has since disappeared from Mountain View’s later communications, reflecting the progressive debasement of AMP).
A transformation of this magnitude obviously risks having ‘psychological’ consequences for SEOs, but it is again Fan Zhang who urges calm: “Sites that are not in this first wave need not panic,” he writes, specifying that mobile-first indexing concerns the way in which “we collect content, not the way in which content is classified. In other words, content picked up by mobile-first indexing ‘has no ranking advantage over mobile content that has not yet been picked up in this way or desktop content’; at the same time, if a site has ‘only desktop content, it will continue to be listed in our index’, Google states.
At the same time, however, the Mountain View developers encouraged “webmasters to make their content optimised for mobile devices”, because Google would evaluate “all content in our index, regardless of whether it is desktop or mobile, to determine how mobile-friendly it is”, pointing out that since 2015 this measure can help mobile-optimised content perform better for those searching on mobile devices, and that the Speed Update, also in 2018, made speed a ranking factor, leading to a potential downgrade for “slow-loading content could be less effective for both desktop and mobile users.
To summarise, the changes made in 2018 are based on the assessment of a number of factors and trends worldwide, starting with the spread of browsing from mobile devices, and take the following form:
- The roll out phase of the mobile indexing system has started: according to Google’s premises, “being indexed in this way has no rankings advantage operates independently of our mobile-optimised rating”.
- Having content optimised for mobile devices is still useful for those looking for ways to perform better in search results on mobile devices.
- Offering fast-loading content is still useful for those seeking better performance for mobile and desktop users.
- In the case of sites that have both AMP and non-AMP pages, Google will index the mobile version of the non-AMP page.
- As always, Google’s ranking uses many factors: Mountain View developers anticipate that they ‘may’ show content to users that is not optimised for mobile devices or that loads slowly, if ‘our many other signals determine that this is the most relevant content to show’.
The history of mobile-first indexing
The most navigated and attentive among you will remember the various stages in the evolution of mobile-first indexing, which has been part of Mountain View’s discourse and SEO interests (at least superficially) for six years now.
It was in fact 2016 when Google first started tests on ‘mobile-first indexing’, while in March 2018 it made official the first transfers of sites to the new index. This then became THE system adopted by Googlebot on a priority basis as of 1 July 2019 for all new websites, meaning by this adjective, as specified, both those that had just come online and those that had just been discovered and emerged in Google Search.
On this occasion, the American company had also ventured to envisage a date for the definitive completion of the switch to the Googlebot smartphone indexing system: the deadline had been set for September 2020 but, due to the pandemic, it had been moved a first time to March 2021 and then finally ‘cancelled’ without any further information on the timing of this initiative, which ‘will be done when it is done’.
Complicating the operation, as admitted by Google, is that there are still many sites that are not ready for the switch due to unforeseen challenges they are facing, which has therefore made it impossible for the time being to enable mobile-first indexing for all sites in the search system.
Mobile First indexing, Google’s tips to avoid mistakes
And thus, Google too has encountered global problems with mobile indexing, which has proven to be a rather hostile topic for the search engine due (above all) to the backwardness and poor mobile-friendliness of so many sites.
Yet, still in 2018, John Mueller was already offering the first official advice to SEOs and site managers to avoid mistakes with the new mobile indexing, also announcing that such a system was already used for “more than half of the pages shown in search results globally”.
In more detail, the Search Advocate provided some insight into how Google’s crawling works, which moves “sites to mobile-first indexing when our tests assure us they are ready,” immediately notifying the site owner via a message in Search Console of this switch. It is possible to confirm the switch by checking the server’s log files, since most requests should come from Googlebot Smartphone, or even simpler, the aforementioned URL inspection tool of GSC.
Basically, however, a site is ready to switch to mobile-first index if it ‘uses responsive design techniques’, whereas for other sites, some frequent problems may occur.
Structured data and alt texts cause problems for mobile indexing
The first limitation is the absence of structured data on mobile pages, which, as we should know, are useful for the search engine to better understand page content and to highlight your pages in search results ‘in imaginative ways’, to use Mueller’s ironic words. Therefore, those who use structured data in the desktop versions of their pages should be careful to include the same information in the mobile versions of their pages: this is important because with mobile indexing, Google will only use the mobile version of the page for indexing, and thus, structured data will otherwise be missing.
To overcome problems in testing pages, Google suggests checking the structured data in general and then comparing it with the mobile version of the page; for the mobile side, it is also a good idea to check the source code when simulating viewing from a mobile device, or to check the correctness of the format with Google’s rich results test. It should also be remembered that ‘a page does not need to be mobile friendly (so to speak, optimised for mobile devices) in order to be crawled through the mobile first index’, as Mueller goes on to say.
The second problem highlighted by Big G’s post concerns the absence of alt-text for images on mobile pages: the value of image alt attributes is a great way to describe images to users with screen-readers, which, Mueller reminds us, are also used on mobile devices. Moreover, they serve for identification by search engine crawlers, because without alt-texts it is much more difficult for Google to understand the context of the images that are used on a site. The advice, therefore, is to check the ‘img’ tags in the source code of the mobile version for representative pages of one’s own websites, either by using a browser to simulate a mobile device or by using the Mobile-Friendly test to check the rendered version of Googlebot: the important thing is that the page provides appropriate alt attributes that one would eventually want to find in Google Images.
Google’s recommendations for mobile sites
A few months later, John Mueller himself returned to the subject with new indications for those who want to optimise the pages of a site to make them mobile-first indexation-proof, giving as a general directive ‘make sites mobile-friendly and scannable by mobile user-agents’.
With the Net inevitably becoming mobile-friendly, as browsing from smartphones is the most common in much of the world, webmasters, developers and online professionals must understand this change and make their sites more crawlable and indexable for mobile user-agents (the original expression is ‘crawlable and indexable’).
In addition, the article also explains the method by which Googlebot assesses the readiness of a site, which takes into account the parity of content (including text, images, videos and links), structured data and other meta-data (e.g. titles, descriptions or robots meta tags). These are technical factors, therefore, which are independent of other elements such as proportions of desktop and mobile traffic, as explained by Martin Splitt in response to a user on Twitter.
Specifically, then, Mueller reiterates the suggestion to use responsive web design to make websites truly mobile friendly, even though Google continues to support systems such as dynamic serving and separate mobile URLs (also known as m.dot) for mobile websites.
According to experience over the years, in fact, the use of different URLs according to devices can generate problems and confusion for both users and search engines, and therefore it would be more appropriate to use a single URL for desktop and mobile websites.
Guidelines for sites with separate URLs
For sites that continue to have separate URLs, Google specifically points out three best practices to observe to avoid crawling errors:
- Have the same content for mobile and desktop sites (including resources such as videos and images with relevant ALT attributes).
- Present structured data for both versions of the site
- Have metadata (titles and meta description) the same in both site versions.
In addition to these general tips, there are also more specific tips:
- Check both versions of the site in Google Search Console.
- Test hreflang links in separate URLs.
- Ensure sufficient server capacity to handle a possible increase in Googlebot scan frequency.
- Check the txt directives in both versions.
- Test the correct setting for rel=canonical and rel=alternate link elements between the mobile and desktop versions.
Mobile first indexing, best practices suggested by Google
Much more specific is the page with Google’s official mobile-first advice, which is constantly updated with new information to provide precise indications to those who want to optimise their site and make it switch-ready.
The first indication is of a general nature: ‘Websites should offer the same experience on mobile and desktop devices’, which in practical terms means
- Ensure that Googlebot can access and render the content and resources of the mobile page and the desktop page.
- Ensure that the mobile site contains the same content as the desktop site..
- Use of the same robots meta tags in both versions.
- Use of the same headings in both versions.
- Use of the same structured data in both versions.
Google warns that intentionally choosing to offer less content on the mobile version of a page than on the desktop version may expose the site to a drop in traffic “when mobile content priority indexing is enabled on the site,” because “Google can’t get the same amount of information from your page as before” when using the desktop version.
How to optimise the site for mobile indexing
Basically, Google invites us to think about what matters, which is to always try to ensure the best experience for our users, following the recommendations and guidelines provided. In addition to these already described, there are other best practices we can apply to our sites
- Ensuring that Google can see lazy-loaded content.
- Use the same metadata on mobile and desktop sites.
- Ensure that ads do not cause a bad experience for the mobile user.
- Use of a supported format for images and videos.
- Provide high quality images on the mobile site.
- Use the same alt-text on the mobile and desktop site.
- Avoid video and image URLs that change every time the page is loaded on the mobile site.
- Ensure that videos are easy to find and view on the mobile site.
How to handle sites with distinct URLs
The Google document devotes an entire section to the optimisation of sites using the aforementioned distinct URL mode (also known as m-dot), providing additional, more specific advice. In these cases, it is reiterated that one must:
- Ensure that the state of the error page is the same on mobile and desktop sites.
- Avoid URL fragments in the mobile site.
- Ensure that desktop pages have equivalent mobile pages.
- Check mobile and desktop sites in Search Console.
- Check hreflang links on separate URLs.
- Ensure that the mobile site can handle a higher crawl rate.
- Ensure robots.txt directives work on both sites.
Mobile usability does not mean Mobile-First Indexing
If Google can crawl the text on our website, and if that text can be viewed on a mobile device, the website can be added to the mobile-first index. Furthermore, even a site that fails the mobile usability test can be moved to the mobile-first index; conversely, even if the Search Console reports that the site has valid URLs for mobile usability, it does not mean that those pages are ready for mobile-first indexing.
In short, it must be understood that mobile usability is a ‘completely separate‘ element from mobile-first indexing, and therefore even pages that are not optimised or considered not usable on a mobile device can be enabled for this type of indexing.
Consequently, layouts that are optimised to be mobile-friendly or mobile-responsive are not mandatory for mobile-first indexing, since pages without mobile versions also work on mobile devices and are considered eligible for indexing.
How to prepare your site for the mobile-first index
Even today, in 2022, the work of Googlebot mobile is not yet complete and there are still many sites that are not yet ready for this switch, so yet another in-depth article was necessary, published on the company’s official blog to offer further hints, tips and advice on how to better prepare sites for mobile-first indexing for all.
The information (and subsequent images) comes from the work of Yingxi Wu, Google’s Mobile-First Indexing team, who begins by mentioning the various types of problems that have emerged over the years of Google’s testing and evaluation of mobile indexing, which have in fact led to the realisation that many sites are not yet ready.
The crux of the matter is simple (but evidently difficult to realise): in mobile-first indexing, Google only obtains information from the mobile version of a site, so it is essential to ensure that Googlebot can see the entire content and all resources in this version, paying particular attention to certain elements.
Meta tag robots in the mobile version
Yingxi Wu’s first advice is to use the robots meta tags on the mobile version than on the desktop version. If we use a different one on the mobile version (such as noindex or nofollow), Google may fail to index or follow links on the page when the site is enabled for mobile-first indexing.
Lazy-loading in the mobile version
The lazy-loading system is more used on mobile devices than on desktops, especially for loading images and videos: Googler recommends to refer to best practices (also for image optimisation), and in particular to automatically load content according to its visibility in the viewport and, above all, to avoid lazy loading of primary content based on user interactions (such as scrolling, clicking or typing), because Googlebot will not activate them.
An example helps us to understand this better: if a page has 10 primary images on the desktop version and the mobile version only has 2, with the other 8 images only being loaded from the server when the user clicks on the ‘+’ button, Googlebot will not click on the button to load these additional 8 images, and therefore Google will not see those images. The result is that these resources will not be indexed or shown Google Images.
Awareness of what we block
Some resources have different URLs on the mobile version than on the desktop version, sometimes even offered on different hosts: if we want Google to crawl these URLs, we need to check that we have not blocked them with a disallow in the robots.txt file.
An example also supports us here: blocking URLs in .css files will prevent Googlebot from correctly displaying our pages, which may damage their ranking in Search. Likewise, blocking URLs of images will make them disappear from Google Images.
Providing the same primary content on mobile and desktop
In principle, therefore, what matters is to ensure that on desktop and mobile there is the same primary content, i.e. ‘the content with which you want to rank or why users come to your site’.
With the new (and not so new anymore…) system, Google only uses the content shown in the mobile version for indexing and positioning in Search: therefore, if our mobile version has less content than the desktop version, it may be necessary to update it and rebalance this situation, or else our site will risk losing some traffic when mobile first indexing is enabled, because Google will no longer be able to get the full information.
Using the same titles and headings
This general advice also applies to more practical aspects.
The first element mentioned in the article is to use the same clear and meaningful headings on both the mobile version and the desktop version: the absence of meaningful headings (the adjective is also repeated by Google) may negatively affect the visibility of our page on the search engine, because Google may not be able to fully understand the page.
For instance, if the desktop page has the following tag for the page header:
<h1>Picture of cute puppies on a blanket</h1>
the mobile version should also use the same tag with the same words, instead of having headings such as:
<h1>Picture</h1> (not clear nor meaningful)
<div>Picture of cute puppies on a blanket</div> (not using a header tag)
Check images and videos
Images and videos play an increasingly decisive role in providing users with a positive and immersive experience: for this reason, Yingxi Wu advises us to check some specific aspects to optimise our site and to respect some best practices on images.
In particular, we should not use images that are too small or with a low resolution in the mobile version, because they might not be selected for inclusion in Google Images or shown positively when indexed.
For instance, if our page has 10 primary images on the desktop version, which are normal and of good quality, it is bad practice to use very small thumbnails in the mobile version in order to adapt them all to the smaller screen size. Such thumbnails could be considered ‘low quality‘ by Google because they are too small and have a low resolution.
The use of less significant alt attributes may also negatively affect the display of images in Google Images.
A good practice is as follows:
<img src=”dogs.jpg” alt=”A picture of a cute puppy on a blanket”> (significant alt text)
while the following wording is not recommended:
<img src=”dogs.jpg” alt> (missing alt text)
<img src=”dogs.jpg” alt=”Foto”> (alt text not meaningful)
Managing different image URLs between desktop and mobile versions
If our site uses different image URLs for desktop and mobile versions, we may experience a temporary loss of traffic from Google Images while the site switches to mobile-first indexing. The reason for this is that image URLs in the mobile version are new to Google’s indexing system and it takes time for them to be properly understood.
To minimise this temporary loss of traffic, we can check the possibility of retaining image URLs used from the desktop.
How to manage videos
If for the desktop version we use the structured VideoObject data to describe videos, it is important that the mobile version also includes the VideoObject, providing equivalent information. Otherwise, Google’s video indexing systems may have troubles obtaining sufficient information, and thus our videos will not be visible in Search.
The structure of the mobile page is also a crucial node, and Googler suggests placing videos and images in an easy-to-find position. Otherwise, we risk negatively affecting the user experience, and Google may not show our media resources prominently in searches.
For example, if we have a video embedded in content in an easy-to-find position on the desktop, we must be careful to replicate this structure in the mobile version as well.
Placing an ad at the top of the page, which takes up a large part of the page, may result in the video being moved off the page and require users to scroll way down to find the video. In this case, the page may not be considered a useful landing page for videos by Google’s algorithms, and the video will not be shown in the search.
Mobile First Indexing, how to solve the main problems
Such a hot topic inevitably also ended up at the centre of an episode of the Webmaster Conference Lightning Talks series, which during the lockdown period replaced Google’s classic live events, with Martin Splitt addressing the possible problems that sites (and SEOs) may encounter on this front and best practices to prepare the site for this switch.
What is the mobile-first indexing, Googlebot’s mobile scan
It understandably starts with a recap on the definition of mobile-first indexing and how Google indexing works.
In summary, says Splitt, Googlebot crawls URLs from the site with desktop and mobile user agents. With the old desktop-first indexing system, the indexing system obtains the page information from the content of the page in the desktop version; then, if the algorithm deems that it contains information of sufficient relevance to the users’ queries, the page will be shown in the search results.
With mobile indexing, the indexing system will look for information on the page in the mobile version instead of the desktop version.
Challenges and possible issues for sites
Having clarified the (basic) difference between desktop-first and mobile-first indexing, the Googler turns to an overview of the possible challenges facing site owners and webmasters with respect to mobile indexing.
There may be errors during crawling, for instance, or problems with the content of the crawled page: let us see in detail what can be done in these cases and how to intervene.
Case 1, crawling problems
A frequent problem concerns possible errors in crawling with Googlebot mobile: our server might treat the request based on the user agent differently, or there might be other problems related to a request to mobile pages.
When these situations occur, Google will be able to extract little (or even no) information from our pages, and thus obviously will not have enough data to decide to show the page in the search results, with the result that it will be excluded from the SERPs.
Case 2, problems with mobile content
Another problematic case is when Google notices that our mobile page has content problems: the bot will get less information from the page, or even the wrong information, and thus will not be able to effectively determine itsb for user queries.
This is especially the case if a site has separate mobile pages, which offer different content from the desktop version.
Identifying the main mobile crawling issues
Both of these situations prevent Google from serving the site well “when it is enabled for mobile-first indexing,” which is “something you probably want to avoid,” the Developer Advocate jokes in the video.
In order not to run into these problems and to solve them, Splitt offers some useful hints and elements to evaluate to make sure there are no issues when crawling mobile pages. In particular, we must NOT:
- Use a Disallow directive in the robots.txt file for mobile pages.
- Use the noindex meta tag on mobile pages.
These two errors do not allow Googlebot to crawl and index mobile pages (as seen in the image).
- Use a disallow directive on mobile CSS files with a rule in the robots.txt.
- Use the nofollow meta tag on internal links.
These two errors do not allow Googlebot to crawl mobile resources.
In general, if we give Googlebot access to resources on the desktop page, we should do the same on the mobile version as well, or else the bot will not be able to fully analyse and understand the mobile page.
In addition to these best practices on robots.txt directives, noindex and nofollow tags, site owners should also check the scanning capacity of their server: ideally, a server should be able to handle the same amount of desktop and mobile scanning.
Identifying mobile page content issues
Setting the mobile page content correctly is just as important as checking that the mobile crawler settings are correct. Google’s general suggestion is to make sure to serve identical primary content in both versions, desktop and mobile, but in some cases this may be different.
Different content between mobile and desktop
For instance, says Splitt, we might have a desktop page that is presented with several images and some text, while in the mobile version we decide to present only two images and very little text: users have to click on the + (plus) button to make the remaining part of the page load.
However, Googlebot will not click on the button, and thus all information not immediately loaded will be invisible to Google. Therefore – also as a general note – the Googler says that if we plan to have less content on the mobile page, we must be aware that Google may not be able to serve our site as well as before mobile-first indexing, precisely because it gets (we give) less information than desktop indexing.
Headings are important for Googlebot to understand our page: using classic tags is a ‘good example of a heading’, whereas it is wrong to use <div> commands.
In this case, regardless of its class, Googlebot will treat this portion of text as if it were normal text and not a title; this will therefore affect Googlebot’s understanding of the page.
The practical advice is therefore to refer to semantic header tags on our mobile pages.
Management of images and videos
“If you care about the traffic generated by images and videos, you need to do a few extra checks on these resources“, Splitt tells us.
First of all, it is important to take care of the creation of the alt text to define an image and not to leave the field empty or insert terms that are too generic, because this practice can cause a bad experience for some users, who would have no way of understanding the topic of the image when they cannot view it. As in the previous case, we cannot put image attributes in a <div> field because Googlebot cannot read and index resources in this way, so only use semantic image tags.
In addition to these classic SEO management tips for images, for mobile pages it is also important to take care of the position of media resources: if a video is in an easy-to-view position in the desktop version, the presence of an ad could displace it on the mobile page, forcing users to scroll far down to finally see the content and again generating a bad user experience.
Check the invisible parts of pages
The last aspect to pay attention to is the management of invisible parts of the page – obviously, Splitt is not referring to black hat SEO tactics and hidden content! – which have no effect on the user experience, but can influence the way Googlebot understands the content present and the topics covered.
An example of an invisible part of the page is structured data: best practice is to maintain the same markups on desktop and mobile pages.
Another example are meta descriptions, and Google urges us not to forget to add this information to our mobile pages, because meta descriptions ‘are also very important for Googlebot’.