Browser testing

When to test

Browser testing should occur as part an agency’s normal development and/or quality control process when:

• a new website or web application is being developed;
• new content is added to an existing website that may introduce compatibility issues. Some types of content are more likely to introduce issues than others. Which types of content to test will be left to Agency discretion;
• the presentation or behaviour layers are changed or updated; or
• a complaint is received that a particular feature, page or specific functionality is not operable in a particular browser. This testing is aimed at replicating the issue and ascertaining the extent of the problem (does this problem occur in any other browser on the full support or reduced support lists?)

Which pages/screens to test

Websites

Websites that use templates for the main layout elements reduce the likelihood of inconsistencies between pages. If templates are being used, agencies must test at least a representative sample of pages, but more thorough testing may also be performed.

Candidate pages for testing should be chosen on the basis that they contain different types of content or functionality.

For websites that do not use templates for the main layout elements, all pages should be tested.

Web applications

For web applications and web forms, typically all screens and functionality should be tested with all browsers on the full support or reduced support lists.

If screens share identical functionality (differ only by content), and the identical functionality is achieved through the use of re-usable components, templates or library code, then testing of a representative sample page is sufficient.

How to test

The following sections outline how to perform manual browser testing by providing questions the tester should consider while reviewing candidate pages.
If the answer to any question is 'no', a different approach or a workaround may be required to improve compatibility with the tested browser.

Manual testing in each browser may be complimented with automated testing ³.

Note:

• It should be noted that web pages cannot always be made to look exactly the same as reference designs (mock-ups), or identical across all browsers. Striving for 'pixel perfection' is not practical and will likely introduce many non-standard workarounds while 'pushing pixels'.
• The word 'like' has been used below to indicate that presentation should appear very similar – and functionality behave similarly – between browsers without needing to be identical.

Testing environment

Ideally, a testing environment should contain each combination of browser with the operating systems on which it is available ⁴.

It is acknowledged that exhaustive testing of each operating system and browser combination is not always practical and does not maximise return on time invested.

Often, modern browsers will behave identically on each operating system for which they are available. So where all browsers on the full support or reduced support list are available for a single operating system, testing on this platform alone can be considered sufficient.

However, where those browsers are also available for other operating systems ⁵, those combinations of browser and platform must also be supported if incompatibilities are reported or otherwise identified.

For example: If the standard ICT setup of your agency primarily uses a Microsoft Windows operating system, then testing under Windows can be considered sufficient if all full and reduced support browser versions are available for Windows. But if an identified incompatibility with a full or reduced support browser cannot be replicated and then fixed under Windows, it may be necessary to gain access to operating systems and/or devices that are not a part of the standard ICT setup for your agency.

General testing

These tests relate to WCAG checkpoints that are required under Information Standard 26: Internet. For more explanation of the significance of the general tests, see the best-practice development techniques.

For each candidate page, consider:

• Does the code validate?
  (Valid code is more likely to be interpreted as intended across different browsers and platforms.)
• Do all linked assets conform to standard and open formats?

Open each candidate page in your main development browser with the behaviour and/or presentation layers disabled, consider:

• Is all content still visible?
• Is the document structure easy to understand?
• Is all functionality operable?
• Do all links and buttons function as specified/expected?
• If alternative screens or content sections are exposed when the behaviour layer is disabled, consider the above questions in relation to this alternative screen or section.

Testing browsers from the full support list

Open each candidate page in each full support browser, consider:

• Does the rendered page look like the same page rendered in the browser used during development?
• Is all content still visible and perceivable?
  (Some presentation and behaviour layer implementations may reduce the perceivability of content even if it is still visible to users with good eyesight.)
• Is all content still visible at small and large screen resolutions/window sizes?
• Is all content still visible and legible when the text is resized (up to two size increases or two decreases)?
• Is all content still visible and perceivable when printed?
• Is all functionality operable?
• Do all links and buttons function as specified/expected?
• Do all dynamic behaviours perform like they perform in the browser used during development?

Testing browsers from the reduced support list

Presentation and behaviour layers should be suppressed for all browsers on the reduced support list.

Open each candidate page in each reduced support browser, consider:

• Has the presentation layer (or specific parts of it) been successfully suppressed where it would otherwise reveal serious bugs in the browser?
• Has the behaviour layer (or specific parts of it) been successfully suppressed where it would otherwise reveal serious bugs in the browser?
• Is all content still visible and perceivable?
• Is all functionality operable?
• Do all links and buttons function as specified/expected?

Testing other browsers

All browsers that are not listed under full support or reduced support will receive unspecified support. Agencies are not required to test in these browsers.

See the unspecified support section for more information.

Footnotes:

³ for example: Selenium can be used to perform automated functional testing on websites and web applications.

⁴ To reduce the number of physical machines required for a comprehensive testing environment, virtualised operating systems should be considered.

⁵ …and officially supported by the browser vendor. In other words unofficial ports and builds receive unspecified support only and do not need to be tested.