Quality of Small Area Statistics

We are excited to start publishing topic data from the census in May as set out on our release schedule. The detailed census population data is vital for community planning right down to neighbourhood level. The topic data will provide amazing insights into the characteristics of Scotland's population.

I’m Esta Clark, Head of Statistical Design for Scotland’s Census 2022 and my focus in this blog is on data quality and why you can be confident in using the census outputs.

The statistics we publish from the census represent estimates of the whole population rather than just a count of who completed the questionnaire. In this blog we’ll discuss how this was fundamental to our design, both for collection and subsequent processing of the data for census outputs.

As part of the releases starting in May we will be making available small area population estimates for Output Areas (OA) and estimates for topics such as religion, language and ethnicity for small areas where they are not disclosive. We will also publish more information about the quality of the statistics so users remain confident in their use of small area census statistics.

We are mindful of the need to have consistently high quality statistics across Scotland. In answer to the lower than expected response rate we undertook a range of interventions, including focusing the efforts of census staff knocking on doors and helping people to complete their forms in areas where response rates were lower than in surrounding areas.  We also in our coverage survey (more later) intentionally sampled more households in harder to count areas and made use of administrative data (data collected about services) in our methods.

In this blog we’ll explore response rates more closely to understand whether there is evidence to suggest that variation (differences) could mean wide variation in the quality of the estimates. This was explored in detail with our International Steering Group (ISG) prior to publication.

Response rates are calculated by dividing the count of sufficiently complete responses by the number we estimated should have responded.

Scotland’s Census 2022 usual resident person response rate was 89.8%, this means we estimated 10.2% of the population so that the results cover 100% of the population. 31 out of 32 councils had a person response rate of 85% or more and all councils were above 82%. Response rates are lower for men (89.1%) than women (90.4%) and lower for younger adults, similar to patterns seen in previous censuses.

Further analysis shows that response rates by small area and various population characteristics, whilst overall are lower, are similar to patterns in the previous census in 2011. Importantly, 97.5% of 2011 Datazone areas (of which there are around 7,000) have a response rate above 72% (75% when only considering the household population and excluding communal establishments). This compares with 2011, and to illustrate the similarity, the pattern of response rates for people living in households are compared between 2011 and 2022 censuses as shown in Figure 1.

For the smallest area census outputs are published – Output Areas (OAs) of which there are 46,373 –97.5% of OAs had a person response rate above 68%, or 66% when including Communal Establishments (CEs) like student halls and care homes. Crucially these areas with lower response were not clustered together.

The response patterns also align with what we expected to see. For example, differences between known harder to count areas, deprivation, urban versus rural areas, ethnicity and activity last week. For example, Figures 2 and 3 show response rates for 2022 and 2011 Censuses by ethnicity and activity last week. Again as we expected to see these patterns, we sampled more households from areas where we expected to have a low response rate in our design.

These response rates show the collection strategy worked well to follow up non-responding households to ensure there is not extremely low response in certain areas or populations. This was really important as the estimation methods to get to 100% coverage of the population work better when the range of responses across areas is not wide. Administrative data was used to estimate the final numbers of students.

Statistical modelling has been used to produce total population estimates across the United Kingdom since the 2001 censuses. We made innovative changes to our approach in response to the return rate during the collection phase of Scotland’s Census. We conducted a Census Coverage Survey (CCS) immediately after the 2022 Census collection as we always intended. The results of this survey were used in our modelling. The key change we made was to supplement the CCS with administrative data to reduce uncertainty in our estimates of the total population. In addition, our estimation methods were different for Communal Establishments (CEs), such as student halls of residence and care homes, as we used administrative data directly for age and sex rather than relying on a coverage survey.

Therefore given these response patterns and also the new methods used to get to 100% coverage of the population, you can be confident in using the 2022 Census data.

More information about the quality of the census results is available in our quality report which was published alongside our first release in September 2023. This report includes confidence intervals for the population estimates, which for 2022 are in line with those achieved for 2011. A confidence interval can be thought of as a range of values around an estimate that is highly likely to cover the true value.

In addition:

• The Office for Statistics Regulation has designated census statistics as Accredited Official Statistics meaning they meet the requirements of the Code of Practice for Official Statistics.
• An International Steering Group (ISG) was established in response to the challenges faced during the collection phase and NRS has been working with these experts as we've been adapting our methods. At the time of publishing the first outputs Professor James Brown, chair said: ‘The International Steering Group commends NRS for its systematic and robust application of the census estimation methodology that has led to today’s published estimates’.

The census provides more than a billion statistics shining a light on the population of Scotland and we are excited to share insights from the topic results with you over the summer.

As part of our quality assurance we have worked closely with Scottish Government analytical topic experts who helped us to quality assure the topic data and its coherence with other existing data sources.

In addition, alongside each topic release we will publish the imputation rates for questions and a report per topic on any quality points to be aware of when using the census data. For example, when comparing to the previous census in 2011, comparing with the 2021 census data for England and Wales and Northern Ireland or with other relevant sources.

We will also continue to publish our methodology papers on our website.

Our blog gives us an opportunity to go into a little more detail on aspects of the census. Over the course of the year, we’ll be continuing to use a range of ways to explain the statistics that are coming out and the stories they tell us about Scotland’s population. The census newsletter is another channel we have for census news. If you are not already subscribed, please do so by emailing [email protected].