The 2019-2020 working conditions survey is comprised of project design, project implementation and data processing. The questionnaire and sampling in the project were designed by the working conditions survey team (hereinafter referred to as "the survey team"). The project implementation was done by a Beijing-based survey company under the supervision of the survey team. The final data processing was completed by the survey team. The survey design, implementation and data processing are presented as follows.

Part One: Project Design

I. Questionnaire Design

    The 2019-2020 working conditions survey questionnaire was slightly adjusted from the structure and content of the questionnaire in 2018, with 7 topics deleted and 58 topics added.

    The final questionnaire consists of seven parts. a. Part A is a basic information interview. Questions such as interview places and interview time, are answered and filled in by interviewers. b. Part B is to screen respondents. Interviewers fill in this part after asking specific questions including the number of people working at home, the age ranking of respondents, and whether the selected object is willing to accept the interview. c. Part C describes the basic situation of respondents' works. There are 122 questions in total, involving the job title, working content, working conditions, and working experience of respondents. Part C, as the main part of the questionnaire, is filled in by respondents. d. Part D presents the situation of respondents' workplaces. There are 117 questions in total, involving workplace types, institution settings, system construction, etc. It should be noted that the respondent can skip this part if he or she doesn't have a work unit. e. Part E, income and welfare of respondents. A total of 29 questions concern gross income, income components, income satisfaction and so on. f. Part F contains the interpersonal relationship and social life of respondents. There are 135 questions in total, containing the relationship with leaders and colleagues, as well as the satisfaction in all aspects of life. g. Part R is the basic information of respondents, which has 45 questions, including age, gender, and family assets.

    The response time is designed to be 30 minutes considering a large number of questions in the questionnaire.

II. Sampling design

(1) Target population

    The target population of the Working Conditions Survey of Chinese Urban Residents 2019-2020 is the employed population aged 16 and over living in urban areas of mainland China. The term of “employed urban population” is operationally defined as the employed population aged 16 and over living in communities (residents’ committees) across different county-level cities, prefectural-level cities and direct-administered municipalities in mainland China between October 2019 and November 2020.

(2) Sampling design

    The working conditions survey of 2019-2020 adopts a complex sampling design, including four-level sampling procedure.

    ① The sampling of PSU. The county-level administrative units (districts and county-level cities) were recognized as PSUs based on the data of China's “sixth population census” conducted in 2010, in combination with the latest information on administrative division defined by the Ministry of Civil Affairs of the People’s Republic of China, to establish data of the PSU sampling frame. According to PPS (Probability Proportional to Size, PPS, and the probability sampling of equal scale), 60 districts and county-level cities were extracted.

    ② The sampling of SSU. For each PSU, we applied the PPS method to select 8 communities (residents’ committees) as SSUs. In cases of inaccessibility, relocation or changed administrative division, we would supplement new SSUs that were selected from the sampling frame using the same method.

    ③ The sampling of TSU. For each selected SSU, our survey implementation institution dispatched on-field interviewers to make district maps and the list of household addresses as the third-stage sampling frame, from which we selected 13 households as TSUs using the simple random sampling without replacement (SRSWOR) method. Given the presence of refusal rate and non-target households, we requested interviewers to provide an estimated value of refusal rate for each TSU in sampling and used the following equation:

the number of selected addresses=int(13/(1-refusal rate)+0.5)

    If the number of successfully surveyed households reached 13, the survey was stopped; if otherwise, the interviewers should recalculate the newly selected addresses based on the number of remaining households to be surveyed and new refusal rate using the above equation.

    ④ The sampling of USU. With respect to selected TSUs (households), we further selected one individual per address as the ultimate sampling unit (USU) based on the random number table (see the Questionnaire) that we established. If a selected individual did not accept to be interviewed and did not allow other household members to do it alternatively, then the next address in the selected address list would be visited.

(3) Sample size design

    With simple random sampling of the population (without replacement), we could obtain the estimated sample size using the following equation:

    where p is the probability that a certain class of the sample appears in the population; u_αis the distribution critical value when the confidence level is α; and d is the difference between sample estimate and population parameter. According to the above equation, if we set the confidence level of the estimation interval α=0.05 and the absolute error d=3%, then we only need to survey a sample of about 1,000 for the estimation of the majority of distributions.

    However, given the fact that this survey used a multistage complex sample instead of a simple random sample, we must also take into account of the design effect (deff). The design effect is the ratio of sample variance generated when using the complex sampling to the sample variance generated when using the simple random sampling under the same sample size. The estimation equation of the design effect is as follows:

    Where b is the number of samples selected from a single sampling unit; and roh is the homogeneity within the sampling unit. This equation indicates that the larger the number of samples selected from a single sampling unit, the larger the deff; and the larger the homogeneity within the sampling unit, the larger the deff. This survey utilizes a sampling scheme that increases the number of sampling units and reduces the number of samples within a single sampling unit to the highest extent possible. Therefore, we set the deff of this survey as 5 based on the design scheme utilized in this survey and our prior experience in sampling. Thus, the sample size that takes deff into account would be 1000×5＝5000.

    To obtain an unbiased parameter estimate, a certain level of response rate r must be ensured for a social survey:

r = (the number of survey participants)/(the number of contacted interviewees)

    Methodologically, the target population can be divided into two potential populations by whether a response is given: the population available for survey and the population unavailable for survey. The size of the former can be calculated by response rate * target population size; while the latter by (1-response rate)* target population size. The lower the response rate, the smaller the population size that can be inferred by the sample estimate. Only by assuming that there is no statistically significant difference between inferred parameters for available and unavailable populations can we generalize the survey results to all population members in the presence of nonresponses. The rule of thumb is that we should at least ensure a 50% or higher response rate in sampling surveys (that is, both available and unavailable populations account for half of the target population). Considering the presence of nonresponses in the survey, we need to appropriately increase the size of the selected samples. We assumed that the response rate of the survey was 80%. Thus, considering the existence of nonresponses, the sample size for this survey should be 5000/0.8=6250. Further considering the specific distributions of samples, we determined the final sample size as 6240 (=60*8*13), which is made up of 60 PSUs, with each PSU comprised of 8 SSUs, each SSU comprised of 13 TSUs and each TSU comprised of 1 USU.

III. Survey quality control

    The objective of survey quality control is to reduce the systematic errors (biases) of survey data under the guidance of the “overall research design”. Based on the research design, the survey may entail systematic errors in the following three stages: 1) First, the household selection stage. For example, factors like incomplete plot drawings for sampling, inaccurate entries on sampling forms and arbitrary replacements of household addresses by interviewers could all cause errors. 2) Second, the respondent selection stage. For example, biases in sample sex and age could be caused if interviewers fail to perform in-home selection based on the Kish grid procedures or if the entries of the Kish grid are non-compliant. 3) Third, the field interview stage. For example, biases could occur if interviewers systematically miss questions, intentionally avoid part of question sets by taking advantage of skipping rules, merge questions that should be asked in an “item-by-item” fashion, or direct or suggest respondents providing certain kinds of answers.

    Revolving around the three stages stated above, data quality control was performed through the following procedures in this survey.

(1) The household selection stage

    A sampler was dispatched to the selected residential community for a field visit, where he or she examined all buildings in the area and drew or updated the Residential Land Plot Drawing for Sampling. On that basis, the sampler also filled out a Land Plot Sampling Form where the numbers of floors and entrances in each building, and the number of households for each entrance per floor were recorded. All residential households in the above building constitute the sampling frame for this sampling round. The sampler must ensure that the identifier numbers of the drawings, residential buildings and rooms are consistent. In the event that the number of households indicated by the Land Plot Sampling Form is evidently lower than the size of a regular residential community in the locality, the sampler should promptly check whether the Land Plot Sampling Drawing and the Land Plot Sampling Form are complete.

    Upon receiving the data of the Residential Land Plot Drawing for Sampling and Land Plot Sampling Form, the contractor’s project team should provide a list of addresses to be visited for each community developed in a random selection process. A visit to a household cannot be regarded as failed unless there have been 3 nonresponses or 2 refusals. In case of any failed visits, the interviewer must truthfully specify the causes on the Registration Form of Home Visits. For communities where a specified number of valid sample interviews can still not be realized after 3 home visits, the contractor’s project team should provide the second set of visiting addresses.

    During home visits, an interviewer must carefully fill out the Registration Form of Home Visits and are prohibited from arbitrarily change the visiting addressing. If it is found in the validation process that more than 100 households’ entries are missing in the Land Plot Sampling Form, the questionnaires of the residential community will be regarded as invalid.

(2) The respondent selection stage

    According to survey procedures, the interviewer should select the respondent from household members using the Kish grid provided on the first page of the questionnaire after entering a household. Respondent selection is crucial for ensuring sample randomness and thus should be properly implemented. The survey institution conducted two rounds of reviews within 2 days after the completion of questionnaires to check whether the Kish grid-base sampling was implemented correctly. If any flaws with the sampling process were found, the in-home interviews must be re-performed. Interviewers must record the audio of the entire process of interviews they performed. The proper “respondent selection” procedure must be reflected in the audio. All questionnaires administered by interviewers that were found to be engaged in falsification were regarded as invalid. Questionnaires with the “respondent selection” procedure missing in the audio were also regarded as invalid.

    Meanwhile, the research group conducts statistical monitoring over survey data by gender and age grading. In the later stage of the survey, a quota sampling procedure will be initiated to control the respondent composition, if there is a sex ratio imbalance or age structure deviation.

(3) The home interview stage

    The survey implementing agency provided effective training for interviewers using the Interviewer Manual and relevant video data. With respect to the field interview stage, the survey institution needed to require interviewers to avoid missed questions or misuse of question skipping. Immediate remedies must be implemented if the above mistakes occurred. Interviewers cannot conduct a new interview before the supplementary survey is completed. If an obvious recording fraud is found, all questionnaires for which an interviewer is responsible will be required to restart the door-to-door interview by the survey company.

    A triple audit system is established by the research group according to the interviewer behavior standardization, which examines the survey data repeatedly with the Beijing survey company and local supervisors (as shown in Fig.1). Specifically, the first audit is completed by local supervisors. The second audit is completed by the survey company. The third audit is completed by the research group.

Figure 1 Audit flow chart

    The data quality is mainly controlled in the triple audit, because the survey company and local supervisors may audit carelessly to complete the task as soon as possible. Therefore, the survey team set up a special audit team, including 1 teacher in charge of audit guidance and communication, and 17 auditors consisting of 10 graduate students and 7 undergraduates. The audit team establishes a detailed audit plan to clarify the personnel allocation, audit tasks of each link, and the major concern. To be concrete, auditors are divided into four groups as shown in Fig.2, including the authenticity audit group, the correctness audit group, the skipped and missed questions audit group, and the error and deviation correction audit group. Their audit tasks are as follows. a. Authenticity audit group: It identifies whether the interview is real through a comprehensive judgment made as per the evidence chain, after checking the consistency among photos, time, GPS and interview features. b. Correctness audit group: It confirms whether the interview is conducted correctly by the confirmation of whether the visitor correctly uses the Kish table in the given door-to-door list, whether the sampling box is correct, and whether respondents are randomly sampled. c. Skipped and missed questions audit group: It records and gives feedback in time if there is any skipped or missed problem, by quickly playing the interview process recording combining the average usage time of each questionnaire or the average time of each module. d. Error and deviation correction audit group: It improves the quality and accuracy of data to reduce the entry error rate by correcting deviation and errors such as error entry. Also, the non-standard interview items from interviewers should be recorded in detail.

Figure 2 Flow chart of triple audit

Part Two: Project Implementation

IV. Project Implementation and Project Adjustment in the Context of the COVID-19 Epidemic

    Three stages are included in the actual process of project implementation:

    (1) Initial phase: December 2019 to January 2020

    The survey implementation officially began in December 2020, with its first survey carried out in the Fangshan District of Beijing, Hongshan District of Wuhan and Chang'an District of Shijiazhuang. 58 questionnaires had been received by January 14, 2020. Then, the survey was temporarily suspended owing to the approaching of the Spring Festival. However, the COVID-19 epidemic in China suddenly escalated during this period. The whole country entered a closed management state to avoid the spread of the epidemic. Thus, the survey had entered a state of persistent suspension against this context.

    (2) Restart phase: April 2020 to August 2020

    In April 2020, the survey project was restarted simultaneously with the resumption of work and production in various places, because the COVID-19 epidemic was effectively controlled in China. The survey was carried out in 36 cities (counties and districts) nationwide in this phase. But, the door-to-door survey had become very difficult due to the impact of the epidemic. On the one hand, the community management was more stringent. On the other hand, residents were more vigilant. Hence, the project only collected 1082 questionnaires in this phase for its slow progress.

    (3) Adjustment and acceleration phase: August 2020 to November 2020

    Considering the slow progress of the survey that could not be effectively improved in the short term, the survey team decided to adjust the survey plan to complete the survey, after several rounds of discussion. The enterprise entering survey was carried out against the continuation of the original door-to-door survey. That is, the interviewer investigated multiple workers in the same enterprise after entering the work unit.

    The survey team implemented a detailed enterprise entering survey program. First, work unit selection. The selection of work units adopted a convenient sampling method because entering work units to conduct the survey required management permission. Specifically, the survey team, the Beijing survey company and the local survey company contacted enterprise managers through their own social network or enquiry after entering the employing unit. Then interviewers entered the investigation with permission. the survey team had to audit all work units to be entered. In addition to the feasibility principle, the survey team also considered the types of work units and the industry diversity as much as possible in the work unit selection. According to the final results, 260 employing units were involved in this phase of the survey. Second, respondents selection. The method of quota sampling was adopted to select respondents. To be specific, there were two requirements. a. The ratio of managers and operators should be controlled at 2 to 8. b. Work categories in the work unit should be covered as much as possible. Third, survey method. The enterprise entering survey remained the form of face-to-face interviews, in which interviewers read questions for respondents to answer and then ticked on the electronic device. This method is necessary to ensure data reliability. Fourth, data audit standards. Apart from adhering to the original audit standards, the survey team also put forward new requirements for the enterprise entering survey, including interviewers to provide unit positioning photos, environmental photos, and working position photos. Moreover, it made further provisions for the whole process of recording, including the whole process recording of respondents selecting and answering.

    Finally, 4,283 questionnaires were collected by the enterprise entering survey. In the meantime, 1,939 questionnaires and 3,079 samples were collected from the door-to-door survey that carried out in 43 cities (counties and districts) in China. At last, the survey team obtained 7,362 samples after three stages of the survey.

V. Data Audit

    The survey implements the triple audit system to ensure the quality of survey data. The first audit is completed by local supervisors. The second audit is completed by the survey company. The third audit (final audit) is completed by the research group. According to the audit, the survey team receives 7362 samples, and 6188 are finally qualified with a pass rate of 84.05%. More specifically, 3079 samples of door-to-door interviews are collected, of which 2258 are qualified with a pass rate of 73.33%. 4283 enterprise entering samples are collected, of which 3930 are qualified with a pass rate of 91.76%.

    As for unqualified samples, 680 samples are rated as unqualified in the first and second audits, and 493 samples are rated as unqualified in the third audit. And there are 232 door-to-door samples in the unqualified samples determined by the third audit. Table 1 indicates the distribution of reasons for unqualified door-to-door samples. Specifically, 35.35% of the unqualified samples are caused by failing to meet the condition of authenticity, such as an abnormal interaction process; 9.48% are due to incorrect indoor sampling; 22.41% are because of skipped questions or subjective assumptions caused by incorrect attitude of interviewers. Another 76 samples are rated as unqualified for exceeding the upper limit of one single community with 13 copies.

Part Three: Data Processing and Quality Assessment

VI. Data Cleansing

    The survey team cleaned 7,362 collected samples. First, 1,174 unqualified samples were deleted after the audit. Second, the correctness of door-to-door sampling, and inconsistent samples should be checked. The inconsistent sample (or with indoor sampling error) should be deleted after comparing the required age ranking in the Kish table with the age ranking of respondents. Only 1 sample not meeting the requirements was deleted after checking. Third, 47 community samples were deleted, each of which had less than 3 samples. Fourth, 2,147 qualified door-to-door samples remained after deleting the sample that only investigated one community in a city. Fifth, the enterprise entering samples with less than 5 samples in the same unit were deleted, in order to ensure that enterprise entering samples can accurately reflect the unit situation. Sixth, based on the logical relationship between the questions, answers without logic were processed as missing values after the logical verification of data mainly involved questions to be answered and filled in. For example, the current working time shall be less than or equal to the first working time. The minimum age for the current and first working shall be not less than 15 years for employed persons and not less than 12 years for non-employed persons. The respondent's organization level shall be less than or equal to the highest level of the employing unit. Seventh, new variables generated were arranged, including the generation of occupational prestige variables based on the occupation and work content filled by respondents, and the generation of organizational level variables with the majority principle based on respondents' answers to the unit situation in the same unit.

VII. Weight and Calibration

    As weight is the key to ensuring the correspondence between the samples and the overall situation, weights of 2,147 qualified door-to-door samples were calculated. In general, weight can be divided into sampling weight and calibration weight. The former determined by the sampling plan, is the reciprocal of the sampling probability of each case in survey samples. There are 4 methods for sampling weight calculation, including design-based (randomization), model-based, model-assisted, and Bayesian[1]. The design-based sampling weight calculation is adopted in this survey, to accurately reflect the particularity of this sampling design.At the same time, to reduce the variance and bias of the sample valuation, we also make up for various potential deviations (including composition deviation) caused by the outdated sampling frame, respondent refusal and interviewer falsification, using the statistics published by the National Bureau of Statistics for calibration. By doing so, sampling weight and calibration weight are contained in this survey data. The former is for adjusting the multi-stage unequal probability in the sampling design. The latter prevents various potential deviations between the samples and the overall situation, especially the deviated population composition, by further adjusting the structural weight on the basis of the former. The generation processes of both weights are shown below.

[1]  Refer to Valliant, Richard, Jill A. Dever, & Frauke Kreuter. Practical Tools for Designing and Weighting Survey Samples, New York: Springer, 2018.

(1) sampling design weight

    This survey is designed to use the multistage complex sampling. In the first stage, 60 urban districts were selected using the PPS method; in the second stage, residents’ committees in these urban districts were selected using the PPS method; in the third stage, households in the residential communities were selected using the SRSWOR method; in the fourth stage, 1 respondent was randomly selected using the Kish grid at the respondent's home.

    Thus, when selecting PSUs using the PPS method in the first stage, the sampling probability pi for the i-th PSU is:

    Where m is the sample size of each stage, and n denotes the employed population or its substitution variable in each sampling unit. N.total is the population aged 16 or over living in the residential communities of urban districts, which was calculated using data of the Sixth Population Census. As can be calculated based on the sampling frame, its value is 616432389[1]. M.psu is the number of selected PSUs, that is, 60. Ni.psu is the employed population in the i-th PSU, that is, the sampling probability of the i-th PSU is the sum of the ratios of its employed population to the total employed population added for 60 times.

[1]Here, we assumed that the working population in Chinese cities has a national closure property, neglecting the effect of international migrants (both inbound and outbound). In the meantime, we also neglected the labor migration from rural to urban areas due to urbanization in the past 10 years.

    In the second stage, we selected 9 SSUs from urban PSUs using the PPS method. With a PSU being selected, the sampling probability of the j-th SSU, denoted by Pjssu|i, is:

    Where mj.ssu is the number of SSUs selected from each PSU, which is set as 8. Nj.ssu is the working population in the j-th SSU, ni.psu is the working population in the i-th PSU. That is, the sampling probability of each SSU is the sum of the ratios of its working population to the total population of PSU, added for 8 times.

    In the third stage, we randomly selected household TSUs from community SSUs, but the sampling frame did not contain the number of households in communities or the total number of households with at least one employed member. Therefore, we employed the SRSWOR method. With an SSU being selected, the sampling probability of the k-th household, denoted by pktsu|j, is:

    Where mk.tsu is the number of households that should be selected from each community TUS, which is set as 15; nk.tsu is the total number of households having at least one employed member in the community. In this survey, we utilized a Land Plot Registry Form, requiring interviewers to register all accessible addresses of a community before conducting in-home interviews. The number of these addresses was used as the substitution variable of nk.tsu[1].

[1]This may cause two problems: first, a low coverage rate. Addresses that are difficult to approach due to their remote locations or tight security controls can hardly be included into the Land Plot Registry Form owing to high contact costs. Therefore, the coverage rate of TSU sampling frame may be low. Second, addresses that are not suitable for interview may be covered, causing an inconsistency between the composition of addresses in the TSU sampling frame and the target population.

    In the fourth stage, interviewers visited addresses randomly selected from the Land Plot Registry Form and initiated selection as soon as they contacted household members, that is, the number of household members that are employed and aged 16 and over under the current address. If the number of employed members is not 0, the interviewer would use the Kish grid on the first page of the questionnaire to select appropriate respondent and ask about his or her willingness to participate. If the respondent is not willing to participate in the survey, the household would be considered as a refusal. The interviewer continued to visit the next household on the list of addresses until obtaining the consensus of the respondent to start the interview[1]. With a home address being selected, the sampling probability of each individual case is plusu|k:

[1]Due to reasons like respondent refusals, time constraints over interviewers, task pressure and incentive motives, there might be instances where a respondent was replaced by another household member. However, there is no way to estimate the magnitude of the replacement rate.

    Where  is the number of employed members in the 1st address of the k-th community, which is obtained from item B1 on the questionnaire.

    According to the above steps, the sampling probability of each respondent is pm_case, that is, the product of the above four probabilities, denoted by:

    Where  denotes the constant sampling probability of each individual case. As we used the PPS method in both the 1st and 2nd stages, the resultant sample is a self-weighting sample (equal probability of selection method sampling, epsem) (Kish, 1965). The second term  is the changed sampling probability of each individual case. In the third sampling stage, we did not use the PPS method to select households due to a lack of necessary information. Instead, we utilized the SRSWOR method, causing changes in the sampling probability of each respondent.

    In addition, we used working population aged 16 and over as the population size for the PPS-based sampling. As working population is highly correlated with, but different from, employed population, we used employment rate as a moderating factor. In other words, we assumed in the sampling stage that the employment rate across different sampling units is constant.

    Substituting relevant values into the above equation, we get:

    Therefore, the survey data is generally not a self-weighting sample (equal probability of selection method sampling, epsem), and the sampling probabilities for individual cases are different (Kish, 1965). That is, the average sampling probability of this survey is about 1.3/100,000.

    The sampling weight wgt_case [1] should be the inverse of the aforementioned sampling probability pm_case, that is:

[1] The name of the variable in data files is wt_ind_dsgn_nrsp.

Where  is the number of employed members in the 1st household of the k-th TSU[1], nk.tsu is the total number of households with at least one employed member in the k-th SSU (community)[2], and nj.ssu is the working population of the j-th SSU. On average, each respondent of the survey approximately represents 76,000 employed urban population.

[1] The variable in data files is b1.

[2] We multiplied the total number of households statiticized from the Registration Form of In-home Visits by (1-2.1%), which is the average ratio of the households without suitable respondents. The product is the estimate of household with employed members in the community.

However, due to the outbreak of the epidemic and irregularities in the actual survey, 2174 individual cases instead of 6240 were finally retained in this survey[1]. Based on the actual number of household samples, the value 6240 incorporated into the above equation should be replaced by 2174. That is, the weight wgtrsp[2] after correcting for the refusal effect is:

[1]To ensure that SSUs have a sufficient level of variation, we deleted SSUs having less than 5 households, which involved 35 cases. As such, the valid cases dropped to 6702 from 6737.

[2] The variable in the data file is wt_ind_dsgn_rsp.

    We recommend that this sampling weight be utilized when using the survey data in descriptive analysis or answering empirical questions. Thus, each respondent in this survey approximately represent 284,000 employed urban population.

(2) raking weight

    In PSU and SSU sampling, we utilized estimated data from the Sixth Population Census as the sampling frame of the survey. However, there is an 8-year gap between the survey and the Six Population Census, during which significant changes have taken place in not only the demographic sizes and structures, but also social structures. To correct for the compositional biases between the survey data and the known overall attributes, we used the latest statistics published by NBSC of China and adjusted the weight of the data using the raking method. At the individual level, our target population was 442.47 million employed urban population in 2019. We drew on relevant statistical communiqués published by the NBSC of China to correct for 4 variables including ownership, sex, educational level and age group.

① Ownership calibration

    As shown in Table 2, in the ownership type, a big difference can be found between the survey data and the calibration composition. Specifically, an obvious calibration composition deviation is caused by the types of respondents' units mainly concentrated in individuals and party and government institutions.

    Note: The calibrated data is derived from the official site of the National Bureau of Statistics. The item of “pubic institutions, parties, governments and mass organizations” is not included in the original forms. Data under that item was estimated based on statistics of other items and the total employed population. http://data.stats.gov.cn/easyquery.htm?cn=C01. If the total of the proportions does not come to 100%, an adjustment would be made using the item with the largest proportion.

② Sex calibration

    In the sex composition (as shown in Table 3), more female samples in this survey also lead to the difference in the calibration structure.

    Note: The calibrated data is derived from the China Population & Employment Statistics Yearbook 2018. The sex ratio of the urban employed population is replaced by the sex ratio of the national employed population.

③ Age calibration

    As shown in Table 4, the age structure is slightly smaller than that of the survey data and calibration composition. The proportion of respondents from 16 to 39 years old is greater than that of the calibration composition, while those in the 40-year-old and above is less than that of the calibration composition.

    Note: The data is calibrated based on “Table 3-22: Age composition of employed urban population by occupation and sex” of China Population & Employment Statistics Yearbook 2018. If the total of the proportions does not come to 100%, an adjustment would be made using the item with the largest proportion.

④ Educational level calibration

    As shown in Table 5, the educational level of respondents in this survey is significantly higher, compared with the calibration composition. Most respondents have a high school education or above.

    Note: The data is calibrated based on “Table 3-24 Education level composition of employed urban population by occupation-sex” of China Population & Employment Statistics Yearbook 2018. If the total of the proportions does not come to 100%, an adjustment would be made using the item with the largest proportion.

    Through the operations stated above, we expected to reduce the compositional bias of the survey data in other areas through calibrating these 4 aspects. Next, we will assess the calibrated survey data from the perspective of data quality.

VIII. The Evaluation of Survey Data Quality

    The evaluation of survey data quality is essentially the error magnitude test of the survey data. In general, survey errors are composed of random error and system error. The former is an uncontrollable error caused by random factors. The latter is a controllable error caused by improper operation in the design or implementation process. The random error mainly affects the data reliability, while the system error mainly affects the data authenticity (validity). Reliability and authenticity are two critical indicators for evaluating the survey data quality. A high-quality survey should guarantee that data results can accurately reflect the objective fact, or the authenticity[1], apart from ensuring that similar data results are obtained under multiple repeated measurements, or the reliability. Therefore, the reliability and validity of the survey data need to be tested.

[1]Refer to Fu Deyin and Huang Hengjun: Measurement and evaluation methods of statistical survey quality, China Statistics Press, 2011, pp.41-44.

It should be noted that all types of samples, including door-to-door samples and enterprise entering samples, were tested for the reliability test is unrelated to the sampling procedure. Only the validity of door-to-door samples was tested in our survey using external indicators with a census nature since door-to-door samples were strictly randomly sampled.

(1) Reliability test

    Data reliability refers to the stability degree of measuring results. In demography and sociology, the measurement reliability[1] of continuous variables in the unit is often measured by the Myers mixed index. In this survey, the age and telephone numbers of respondents are investigated as test variables for the survey data reliability. The former is non-sensitive information, while the latter is sensitive information.

    Respondents were required to leave their telephone numbers or mobile phone numbers, with the last number digits distributed below (referring to Table 6):

    As can be seen, there is only one number of last digit - the number “4” - whose proportion is 6.42%, evidently lower than other numbers. The Myers' composite indicator of telephone numbers is 10.08, indicating that only 10.08% of telephone numbers need to shift their digits in order for the last digit numbers to follow a statistically uniform distribution. Surveyed data are highly reliable according to this indicator.

    With respect to the last digit number of ages, as can be seen from Table 7, the numbers of 0 and 5 are relatively high. The estimated Myers' composite indicator is 24.08, indicating that at least 24.08% of the last digits of respondents’ ages need to be adjusted in order for them to follow a statistically uniform distribution. The age tendency of this survey is beyond the standard range to some extent, with the Myers index below 20 as the standard.

    In general, the survey data is in the upper middle quality range in terms of its internal reliability, and thus can basically reflect the general picture of the researched area. However, further improvement in measurement accuracy is needed.

(2) Validity test

    The validity of survey data refers to the degree to which its various estimates are consistent with other authoritative statistical data. Here, we selected the number of members of the Communist Party of China (CPC) as an external reference indicator. The CPC is the largest party in the world and the governing party in China; the number and composition of party members released by its organizational department can be considered a highly reliable external indicator, as the information is basically unlikely to be influenced by economic interest or administrative factors like official performances. More importantly, the age criterion for joining the CPC basically overlaps with that of employment. After removing retired persons and the elderly persons who have never been employed, it basically covers the entire employed adult population.

    After calibration weighting, this survey data shows that the proportion of CPC members among employed urban population is 11.3%, and the 95% confidence interval of that proportion is [8.1%, 15.5%]. According to the Statistics Bulletin of the Communist Party of China: 2019[1], the population of urban employed CPC members excluding those engaged in agriculture, animal husbandry and fishing, students and retired persons is 45.734 million, accounting for 10.34% of the total 442.47 million urban employed persons, fitting in the confidence interval of the CPC member proportion estimated in this survey. As mentioned earlier, we hadn’t used the CPC member as a variable in the calibrated survey data, and it thus can be used as an internal validity indicator.

[1] https://baijiahao.baidu.com/s?id=1670921547438002495&wfr=spider&for=pc。

IX. Conclusions

    The 2019-2020 working environment survey implemented a complete project design, strict process supervision and cautious data processing. Although the investigation was completed with great difficulties against the COVID-19 epidemic, there are still deficiencies, such as the inability of the social survey to cover high-end workplaces, confidential places, workers without addresses or workers with unapproachable addresses, as well as the survey failed to conduct within the original feasible scope owing to the epidemic. To some extent, the survey design, implementation, adjustment process and the final data are true indicators for the particularity of this period.

    Social survey, as a process of information collection and knowledge production in public space, depends on the stable social environment, the willingness of respondents to open their doors of hearts, rooms, houses and companies, together with the professional ethics and diligence of survey executors. As the special survey data of the national urban employment environment in the sociology field, they can still be used as a solid foundation for theoretical research, hypothesis testing and policy analysis, even with their imperfect quantity and quality.