An overview of mathematical literacy in Brazil: an interview with PISA analyst Esther Carvalhaes

PISA analyst Esther Carvalhaes during her talk at the
University of São Paulo,
in May (Credit: RIDC NeuroMat)

The 2015 OECD’s PISA result “tells us that most students in Brazil, after spending many years in compulsory education, are unable to make use of mathematics concepts and procedures to perform even basic operations and solve very simple problems.” PISA stands for Program for International Student Assessment, a study to assess the performance of 15-year-old students in various countries on three academic areas: reading, mathematics and science. The recently published 2015 assessment has indicated that Brazil “was well below the average performance of students in the developed world in all three areas tested.”

Statements above are made by PISA analyst Esther Carvalhaes, in an exclusive interview for the NeuroMat Newsletter. Carvalhaes came to Brazil as a speaker guest for the meeting “Numeracy in Brazil: diagnoses and perspectives,” that the Research, Innovation and Dissemination Center for Neuromathematics (RIDC NeuroMat) held on May 16. This event was part of the Brazilian Mathematics Biennium and had the support of the São Paulo Research Foundation and the University of São Paulo School of Education. Slide presentations are available at the official web page of the event.

Speakers at the roundtable on the “Diagnosis of Mathematical Literacy in Brazil,” organized by NeuroMat in May
(Credit: RIDC NeuroMat)

In what follows Carvalhaes provides an overview of PISA results and indicates challenges and possible solutions for pervasive innumeracy in Brazil. As she pointed out, “Teachers are the backbone of any education system. Brazil needs to support the profession in any way possible: offer good training, work to raise the status of the profession in society, ensure decent salaries, open paths for career growth in the profession that recognise various levels of expertise, look for opportunities to reward excellence in teaching and improve the working conditions of teachers.”

In OECD’s PISA, Mathematics was given a special emphasis, as it is considered one of the three areas being assessed, along with reading and science. Could you please explain why this emphasis has been put on Mathematics?

The test is administered every three years since 2000 and each time, the focus is on one of these areas (even though students also respond to questions related to the other two areas). For example, in 2015, the focus was on science; the previous test in 2012 focused on mathematics and the next one in 2018 will have an emphasis in reading. The interest is in assessing not only what students know in these three areas, but most importantly, how they apply what they learned in school to solve problems in the real world or in new contexts. At age 15, students in most countries are coming to the end of compulsory education and it is important to ask at that point to what extent students were able to develop some foundational skills in school, such as literacy and numeracy skills but also an understanding of the world around us and how scientific knowledge evolves. These are basic requirements for the integration of these young people into society whether they decide to pursue further education or to join the labour market.

What are the known impacts of mathematical illiteracy in a country?

The OECD measures not only the performance of young students through PISA. A separate international study called PIAAC (The Survey of Adult Skills) is dedicated to measure some important cognitive and workplace skills that individuals need to participate productively in society thus contributing to economic growth and social development. This survey is conducted in over 40 countries and it assesses the literacy and numeracy skills of adults as well as their ability to solve problems in technology-rich environments.

The survey reveals that adults with better literacy, numeracy and problem-solving skills are much more likely to be employed, to remain employed, to earn better salaries, to increase their skills at the workplace and to have better health and social outcomes over their lifetime. The impact of numeracy skills on employment outcomes, for example, is remarkable: improvements in numeracy skills at every level of proficiency are associated with increasingly greater chances of being employed. Among tertiary-educated adults, those with the highest level of skills can be more than seven times as likely to be employed as those with the lowest levels of proficiency who have difficulty accessing, using, interpreting and communicating mathematical information in a range of contexts. The same happens with earnings: the greater the numeracy skills of adults, the more they earn. Data from 2012 show that the earnings advantage, on average across participating countries, corresponded to 56% between tertiary-educated adults with high proficiency levels and similarly educated adults with low numeracy skills. In Estonia, adults with advanced skills earned more than twice the salary of those with the same degrees but with low numeracy skills.

Speakers at the roundtable on the “Impact of Assessment in Public Policies and Education Work,” organized by NeuroMat in May (Credit: RIDC NeuroMat)

Better educated adults also enjoy better social outcomes: they are likely to enjoy better health and be more satisfied with their lives. The benefits of better education go way beyond individual benefits. Society as a whole benefits: better educated adults can not only find better jobs but also help create them thus boosting innovation and productivity in the country. They also pay more taxes, participate more in the political life of their country and volunteer more than less educated adults. The benefits are not only economic but also social.

Is Brazil doing especially poorly in Mathematics, or is the level of numeracy strongly associated to how we achieve in reading and science?

Among 72 countries and economies that participated in PISA 2015, Brazil’s score was among the 10 countries with the lowest performance. But rankings are not the only way to look at the results. When we look at the proportion of underperforming students within the country, we find that the results for Brazil are disappointing in all three areas, but particularly so in mathematics. In mathematics, around 70% of our students did not attain a basic level of proficiency where students can solve simple problems. At this level, students are able to extract information from a text or a table, conduct simple operations that involve whole numbers and make a literal interpretation of the results. In Brazil, the proportion of students below the basic level of proficiency corresponds to 57% in science and to 51% in reading.

Are there specific causes for high levels of innumeracy, or is this mostly associated to features of the school system?

The PISA test has the advantage of observing very high technical standards, which allow the resulting scores to be comparable across different countries (not an easy task!). The same technical qualities allow us to also observe trends over time in each country. On the other hand, most of the data published in the PISA reports are correlational and as such, do not allow inferences of causality. Therefore, it would be a mistake to see the factors analysed by PISA as causes that explain either the success or failure of students. That being said, it is possible, however, to have some insights and raise interesting hypotheses about what could be behind successful schools and successful school systems. PISA does that very well. Some lessons we learned follow common sense, but PISA substantiates some claims with empirical evidence.

For example, we know that students who miss class frequently or who are often late for class tend to have a low performance in the test. It makes sense as these behaviours may reflect not only a lack of motivation to learn, but also represent reduced opportunities to learn, which does impact performance. Once in class, students need to receive high quality instruction. Simply spending time in school and in class is not enough to guarantee that students are learning. Of course, they need sufficient time to learn but we know by looking at comparisons between countries that beyond a certain threshold of learning time, simply adding hours of instruction per week or per year does not necessarily correspond to gains in performance. What happens in the classroom is very difficult to assess, but it is critical for learning. We know, for instance, that students whose teachers engage in practices designed to encourage students’ active participation and those whose teachers use “cognitive activation” strategies tend to do better in the mathematics test than other students. Cognitive activation strategies are those that encourage students to think about the problems and ways to solve them, like asking students to explain how they arrived at their conclusions.

The strategies that students use to learn mathematics also matter. In PISA, for example, students who often use memorisation when studying mathematics are more likely to correctly answer easy questions whereas those who use control or elaboration strategies, such as deciding what parts of the material are more important to learn or trying to connect new concepts to what they already know, have a greater chance to correctly answer more difficult items. And it turns out that the way teachers teach is also related to the way students learn. Having well educated, well prepared, well paid teachers is, of course, an essential element of successful school systems.

In the 2015 PISA, Brazil appears to remain in a worrisome situation in terms of achievements in Mathematics. What is this assessment specifically telling us about our current level of mathematical literacy?

Attendees at the meeting NeuroMat organized on mathematical literacy in Brazil,
in May (Credit: RIDC NeuroMat)

Well, this assessment tells us that most students in Brazil, after spending many years in compulsory education, are unable to make use of mathematics concepts and procedures to perform even basic operations and solve very simple problems. It is a serious problem: in spite of all the public investment in education, the desired goal of offering quality education to all children is not met. And the proportion of students who can be considered top performers in mathematics is also very low: less than 1% of the students. These are the students who can solve the most difficult and complex problems, who show flexibility in the way they apply what they know to new situations, who demonstrate a solid understanding of basic mathematics concepts and use them appropriately, who can make connections between different sources of information and generalize results beyond their literal interpretations.

This has important consequences for the future. Economic and social development in contemporary societies depend on people’s ability to use what they know to address current and future challenges.

Climate change and technological development, for example, are some examples of the types of realities that require people to develop more than only basic literacy and numeracy skills. Countries need both, people with a reasonable, basic level of literacy and numeracy skills to adapt to new technologies in the workplace, for example, but also a pool of top performers who can lead the way to innovation and scientific development. The reality is that nowadays, about half of the jobs in the industrialised world can potentially be automated, digitized or outsourced. Basic skills are no longer enough. The global economy will increasingly reward those who have more advanced skills, who can use higher-order and flexible thinking and who can creatively apply what they know to create new knowledge and new value. Brazil still has a long way to go to improve both, the basic level of numeracy in the country for most students and to enlarge its pool of top performers.

Could you please provide examples of skills and knowledges that are being assessed in Mathematics and that specifically Brazilian students have difficulties with?

The mathematics PISA test is interested not only in what students know, but also in how they use what they know to solve problems in different contexts, often in the real world. Besides being tested on some conventional mathematics contents (quantity; uncertainty and data; change and relationships; space and shape), PISA wants to assess how well students do in key processes that are required for successful problem solving. In particular, students are expected to “formulate” situations mathematically (e.g. translate a real world problem into mathematical terms), “employ” mathematical concepts, facts, procedures and reasoning to arrive at solutions; and “interpret” the solutions found (including evaluating whether the solution “makes sense” and putting numerical results in the context of the problem). The report from 2012 (the last cycle where mathematics was the focus of the PISA test) compared the results of these process “subscales” in mathematics to the average score of students who took the test in Brazil. We found that the students did better than average in the “interpreting” category, but showed weaknesses (scores lower than average) both in “formulating” and “employing” mathematics concepts. Similar results were found in PISA 2015.

The Ministry of Education in Brazil through its analysts in INEP (Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira) further analysed these results within the country and looked at differences in outcomes by types of context and by types of items in the test. They concluded that students in Brazil did better in questions presented in personal and social contexts (e.g. making decisions about the purchase of a car), but had difficulty with items placed in scientific or occupational contexts. They also found that students did better in items of simple multiple choice, but not so well in more complex multiple-choice items and did worse in open-ended questions that require students to explain, for example, how they arrived at a particular answer. In terms of contents, some items involving geometry, such as concepts of space and shape, and others involving algebra were found to be particularly difficult for the students.

OECD’s PISA is a cross-national study. How does Brazil compare to other countries from the developed and developing worlds in 2015 in terms of Mathematics?

PISA analyst Esther Carvalhaes presented on the 2015 PISA results for Brazil at
the NeuroMat event on mathematical literacy, in May (Credit: RIDC NeuroMat)

In Brazil, the overall performance of students who participated in the PISA test in 2015 was well below the average performance of students in the developed world in all three areas tested. Brazil scored 407 points in reading (OECD average: 493); 377 in mathematics (OECD average: 490) and 401 in science (OECD average: 493). These results were somewhat expected as rich nations can invest more heavily in their education systems. But the results cannot be fully explained by the different level of investment in various countries.

In mathematics, Brazil’s score was among the 10 countries with the lowest performance, among all the 72 participating countries and economies in 2015. And a number of other Latin-American countries outperformed Brazil, such as Chile, Uruguay, Costa Rica, Colombia, Peru and Mexico. When we compare Brazil to Chile, Uruguay or Mexico, for example, the difference in score points corresponds to at least one full year of schooling.

Brazil has been assessed through PISA in several occasions now. What is the longitudinal narrative for Brazil in this assessment in terms of Mathematics?

Brazil has participated in PISA since its inception in 2000. While the results in reading and in science have shown stagnation at low levels of performance, in mathematics there are more positive developments: the average mathematics score increased by 21 score points between 2003 and 2015 (a significant increase of 6.2 score points every three years). At the same time, however, the most recent period (2012-2015) has seen a decline in mean performance in mathematics by 11 score points. To have a point of reference for what this means, 30 score points in the PISA scale correspond approximately to one year of schooling.

This shows that the country is trying to improve and has made some progress, albeit modest, in the right direction. Even though the results in 2015 are still very low, long-term improvements such as these are very important, especially as they happened during the time when Brazil was also increasing the number of 15-year-old students who are in school. Expanding the school offer in secondary education without dramatic declines in performance is already a positive development. But a lot remains to be done.

In the last decade or so, the Brazilian government made some important moves in the direction of investing more in the public school system. One of these investments was the Brazilian Olympiad of Mathematics. What went wrong? How did we end up with sustaining a low level of mathematical literacy while money seems to have been flowing?

Increases in investment in Brazil are a positive development. It is important to remember that the cost of not investing in education is often higher than the cost of investing in the future. And Brazil has made efforts in this respect. The cumulative expenditure per student between the ages of 6 and 15 in Brazil (USD 38 190) corresponds to 42% of the average spending per student across OECD countries (USD 90 294), but it used to be 32% in PISA 2012. Increases in spending, however, still need to translate into better learning outcomes. Other low-spending countries, such as Colombia, Mexico and Uruguay, spend less per student than Brazil does, and perform better in mathematics. Chile, whose expenditure per student is similar to that of Brazil (USD 40 607), score significantly higher in mathematics.

What we know from the PISA data is that Brazil has been moving in the right direction by steadily improving its overall student performance in mathematics since 2003. This is a good sign. This positive trend, unfortunately, was interrupted by a decline in performance in 2015. The data do not tell us the reasons behind such performance gains or losses, but this decline seems to be associated with an increase in the number of students who are enrolled in secondary school during the same period. In particular, 71% of 15-year-olds are enrolled in school, in grade 7 or above, which is 15 percentage points higher than in 2003. This is in itself a remarkable improvement.

I ignore studies showing the impact of the latest waves of investment in mathematics education on students learning outcomes in the country, so I would not know what evidence exists linking the Mathematics Olympics to overall student performance. But based on the international evidence we have, it is clear that another problem needs to be addressed urgently: the large gap in performance that exists between advantaged and disadvantaged students. The recent downward trend is driven, partially, by widening inequalities in achievement: while the best-performing 10% of students score, as in the past, close to the OECD average, the lowest-performing 10% of students fell farther behind over the most recent period. A similar widening of achievement gaps is also observed in science and reading. The widening of the achievement gap may underline the significant challenges faced by Brazil in integrating the weakest students. The performance divide is also socio-economic as most low performing students come from disadvantaged background.

And that requires investments to give disadvantaged students the same opportunities to learn that advantaged students already have. Other successful education systems in the world already discovered that: policies that focus on decreasing the socio-economic disparities in performance also tend to lead to improvements in the overall performance of the country. This should be particularly interesting to policy makers in Brazil where large sections of the student population come from disadvantaged families. As I mentioned before, any country needs both: students with basic, solid literacy and numeracy skills, but also a pool of top performers to lead the way to new discoveries and economic prosperity. Public investment needs to be allocated to enhance learning opportunities for all and not only a few.

Through their communications channel, OECD PISA’s analysts provide insights on how to improve a country’s level of mathematical literacy. What are the most important ingredients for improving the level of mathematical literacy in Brazil?

One of the priorities in Brazil would be to allocate resources to help disadvantaged students. As we mentioned before, although these students are the majority in the country, they are the ones who suffer the most with very low performance. That does not mean that the performance of advantaged students is satisfactory either. Students with a comparable socio-economic background are doing a lot better in other countries, such as Portugal and Spain. The country can advance on both ends: raising the performance of both groups while trying to minimize the achievement gap between them. This cannot happen overnight. Given the socio-economic map of underperformance in the country the best chance for increasing the overall level of mathematical literacy in Brazil is really to improve learning for those who are at the bottom.

This cannot be done without good teachers. Teachers are the backbone of any education system. Brazil needs to support the profession in any way possible: offer good training, work to raise the status of the profession in society, ensure decent salaries, open paths for career growth in the profession that recognise various levels of expertise, look for opportunities to reward excellence in teaching and improve the working conditions of teachers. Teachers need to have their time protected to prepare lessons and give feedback to students, to collaborate and learn with other teachers, and to keep learning. Teachers who are excited about continuous learning and who are able to build and identify best practices are more likely to effectively help students learn. Another area for optimising learning has to do with the distribution of talent. This is not an easy task, but better overall results would include finding ways to assign the best teachers to those areas that need them the most and creating incentives for them to teach in at-risk areas. This requires a general awareness in the profession and in society that a concentration of success and talent does not serve the country in the long run. Brazil needs to counter deep historical influences that traditionally favoured a certain elitism in education. We inherited centuries of a society based on slavery, on economic models of export of raw materials and a recent past of political authoritarianism. All of these influences have reinforced the notion that most people would not need more than some rudiments of reading, writing and numeracy in life. A model of schooling based on social and economic exclusion is simply not sustainable and Brazil needs to work very hard to overcome these tendencies.

Other areas that could be targeted have to do with minimizing waste of any sort: waste of talent or resources. We know, for example, that the use of time in the classroom is one of the most precious resources students and teachers have in their hands, and countries differ a lot in what they get in terms of learning for each hour a student spends in the classroom. Of course, the types of students also make a difference, but for most kids especially disadvantaged ones the classroom is still the best opportunity to learn they will ever have and we need teachers well prepared to make good use of the time they spend with their students. Waste in the system can also manifest itself in grade repetition policies. We know nowadays that retaining students in the same grade is a very costly policy with no guaranteed results. It would be best to invest in preventing failure in the first place by identifying early students who are at-risk and helping them overcome their difficulties as they progress.

This piece is part of NeuroMat's Newsletter #40. Read more here

Featuring this week:

Stay informed on our latest news!

Previous issues

Podcast A Matemática do Cérebro
Podcast A Matemática do Cérebro
NeuroMat Brachial Plexus Injury Initiative
Logo of the NeuroMat Brachial Plexus Injury Initiative
Neuroscience Experiments System
Logo of the Neuroscience Experiments System
NeuroMat Parkinson Network
Logo of the NeuroMat Parkinson Network
NeuroMat's scientific-dissemination blog
Logo of the NeuroMat's scientific-dissemination blog