Driving Assessment Scores with Effective Calculator Implementation

This study presented a secondary analysis of the National Assessment of Educational Progress (NAEP) dataset. The paper examined the impact of calculator exposures on eighth-grade students' 2019 NAEP mathematics assessment scores. To better understand the impact of calculator use on the mathematics achievement of eighth-grade students, this study used a quantitative descriptive research design to analyze secondary data extracted from the 2019 NAEP data set. The findings are: (1) the frequency of calculator use does not impact assessment scores during math lessons. (2) Students who practice using calculators on math tests and quizzes have higher math assessment scores. (3) Frequent primary calculator use can impact math assessment scores. (4) Students who use graphing calculators frequently score higher on math assessments. In conclusion, these findings indicate that using calculators in mathematics is impactful; however, teachers must be trained to implement them effectively.


Introduction
Throughout the years, technological advancements have directly impacted the way students learn and interact with mathematics. From the development of the slide rule to today's advanced computer algebraic systems, teachers have had to adapt their instruction to include these tools to prepare students for the world they live in. Digital math tools can offer many benefits to students, such as their ability to solve complex mathematical computations in seconds. They can be used to do numerical calculations and create graphical representations, allowing students to save time on paperpencil skills and shift their focus to the problem's solution (Korenova, 2015;Tobin & Weiss, 2016). Learning and teaching are intrinsically linked with the assessment to measure the effectiveness and attainment of instructional objectives (Phuthi, 2021). However, like many significant advancements, it is feared that improper exposure to these tools may cause harm to students' mathematical skill sets. Based on the technology used, the Directorate of Senior High School Development (2008) classifies teaching materials into four categories, namely printed teaching materials, including handouts, books, modules, student activity sheets, brochures, leaflets, wallcharts, photos/pictures, and model/mockup (Zarvianti & Sahida, 2020).
When implemented in the classroom, digital math tools such as basic calculators, graphing calculators, and computer algebraic systems (CAS) can enhance student achievement. Research comparing assessment scores of students who did and did not use calculators has shown that students who reported calculator use outperformed those who reported no calculator use (Walcott & Stickles, 2012). Similar results were demonstrated in Schmidt et al.'s (2009) study, concluding that a majority of students who learned using CAS technology did noticeably better on assessments when compared to students who knew without CAS. Students who have learned to use this tool to support their mathematical thinking effectively can transfer these skills to assist them on standardized tests and provides an advantage over students who do not know how to use these tools.
While digital math tools offer much promise to student achievement in mathematics, research has also shown that technology use can have adverse effects on student learning. Many educators fear too much calculator use can damage students' cognitive development (Tobin & Weiss, 2016). Students who do not have a solid mathematical foundation will not be able to comprehend the outputs given by these devices to apply them effectively. Technology in math class must be used purposefully and not be exclusively the only tool used to learn and solve (Tobin & Weiss, 2016). However, implementing and teaching appropriate technology use is often pushed to the side, as it requires a significant time commitment (Tobin & Weiss, 2016).
The current study will explore the relationship between calculator use and student performance on mathematical assessments. The findings in this study will provide insight for math teachers and coaches regarding the impact calculators have on student performance. Educators who are resistant to integrating technology in the classroom and implement it regularly can use the results from this study to make data-driven instructional choices regarding the use of technology in their classrooms.

Purpose Statement
This study aims to examine the impact that different calculator exposures have on the 2019 NAEP Eighth-Grade Mathematics Assessment.

Research Questions
(1) How does calculator use during math lessons impact math assessment scores?
(2) How does the frequency of calculator use on math tests impact math assessment scores?
(3) How does the frequency of primary calculator use impact math assessment scores? (4) How does the frequency of graphing calculator use impact math assessment scores?

Methods
Our theoretical framework for this research adopts a scientific inquiry-based approach. The framework was described in detail in The Impact of Conversations on Fourth Grade Reading Performance -What NAEP Data Explorer Tells? The research methods combined the inquiry process with scientific knowledge, reasoning, and critical thinking. We started with an extensive exploration of the dataset, which led to designing the research questions. The research questions further guided us to mine the data with great in-depth.
Data for the current study were collected from the National Assessment of Educational Progress (NAEP) online database. Starting in 1969, NAEP was developed to measure student achievement nationally and is the only national representative and continuing assessment that counts what American students know and can do in various subject areas. For years, educational policies and practices have been improved to reflect the results of these assessments (NCES, 2019). In addition to reporting various content areas and grade levels, different variables may be explored to identify trends in assessment outcomes. The present study used NAEP's embedded Data Explorer tools to produce data for the 2019 eighth-grade mathematics assessment. The data was further analyzed with four coded questions regarding calculator use.

Participants and Sampling
NCES (2020) reported that the NAEP mathematics assessment is given to students in grade 8 every two years. The examination is administered digitally to measure the students' mathematics

Vol. 3 (3), August 2023
knowledge and ability to apply these skills to real-world scenarios. The current study will focus on the data collected from the 2019 mathematics assessment taken by 147,400 students.

NAEP Sampling and Data Collection
NAEP does not assess entire student populations but utilizes a probability sample design to create a sample representative of the entire student population (NCES, 2019). For the mathematics assessments, students were given 60 minutes to complete the test. Following the assessment, students were given a questionnaire regarding activities inside and outside the classroom pertaining to math (NCES, 2020). The test measures content knowledge from the following categories: number properties and operations, measurement, geometry, data analysis and probability, and algebra (NCES, 2011).

Public School Selection in State Assessment Years
A multistage design is used to select samples of students attending public, private, Bureau of Indian Education, and Department of Defense schools. Therefore, the results only represent a portion of the whole student population. To combat this issue, results are weighted to account for disproportionate representation, such as the oversampling of schools with a large population of certain racial/ethnic groups (NCES, 2019). The most recent assessment, taken in 2019, will be analyzed for the current study.

Data Analysis
This study identified four variables used in the 2019 NAEP (Kolhoff, 2021). Each variable focused on student factors related to calculator use. Variable one (M832303) asked students how often they used graphing calculators (student-reported). Variable two (M815301) asked how often students use basic calculators (student-reported). Variable three (T089301) asked how much students can use calculators during mathematics lessons (teacher-reported). Lastly, variable four (T089601) asked educators how often their students use calculators on math tests or quizzes. These four variables were utilized to analyze the relationship between math scores for 8th-grade students and calculators. Standardized testing allows assessment of student academic achievement in reading. The Every Student Succeeds Act (2015) contains provisions allowing state lawmakers to determine the assessment tool and standards test (Hamilton, 2021).

NAEP Data Explorer
The NAEP Data Explorer provides a variety of tools that can be used to explore NAEP assessment results. Tables and charts were created and analyzed using the specific variable chosen for this study. Cohen's d effect size was calculated to determine the effect size (Becker, 2000). The means of the data were compared to determine the differences in standard deviations (Cohen, 1988). This analysis can identify the strength and importance of significant data found. This study explores the relationship between calculator use and 8th-grade math scores for students in national public school settings. However, it must be noted that no causal relationships can be determined from this study.

Results and Discussion
The NAEP Data Explorer provided data used for this study regarding the 2019 NAEP mathematics assessment against variables about calculators. The average score for all students who participated in the 2019 NAEP mathematics was 281 (scale range from 0-500) with a standard deviation of 40. For the current study, two student and two teacher factors were analyzed. The first teacher factor was calculator use during math lessons, and the second was how often students can use calculators during math tests and quizzes. The two student factors reported on basic calculator and graphing calculator use in the classroom. The results include the means and standard deviations for each variable examined and significance testing results. Cohen's d effect size was calculated when significance was found using the University of Colorado's effect-sized website https://lbecker.uccs.edu.

Calculator Permission During Math Lessons
The first research question asked about the impact of the 2019 NAEP mathematics assessment based on calculator permission during math lessons for 8th-grade students. Utilizing the NAEP 2019 assessment, research question one looked at the variable "To what extent are students permitted to use calculators during mathematics lessons." This variable was grouped under the Instructional Content and Practice and the Modes of Instruction/ Classroom Activities subcategory. The data from this variable was compared to the average NAEP composite mathematics score of national public school students in 2019.  Table 1 shows the 2019 average mathematics score for eighth-grade national-public school students based on the frequency of calculator use during math lessons. Students with unrestricted (SD = 39) or restricted use of calculators (SD= 40) had an average scale score of 281. Students not permitted to use a calculator during math lessons had an average scale score of 280, SD=39.

International Journal of Social Learning
(IJSL) Table 2 shows the results of the significance test run to determine the difference in average scale scores between variables for calculator use during math lessons. No significance was found, so Cohen's d-effect size was not calculated.
x No significant difference.

Calculator Use During Tests
The second research question asked what the impact was on the 2019 NAEP mathematics assessment based on the frequency of calculator use during math tests and quizzes of 8th-grade students. Utilizing the NAEP 2019 assessment, research question two looked at the variable "When you give students a mathematics test or quiz, how often do they use a calculator." This variable was grouped under the Instructional Content and Practice and the Modes of Instruction/ Classroom Activities subcategory. The data from this variable was compared to the average NAEP composite mathematics score of national public school students in 2019. Table 3 shows the 2019 average mathematics score for eighth-grade national-public school students based on calculator use on math tests and quizzes. Students who were never allowed to use calculators on math tests or quizzes had the lowest scale score of 275, SD=40. Students who are always allowed to use calculators on math tests and quizzes had a scale score of 281, SD=39. Students who are sometimes allowed to use calculators on math tests and quizzes had the highest scale score of 282, SD=40. Table 4 shows the results of the significance test run to determine the difference in average scale scores between variables for calculator use during math tests and quizzes. The p-value was significant for students who sometimes or always use calculators on math tests compared to those who never use calculators on math tests or quizzes (p=.002 sometimes; p=.0036 never).   To report the effect sizes of significant variable responses, Cohen's d was calculated and is presented in Table 5. An effect size of 0.2 is small, 0.5 is medium, and 0.8 is immense (Cohen, 1988). Cohen's d effect size was calculated to determine the effect size of those students who sometimes use calculators on math tests or quizzes to students who never use calculators on math tests or examinations. Cohen's d effect size for those who sometimes use calculators to those who never do

Basic Calculator Use
The third research question asked about the impact of the 2019 NAEP mathematics assessment based on the primary calculator used by 8th-grade students. Utilizing the NAEP 2019 assessment, research question three looked at the variable "In your math class this year, how often have you used the following types of calculators? Basic calculator." This variable was grouped under the Instructional Content and Practice and the Modes of Instruction/ Classroom Activities subcategory. The data from this variable was compared to the average NAEP composite mathematics score of national public school students in 2019.  Table 5 shows the 2019 average mathematics score for eighth-grade national-public school students based on the frequency of primary calculator use in math. Students who never or hardly ever use a basic calculator in math had the highest average scale score of 294, SD= 43. Students who use a basic calculator less than half of the time in math had the second highest average scale score of 284, SD=42. Students who use the primary calculator half the time in math had the lowest average scale score of 274, SD=39. Table 6 shows the results of the significance test run to determine the difference in average scale scores between variables for primary calculator use in math class. The average scale score of students who use the basic calculator less than half the time, about half the time, more significant than half the time, and all or most of the time in math class was significantly (p<0.001) lower than students who never or hardly use basic calculators in math (p=0). Students who use the basic calculator about half of the time, more than half the time, or all or most of the time had a significantly (p<0.001) lower average scale score than students who use the basic calculator less than half of the time in math class (p=0.00) about half of the time; p=0.0032 > ½ the time; p=0.00 all or most of the time). On the other hand, students who use the basic calculator more than half the time or all or most of the time had significantly (p<0.001) higher average scale scores than students who use the calculator about half of the time (p=0.00). Lastly, students who use the basic calculators all the time had significantly (p<0.001) lower average scale scores than students who use the basic calculator more than half of the time (p=0.0018). To report the effect sizes of significant variable responses, Cohen's d was calculated and is presented in Table 7. An effect size of 0.2 is small, 0.5 is medium, and 0.8 is immense (Cohen, 1988

Vol. 3 (3), August 2023
who never or hardly ever use the basic calculator was d=-0.235, showing a small effect size. Cohen's d between the mean score of students who use basic calculators about half the time compared to those who never or hardly ever use basic calculators had a medium effect, d=-.497. Students who use the basic calculator more than half the time compared to those who use the basic calculator never or hardly ever, less than half the time, or about half the time had a small effect size (d=-0.32 never or hardly ever; d=-0.075 < half the time; d=-182 about half the time). Lastly, Cohen's d between the mean scores of students who use the basic calculator all the time compared to those who use it never or hardly ever, less than half the time, about half the time, and more than half the time were respectively -0.374, -0,126, -0.132, -.053, all showing a small effect

Graphing Calculator Use
The fourth research question asked what the impact was on the 2019 NAEP mathematics assessment based on graphing calculator used by 8th-grade students. Utilizing the NAEP 2019 assessment, research question four looked at the variable "In your math class this year, how often have you used the following types of calculators? Graphing calculator." This variable was grouped under the Instructional Content and Practice and the Modes of Instruction/ Classroom Activities subcategory. The data from this variable was compared to the average NAEP composite mathematics score of national public school students in 2019. Table 8 shows the 2019 average mathematics score for eighth-grade national-public school students based on the frequency of graphing calculator use in math. Students who use the graphing calculator in math all or most of the time had the highest average scale score of 290, SD= 42. Students who never or hardly ever use the graphing calculator in math had the second highest average scale score of 281, SD=37, followed by students who use it more than half the time with an average scale score of 280, SD= 42. Students who use a graphing calculator about half the time in math had the lowest average scale score of 273, SD=41. Table 9 shows the results of the significance test run to determine the difference in average scale scores between variables for graphing calculator use in math class. The average scale score of students who use graphing calculators less than half the time, or about half the time, in math class was significantly (p<0.001) lower than students who never or hardly use basic calculators in math (p=0.002 < half the time; p=0 about half the time). However, the average scale score of students who use graphing calculators all or most of the time in math class was significantly (p<0.001) higher than students who never or hardly use basic calculators in math (p=0). Students who use graphing calculators about half of the time had a significantly (p<0.001) lower average scale score than students who use the graphing calculator less than half of the time in math class (p=0). On the other hand, students who use graphing calculators all or most of the time had a significantly (p<0.001) higher average scale score than students who use the graphing calculator less than half of the time in math class (p=0). The average scale score of students who use graphing calculators more than half the time or all or most of the time in math class was significantly (p<0.001) higher than students who use the graphing calculators about half of the time (p=0). Lastly, students who used graphing calculators all or most of the time scored significantly (p<0.001) higher than students who used them more than half of the time (p=0). To report the effect sizes of significant variable responses, Cohen's d was calculated and is presented in Table 10. An effect size of 0.2 is small, 0.5 is medium, and 0.8 is immense (Cohen, 1988). Cohen's d effect size for those who use graphing calculators less than half the time compared to those who never or hardly ever use the basic calculator was d=-0.052, showing a small effect size. Cohen's d between the mean score of students who use graphing calculators about half the time compared to those who never or hardly ever use graphing calculators or use graphing calculators less than half the time had a negligible effect (d=-205 never or hardly ever; d=-0.148 <half the time). Students who use the graphing calculator more than half the time compared to those who use the graphing calculator about half the time had a small effect, d=.169. Students who use the graphing calculator all or most of the time, compared to those who use it more than half the time, had a small

Discussion
This study was designed to examine the impact of different calculator exposures on the 2019 NAEP Eighth-Grade Mathematics Assessment. Through a quantitative analysis utilizing the 8thgrade NAEP mathematics data from 2019, various calculator factors were cross-examined with mathematics achievement scores to identify different trends. The following sections will discuss how findings from the NAEP Data Explorer provide insight related to this study's research questions.

Calculator Permission During Math Lessons
The study found no significant difference in average scale scores on the 2019 NAEP mathematics assessment based on students' responses to calculator permission during math lessons. The NAEP data revealed that students who had unrestricted calculator use, restricted calculator use, or no calculator use during mathematics lessons all had similar average scores on their NAEP mathematics assessment. Results from the current study did not align with Schmidt et al.'s (2009) study on student achievement, comparing students who were allowed to use calculators during math class and those who did not. Student achievement had no significant difference based on the use of calculators. Many teachers feel they should restrict using calculators in the classroom because they believe it hinders their students' ability to do basic mathematical computations (Cabanilla-Pedro, 2006). However, based on the repeated results of studies of calculator impact during math lessons, restricting calculators is not benefiting students' ability to perform better on math assessments. The opposing viewpoint of allowing unlimited calculator use during math lessons also does not indicate higher performance on mathematics assessments.
Based on the results of the current study, the only information provided by the questionnaire is the frequency of calculator use during math lessons. How the technology was being implemented in the classroom or the teachers' experience with effectively implementing these devices needs to be clarified. There must be more than the presence or absence of calculators in the classroom to enhance learning. As shown in Cabanilla-Pedro's (2006) and Burrill et al.'s (2002) study, the frequency and quality of technology integration will lead to increased student achievement.

Calculator Use During Tests
The research found that teachers who responded that their students never use calculators on math tests scored lower on the 2019 NAEP mathematics assessment than those who sometimes or always use calculators on math tests. However, the scores of students who sometimes use calculators on math tests compared to those who always use calculators on math tests were comparable. Based on these results, restricting students from any calculator use on math tests puts students at a disadvantage over students who had the opportunity to use calculators during some or all math tests.
The results of the current study confirm Schmidt et al.'s (2009) study that revealed that students who have experience using calculators will have an advantage on assessments over students that do International Journal of Social Learning (IJSL) not. Previous studies have shown that teachers restrict calculator use on evaluations because they believe it is detrimental to their learning. However, based on the results of the current study, teachers are putting their students at a disadvantage if they are never allowing their students to practice using calculators to assist their mathematical performance. While students don't have to have access to calculators on every exam or test, teachers should still allow students to use these devices on some assessments. Even with limited exposure to calculator use on tests, as also demonstrated in Sheets' (2007) study, students will be at an advantage on future assessments over those with no exposure.

Basic Calculator Use
This study found that students who spent the least time using basic calculators in math had the highest score on the 2019 NAEP math assessment. Technology offers many affordances to student learning; however, basic calculators only have the capabilities to assist with computational skills. Suppose these devices do not enhance higher-order mathematical thinking and only supplement basic solving. In that case, students who rely too heavily on basic calculators may need to gain basic foundational skills (Cabanilla-Pedro, 2006). Sheets (2007) found that if students' calculator use is limited, their computational skills will benefit.
In the current study, the frequency "never or hardly ever" had a medium effect size over the frequency of using the basic calculator "about half the time." Students need to be taught how to use calculators as an effective tool to assist them in problem-solving, as more than their presence is required to enhance mathematical proficiency. Students not being taught to read and understand the problem or interpret solutions to pick the correct answer are not benefiting from the devices (Cabanilla-Pedro, 2006). About half the time, students who use these devices need more practice effectively and show lower average scale scores on the 8th-grade 2019 NAEP math assessment.

Graphing Calculator Use
The results of this study show that students who use graphing calculators all or most of the time scored the highest on the NAEP mathematics assessment. These supported Ellington's (2006) study showing that students improved when calculators were included in testing and instruction.
Similar to the impact of primary calculator use, the current study shows that students who use graphing calculators about half the time score the lowest. For these devices to be practical tools, they must routinely be part of the learning process (Burrill et al., 2002). Students familiar with graphing calculators can use these devices to attempt various problem-solving strategies when they become stuck solving a problem (Campagnone, 2005). However, students unfamiliar with using these devices have fewer options to assist them in solving a problem when they get stuck. Students who are taught to solve problems using graphing calculators effectively will perform higher on math assessments, as shown in Tajudin et al.'s (2007) study. Frequent use of graphing calculators in the classroom and on reviews offers more promise of a positive impact on math assessment scores. The Learning Implementation Plan is a guideline for implementing learning because it contains Core Competencies, Basic Competencies, achievement indicators, learning objectives, materials, methods, learning activities, and an assessment of learning outcomes (Abdullah, 2023).

Conclusion
The study results indicated that the frequency of calculator use impacted the assessment scores. The type of calculator used, essential or graphing, directly impacted the frequency necessary for achievement. Students who use basic calculators too frequently hurt their assessment scores. On the other hand, students who used graphing calculators more frequently had better math assessment scores. The study's results indicated that the presence or absence of calculators in math lessons does not impact students' assessment scores. The data showed that the time spent using calculators during math lessons was insignificant to their assessment scores. The presence of calculators during math lessons is only helpful if paired with purposeful use and implementation. Students need to know how to use these tools effectively, or they will not enhance their learning.
Lastly, the study revealed that experience using calculators during math tests will positively impact students' math assessment scores. Data showed that it did not matter how frequently the student could use a calculator on math tests in school. However, any exposure resulted in higher NAEP assessment scores than students without opportunities to use calculators on math tests. Students need to practice using them on tests so that they can use them effectively when they are taking future assessments.

Implications
An important implication of this research is that K-12 teachers should only use calculators in the classroom if they purposefully use them. Based on the study, the benefits that come with the use of graphing calculators serve students more than just simply using basic calculators to supplement computations. Graphing calculators allow students different opportunities to model, solve, analyze, and explore mathematics, creating skills that lead to mathematical proficiency. However, by only using basic calculators to substitute mental math and computations, students need to become proficient in these skills and avoid missing the necessary foundational skills. The affordances of calculators only exist if teachers show their students how to use these devices as digital tools to assist them in their problem-solving. Students who are not taught to use these devices effectively will not benefit from their use. Lastly, educators should offer opportunities for students to use these digital tools on math tests so that they can practice effectively using them on future assessments, such as the NAEP mathematics assessment.
Implications of this research can guide administrators to provide educators with the proper tools and training to implement technology in the classroom successfully. Educators who only provide basic calculators for their students cannot enrich their lessons and impact student learning as they could if graphing calculators were available. These tools teach students to be effective digital mathematicians and prepare them for life outside of school. Additionally, teachers need proper training to implement graphing calculators in the classroom. More than simply simply simply providing these devices to students is necessary to impact learning. Teachers need training on implementing them into lessons and teaching students to use them as digital tools effectively.

Limitations
Limitations exist in this study due to the nature of the data. The data obtained from the 2019 NAEP math assessment is secondary and has therefore inherited all the potential validity problems when the data was collected. Since the variables in this study were pre-decided and may appear nonnatural, they should not be interpreted as a cause-and-effect relationship. The data analysis was limited because they were only based on models in the NAEP Data Explorer. The sample size usually ranges from 10,000 to 20,000 students; however, due to confidentiality, the NAEP does not provide the exact sample size (NCES, 2018). Based on this, the validity of the research findings could be improved.
Limitations also exist based on the nature of the test and questionnaire. Standardized tests do not always represent a student's mathematical abilities best. For example, students with test anxiety or English Language Learners are often disadvantaged on these tests (Wolfe, 2010). Therefore, their test results may need to represent their abilities correctly. Additionally, there are limitations to this study because the survey assumes that all students interpret the survey questions in the same way. For example, one student's interpretation of calculator frequency may differ from another student's. Due to these examples, the impact of calculator frequency may hurt the validity of the results.

Recommendations for Future Research
This study used limited data from the 2019 eighth-grade NAEP math assessment to examine how the nature of calculator use in the classroom impacts students' average scale scores. The findings of the research question led to further questions that should be explored. The study showed that graphing calculators have a higher impact than basic calculators in the classroom. However, more research is needed to determine which strategies should be used to implement these tools in the classroom effectively. Based on prior research, other digital math tools, such as computer algebraic systems, have been shown to have an advantage on student learning. More research is needed to show the impact of CAS technology on math assessment student achievement. As technology develops, more research is required on these advances and how they can be utilized in the classroom to make students mathematically proficient in the digital world.