This study aims to quantify the impacts of the COVID-19 pandemic on driver behavior as expressed by harsh accelerations (HA) measured by smartphone telematics data. Method: Over 35,5000 naturalistic driving trips were analyzed, fused with additional data sources such as: (a) Apple driving requests; (b) Oxford government response metrics; and (c) Our World in Data metrics for the COVID-19 pandemic. Machine learning algorithms were implemented on two scales: (a) a macroscopic scale involving daily analysis of aggregate driver behavior across the network with an SVM algorithm; and (b) a microscopic scale, involving trip-based analysis of driver trips with an XGBoost algorithm. SHAP values interpret the outputs of both algorithms, quantifying the influence of pandemic indicators with driver behavior and aggressiveness. Results: Macroscopic results (i.e., daily analysis) indicated that high total average speed values reduce HA rates, while this trend reverses with high driving speed. High values of Reproduction Rate, Total Cases per million people were found to reduce HA rates, while Total Fatalities per million people have little contribution on HA rates. Microscopic results (i.e., trip-based analysis) indicated that high speeding, total trip distance, and trip duration are associated with increased HA counts. Drivers perform more HAs on speeds between 30–50 km/h, while after 50 km/h, the contributions of speed lead to fewer HAs. A mild HA reduction was observed as Apple driving requests increase. Mild HA reductions also manifest when COVID-19 new daily cases and total cases per million increase as well. Drivers performed more HAs when daily deaths from COVID-19 were either relatively low (around 0–20 fatalities) or relatively high (around 110–120 fatalities), while the Stringency Index has an unclear contribution, indicating that pandemic measurements were more influential on HA counts compared to policy measures taken by the state.