Unusual Times in Shasta County
Those conducting an election are responsible for managing it such that most thoughtful voters can have confidence in the outcome. A trustworthy election system must be robust against human error, machine error, and exploitation. The system must be comprehensible to the average voter. Analysis of Shasta County’s audit logs from the 2024 March Primary leaves more questions than answers.
An audit log is a software-generated record of user actions that occurred during an election, specifically those actions pertaining to ballot processing from scanning to final submission for counting. These events reveal the who, what, when, where, and how of ballot processing. The audit log serves as a record that can be used to replay the activities to validate other artifacts of the election process. The audit log under examination in this study was produced by a cluster of workstations running Hart Intercivic’s Verity Central application.
The California Secretary of State uses a set of guidelines called the California Voting System Standards (CVSS) to carry out the testing and certification of voting system technology. The introduction to Section 2.1.5 of the CVSS states about audit records:
“This subsection describes the context and purpose of voting system audits and sets forth specific functional requirements. Election audit trails provide the supporting documentation for verifying the accuracy of reported election results. They present a concrete, indestructible archival record of all system activity related to the vote tally, and are essential for public confidence in the accuracy of the tally, for recounts, and for evidence in the event of criminal or civil litigation.”
An auditor/investigator begins with expectations of what should or should not occur. Ultimately the two following questions form the basis of the analysis and conclusions:
What was found that was unexpected?
What was expected that wasn’t found?
Observations may be explained, in order of increasing severity, by one or more of the following causes:
Legitimate operation or activity.
Human error
Software or system error
Nefarious efforts to exploit the system.
Election Day was on March 5, 2024. Daylight Savings Time (DST) adjustment officially occurred at 2:00 am on Sunday, March 10. All four of the Verity Central workstations ceased operations between noon and 4:00 p.m. on Friday, March 8. The following Monday all workstations resumed operations between 10:00 a.m. and noon PDT, according to the log file. Except for two workstations that did not automatically update for DST the cessation of activity on Friday afternoon and its resumption on Monday morning appear consistent with approximate chronometric synchronization of the workstations.
Examination of the audit logs yielded several unexpected findings, mostly associated with timekeeping, but also raised a question about the final tally. We will now address these findings individually.
1. Precisely 100 ballots identified in the logs do not appear in the official tally. All other ballots in the official tally are justified by the number of ballots shown to have been processed in audit logs. The ballots in question belong to Shasta County District 2, in which a single contest was decided by 14 votes. It is possible that an election worker legitimately excluded these ballots from the final total, but how was that determination made? This question was raised in May of 2024, and there is still no answer.
2. Two workstations automatically adjusted for DST, while two others did not. The same workstations that did not automatically adjust are also noted in the audit log as having corrupted log files on Election Day and the day prior. What is the cause of the corrupted logs? What is the protocol used by Elections Office personnel when responding to an alert about a corrupted log file? Is there a connection between the corrupted logs and the lack of an automatic clock adjustment? At the very least we have human error that resulted in workstations with different configurations or operating parameters. While clock adjustment could be a matter of system configuration, it could also result from having the workstations connected to the internet so that they know when to adjust for DST.
3. Across 4 workstations the audit log recorded 20 separate clock-update events. These 20 are in addition to the two automatic adjustments noted above. Why 20? One can justify corrective action to adjust the two workstations that failed to update automatically, but what is the need to make 20 adjustments?
4. Clock-updated events can result in an incorrect order of events in the audit log, as seen here for a single workstation:
2024-03-12T15:40:05-07:00 Clock updated Before change: 3/12/2024 3:37:25 AM; After change: 3/12/2024 3:40:05 PM2024-03-12T22:25:39-07:00 Clock updated Before change: 3/11/2024 10:26:07 PM; After change: 3/12/2024 10:25:39 PM
When describing a clock-updated event the audit log contains the time at which the update was initiated (“before change”), and the time to which the clock was adjusted (“after change”). What I define as the “Clock Order Rule” states that each clock-updated event must have a “before change” time that is later than the “after change” time of the prior clock-updated event, if a prior event exists. The above event order doesn’t satisfy this requirement because the “before change” time of the second event is prior to the “after change” time of the first event. The order of these two events apparently violates Section d(ii) of paragraph 2.1.5.1 of the CVSS which states “The precision of the timekeeping mechanism shall be able to distinguish and properly order all audit records.”. The timestamp of record for clock update events always reflects the time “after change”. An update that reverses the time can therefore appear in the audit log prior to events that actually occurred before it. In fact, three of the four workstations used in Shasta County’s Verity Central ballot processing cluster record a series of clock-updated events in an order that is impossible based on the before- and after-change clock records. If the given order is impossible, can one reconstruct the actual order based on evidence in the log file? Attempts to reconstruct the correct order of clock-updated events using the Clock Order Rule led sometimes to an indeterminate event order, in which multiple orders satisfy the rule and, in the case of one workstation, to a condition in which no possible order of events satisfies the rule. Either the audit log used for analysis is incomplete, or we have a software or system error (or worse).
5. The clock adjustments ranged widely in magnitude, from a few seconds to over 24 hours, and seven of these adjustments were apparently made at times outside of working hours, between 9:20 pm and 3:40 am. Video surveillance footage of the workstations shows no one present at the time of these supposed late-night changes. Thirteen adjustments reversed the time, seven advanced it. As the election was held in March, one would expect the adjustments to “spring forward”, advancing the time. One workstation saw a clock adjustment of 12 hours forward, but the very next adjustment for that workstation advanced another 24 hours forward. Was the system truly 36 hours behind actual time? For how long? How did it get that way?
6. The time interval between the time “after change” of an event and the time “before change” of another event was unexpectedly predictable. That is, of the 16 clock-updated events that followed an earlier update event on the same workstation, 12 of these occurred a multiple of 60-seconds after the prior event. 10 of these occurred precisely 60 seconds after the prior event. This can only reasonably be explained as an effect of automation, even though the clock update is supposed to be manual. Additional information is required from either the Shasta Registrar of Voters or from Hart Intercivic to explain the predictable update intervals. Without such information the prospect of system exploitation remains plausible.
Because of their importance and detail audit logs are a natural target of tampering, either to obfuscate activity that occurred, or to insert activity that did not occur. In either case we don’t know what we don’t know, so we cannot assume that the tally was unaffected. Of those ballots identified in the audit logs that were NOT included in the final tally we continue to seek explanation in either human or machine activity beyond what is evident in the logs. Election staff most likely did not review the audit records to identify the evidence that the voting system itself behaved in unexpected ways, and/or system operators failed to follow procedures or best practices.
A Shasta resident who obtained the audit records from the County relied on outside technical assistance to understand their contents. Complexity leaves our elections open to many attack vectors, one or more of which may explain the unusual findings in the audit logs. A simpler voting system would result in fewer operational errors, easier understanding by the voters themselves, and would reduce the opportunity for exploitation by hidden actors. The average citizen could trust the election results because they understand the process.
California’s Secretary of State has now restricted public oversight of log files. Concerned citizens who want an in-depth look at election files are left behind closed doors and must content themselves with election results only. Our Secretary of State and our legislature have also tightened access to other election records, and they have required counties to use complex electronic voting systems instead of less expensive comprehensible alternatives. In some states, including Oregon, Washington, and Maryland, citizens who question election outcomes are placed on lists, and taxpayer money is used to maintain and monitor these lists. What is to stop the government, or its appointed agents, from silencing or suppressing these concerned citizens? The people in power are the very same who declare to the citizens who will remain in power, by virtue of systems and processes that those in power require to be used.
Only by simplifying the election process can voters take comfort in a declaration that an election “was the most secure election ever.” Without simplification the obvious response to such a declaration can only be “Secure for whom?”