There is no error between the internally generated clock based on fxx and the calculated value.
If there is an error, it is caused by the variation of the oscillation clock (main system clock).
Now, judging from the contents of your inquiry, what you are describing is more likely to be a timer interrupt variation rather than a clock error.
The timer interrupt variation is the variation that occurs in the interval from interrupt occurrence until the start of processing, and varies between 7 and 32 clocks depending on the instruction that is currently being executed.
(It's also based on the relationship with the other interrupts. The interval is even longer if the status is the interrupt disabled state.)