To answer the question of accuracy while disconnected a typical crystal may have an accuracy of say 50ppm which equates to around 4.3 seconds of error over a day or about 180ms per hour so it could exceed your 20ms limit in under 10 minutes. A lot of the error depends on temperature and the age of the crystal so you probably could calibrate it to get much better results, but if you're talking about extended amounts of time and your temperature will vary quite a bit it may not be practical.
Now that GPS modules are so cheap that's worth looking into as an accurate time source assuming your application has a relatively clear view of the sky or you can run an antenna. Technically the GPS system is accurate to somewhere around 20ns, but most receivers I've seen with an external PPS (pulse per second) output claim an accuracy of 1ms or less.
For that to work normally you'll want a UART port to read the NMEA output from GPS that includes the UTC date and time and connect the PPS output to an interrupt line. You'll need to check the datasheet for the specific GPS module but most seem to pulse the PPS line and the NMEA sentence that follows includes the time that pulse related to, so a method I've used is when the interrupt occurs is to set the time to previous time received plus one second (because the last NMEA data is now one second old) and use that to calibrate the free-running timer.