current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.
up vote 0 down vote favorite

Im running a percona cluster of 3 nodes.

Yesterday I needed to throw some more memory in them, so I had to power down all nodes Now I booted up and then my 3rd Node, Gave me this error : 

140701 23:40:51 [ERROR] Slave SQL: Could not execute Update_rows event on table database_prod.cron_schedule; Can't find record in 'cron_schedule', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 206, Error_code: 1032
140701 23:40:51 [Warning] WSREP: RBR event 2 Update_rows apply warning: 120, 69137050

Whats my options regarding fixing this error. And what does it actually mean.

share|improve this question

1 Answer 1

up vote 0 down vote accepted

The 3rd node can not update record in cron_schedule because the record is missing. That could happen because some binlog events were lost when the servers crashed.

First try to skip those errors and start replication. Run this on the faulty slave:

pt-slave-restart

The script skips binlog events that return error and start replication. In some cases a slave is heavily inconsistent and replication never recovers. In that case you need to rebuild the slave. But if replication is restarted and running proceed further.

Now you need to find all inconsistencies and sync data on the master and its slaves.

First run pt-table-checksum on the master(add user/password options if needed):

pt-table-checksum -q

It will show you which tables are inconsistent i.e. have different data on the master and slaves.

Then let pt-table-sync fix the inconsistencies. Run this on the master:

pt-table-sync --execute --replicate percona.checksums --noforeign-key-checks localhost

Afterwards you may want to run pt-table-checksum again to make sure the slaves are in-sync

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.