Initial port to 5.7.9

Registered by Laurynas Biveinis

https://github.com/percona/percona-server/commit/93b7ddb5f77d9d89309a1545ff5224839e2f8222

Merge Percona Server 5.6 to MySQL 5.7.9, producing Percona Server 5.7.9

This implements blueprint
https://blueprints.launchpad.net/percona-server/+spec/initial-5-7-port,
with user-visible changes as described at
https://blueprints.launchpad.net/percona-server/+spec/initial-5.7-docs.

Non-exhaustive list of non-trivial code changes in the port, core server:
- Percona Server version now consists of only single number.
- Annotate core server, MEMORY engine, and XtraDB heap memory
  allocations for Performance Schema
  (http://dev.mysql.com/worklog/task/?id=3249,
  http://dev.mysql.com/worklog/task/?id=7777). Bump
  performance_schema_max_memory_classes so that extra PS memory
  instrumentations fit into default sizing. Bump
  performance_schema_max_rwlock_classes so that extra PS rwlock
  instrumentations still leave room for at least one of the bundled
  plugins, when loaded dynamically.
- Snapshot cloning, kill idle transaction features, thread pool, and
  INFORMATION_SCHEMA.GLOBAL_TEMPORARY_TABLES table have been updated
  to use Global_THD_manager
  (http://dev.mysql.com/worklog/task/?id=6407).
- Even though bug 57430 is fixed in upstream 5.7, keep our patch for
  the same, as it, combined with other TokuDB patches to prefer
  covering index accesses, still results in improved query plans.
- Removed debug-only Q_QUERY_EXEC_TIME binlog event status flag.
- Update thread pool for the connection handling refactoring of 5.7
  (https://dev.mysql.com/worklog/task/?id=6606).
- Added support for global backup and binlog locks for
  PERFORMANCE_SCHEMA.METADATA_LOCKS table.
- Per-statement variable feature parser part rewritten for the
  bottom-up parser (http://dev.mysql.com/worklog/task/?id=6707,
  http://dev.mysql.com/worklog/task/?id=7199,
  https://dev.mysql.com/worklog/task/?id=7203).
- Utility user feature updated for auth code refactoring
  (https://dev.mysql.com/worklog/task/?id=6960).
- Query optimizer patches updated for
  http://dev.mysql.com/worklog/task/?id=6042,
  https://dev.mysql.com/worklog/task/?id=6016, and
  http://dev.mysql.com/worklog/task/?id=7082 changes.
- Slow query log and binlog file management features have been updated
  for binlog/slow log split in the code
  (https://dev.mysql.com/worklog/task/?id=6613).
- Query cache patches now have search keys decoupled from its memory
  layout, QueryStripComments_Backup has been converted to LEX_CSTRING
  and made local to sql_cache.cc, and
  Query_cache::send_result_to_client no longer patches up THD fields
  in place for the query cache key computation.
- Updated code for diagnostics API changes because of
  http://dev.mysql.com/worklog/task/?id=6406.
- Updated code for THD field locking changes because of
  http://dev.mysql.com/worklog/task/?id=6369.
- Updated code for LOCK_thread_count split
  (http://dev.mysql.com/worklog/task/?id=7260).
- Updated code for MY_ATOMIC_MODE_RWLOCKS removal
  (http://dev.mysql.com/worklog/task/?id=7655).
- Updated code and testcase for removed INSERT DELAYED
  (http://dev.mysql.com/worklog/task/?id=6073).
- Cleaned up incomplete revert of our fix to bug 915814 / 64624.

XtraDB:
- Buffer pool mutex split updated for online buffer pool resizing by
  protecting buf_pool_t::withdraw_list with free_list_mutex
  (http://dev.mysql.com/worklog/task/?id=6117).
- InnoDB redo log archiving feature removed due to W#8845 changing the
  log format (including clobbering the "log archived up to this LSN"
  header fields) and general lack of users of the feature.
- XtraDB fake changes prefetching feature removed.
- LRU manager thread not ported forward due to upstream multi threaded
  flusher requiring a different design for independent LRU flusher
  (http://dev.mysql.com/worklog/task/?id=6642).
- innodb_empty_free_list default has been changed to LEGACY as BACKOFF
  (the 5.6 default) may currently cause deadlocks due to lack of
  LRU flusher which would be separate from flush list flusher.
- Priority mutex framework not ported forward due to it needing
  updating for the upstream policy mutex
  (https://dev.mysql.com/worklog/task/?id=6044), and our plans to
  achieve its scalability gain differently.
- Priority rwlock framework not ported forward due to it needing
  updating for the upstream SX-lock
  (http://dev.mysql.com/worklog/task/?id=6363), and due to its need
  alleviated by converting the index latch to this SX-lock
  (http://dev.mysql.com/worklog/task/?id=6326).
- Snapshot cloning updated for 5.7 transaction management changes:
  remove the assumption on RO to RW transaction upgrade that a
  transaction always sees its own changes, which is a reasonable
  assumption in general, but is not the case for snapshot
  cloning. Also, preassign transaction ids for RO transaction views
  being cloned to use in the case of later RO to RW upgrade. Do not
  port forward the transaction descriptor
  feature. (https://dev.mysql.com/worklog/task/?id=6047,
  https://dev.mysql.com/worklog/task/?id=6578). read_view_print
  function has been resurrected as ReadView::print(). Expand
  clone_consistent_snapshot with tests making sense in this context:
  repeated cloning of the same transaction, and cascaded cloning with
  different RO/RW transaction combinations.
- Update changed page tracking to handle new MTR types
  (http://dev.mysql.com/worklog/task/?id=6501,
  http://dev.mysql.com/worklog/task/?id=7142), and for different
  format/checksum algorithm
  (http://dev.mysql.com/worklog/task/?id=8845).
- Updated XtraDB corrupt table action feature to handle InnoDB native
  partitioning (where ha_innobase::m_share can be NULL more often than
  before).
- Make XtraDB track redo log synchronously in case of redo log
  resizing.
- XtraDB diagnostics were converted from fprintf to
  ib::info()/warn()/error()/fatal().
- Updated code for innodb_use_sys_malloc removal
  (https://dev.mysql.com/worklog/task/?id=7628).
- INFORMATION_SCHEMA.XTRADB_RSEG column ZIP_SIZE has been replaced
  with three columns PHYSICAL_PAGE_SIZE, LOGICAL_PAGE_SIZE, and
  IS_COMPRESSED.
- INFORMATION_SCHEMA.XTRADB_READ_VIEW column READ_VIEW_UNDO_NUMBER has
  been removed.

Plugins:
- Audit_log, query response time, and scalability_metrics plugins
  updated for 5.7 audit plugin API changes.
- Audit_log was converted to use MySQL plugin services for heap memory
  allocation.
- Plugin MTR tests were moved to plugin/tests/mtr directory from the
  main suite, effectively creating one testsuite per plugin.
- Removed HandlerSocket plugin.

Tests:
- Testsuite updates to account for: MBR temp table binlogging
  differences caused by our fix for bug 1313901 / 72475; for memory to
  disk table conversion happennning at a different point due to
  combination of dynamic VARCHAR patch for MEMORY engine and bug
  55559; for INFORMATION_SCHEMA.GLOBAL_VARIABLES and
  INFORMATION_SCHEMA.SESSION_VARIABLES being deprecated
  (http://dev.mysql.com/worklog/task/?id=6629,
  http://dev.mysql.com/worklog/task/?id=8786); InnoDB made the default
  MTR engine (http://dev.mysql.com/worklog/task/?id=6731).
- Updated partitioned InnoDB table testcases that were testing the
  partition handler to use MyISAM instead in order to avoid InnoDB
  native
  partitioning; remove --source include/have_innodb.inc from
  non-InnoDB-dependent testcases to implement WL#6731; converted
  testsuite to use include files for server restart/shutdown/startup
  more; added various new testcases.
- Since 5.7 removed MDL_map_partition::m_mutex, the scenario of bug
  1405076 disappears, and its testcase is removed, as debug sync
  points there do not exist anymore nor it's possible to re-introduce
  them.
- percona_expand_fast_index_creation modified to decrease not increase
  the VARCHAR size as the latter operation became online in 5.7.1.
- Removed tests that do not make sense after upstream remove
  --skip-innodb option, and some supporting code
  (RETURN_IF_INNODB_NOT_STARTED macro) too.
- Removed MTR Subunit patch. Jenkins reporting will be handled externally
  to the source tree.

Other:
- Mechanical code changes: removal of HAVE_ATOMIC_BUILTINS_64,
  DYNAMIC_ARRAY in C++ code, ib_logf, HAVE_purify, UNIV_INTERN,
  EXTENDED_FOR_*, QUERY_CACHE_STRIP_COMMENTS, LOCK_stats; ULONGLOGMAX
  replacement with ULLONGMAX, ibool/FALSE/TRUE with bool/false/true,
  string constification by using LEX_CSTRING in place of LEX_STRING
  (http://dev.mysql.com/worklog/task/?id=7582,
  http://dev.mysql.com/worklog/task/?id=7583).

This also takes upstream fixes for the following bugs fixed in upstream
5.7:
- https://bugs.launchpad.net/percona-server/5.7/+bug/371752 /
http://bugs.mysql.com/bug.php?id=45379 (MySQL shouldn't "guess" at the
variable names);
- https://bugs.launchpad.net/percona-server/5.7/+bug/1384656 /
http://bugs.mysql.com/bug.php?id=74584 (handle_fatal_signal (sig=11) in
handler::ha_index_or_rnd_end - sql/handler.h:2038);
- https://bugs.launchpad.net/percona-server/5.7/+bug/1385062 /
http://bugs.mysql.com/bug.php?id=74810 (ALTER TABLE crashes on corrupted
InnoDB table);
- https://bugs.launchpad.net/percona-server/5.7/+bug/1441362 /
http://bugs.mysql.com/bug.php?id=56155 (mysql_upgrade binlogs queries
that fail on master);
- https://bugs.launchpad.net/percona-server/5.7/+bug/1447527 /
http://bugs.mysql.com/bug.php?id=75368 (--ssl option should enforce SSL);
- https://bugs.launchpad.net/percona-server/5.7/+bug/1506697 /
http://bugs.mysql.com/bug.php?id=57552 (Item_row::illegal_method_call |
void Item_row::illegal_method_call(const char*): Assertion `0' failed.).

At the same time fix
- https://bugs.launchpad.net/percona-server/5.7/+bug/1192052 (the
deadlock counter in PS5.6.11 is not needed) by dropping the
innodb_deadlocks status variable;
- https://bugs.launchpad.net/percona-server/5.7/+bug/1466414 /
http://bugs.mysql.com/bug.php?id=77399 (Deadlocks missed by
INFORMATION_SCHEMA.INNODB_METRICS lock_deadlocks counter) by trivial
metric collection site adjustment.

Revert our features and bugfixes implemented in upstream 5.7:
- --innodb-buffer-pool-populate server option and --numa-interleave
  mysqld_safe.sh option due upstream --innodb-numa-interleave;
- super_read_only (http://dev.mysql.com/worklog/task/?id=6799);
- innodb_log_checksum_algorithm (75595);
- bug 1334330 / 70860;
- bug 1157037 / 68713;
- bug 1351148 / 73979;
- bug 1169494 / 68970;
- bug 1132350 / 68476;
- innodb_sched_priority_cleaner and innodb_foreground_preflush
  (http://dev.mysql.com/worklog/task/?id=7868);
- --secure-file-option with empty arg because of NULL arg feature;
- --innodb-log-block-size because of --innodb-log-write-ahead-size;
- all the XtraDB status counters that have enabled-by-default
  INNODB_METRICS counterparts;
- bug 1268729 / 71374;
- SHOW SLAVE STATUS NOLOCK
  (http://dev.mysql.com/worklog/task/?id=6402);
- mysql --syslog option (http://dev.mysql.com/worklog/task/?id=6788);
- bug 1042517 / 66560;
- bug 1132351 / 68477;
- bug 1068210 / 67504;
- bug 606811 / 54814;
- bug 1255551 / 70854;
- slow_query_log_timestamp_precision;
- slow_query_log_timestamp_always;
- bug 1197524 / 41975;
- mysqlbinlog --rewrite-db option;
- fusionIO atomic writes;
- bug 1163439 / 68845;
- bug 1279671 / 71708;
- multiple GET_LOCK per connection feature
  (http://dev.mysql.com/worklog/task/?id=1159);
- bug 1238039 / 71988;
- bug 1115048 / 60682;
- bug 1331586 / 73066;
- bug 1220544 / 70228;
- max_statement_time because of max_execution_time
  (http://dev.mysql.com/worklog/task/?id=6936);
- Online GTID migration patch (gtid_deployment_step) because of
  dynamic gtid_mode variable
  (http://dev.mysql.com/worklog/task/?id=7083);
- mysqldump --ignore-create-error because of --ignore-error;
- innodb_adaptive_hash_index_partitions (62018) because of
  innodb_adaptive_hash_index_parts;
- INFORMATION_SCHEMA.PROCESSLIST.TID because of
  PERFORMANCE_SCHEMA.THREADS.THREAD_OS_ID;
- mutex names in SHOW ENGINE INNODB MUTEX;
- bug 1396330 / 74987;
- bug 1361568 / 73736;
- bug 1380010;
- bug 1083377 / 67685;
- bug 1262500 / 54430;
- majority of bug 878404 / 45702, except for the partial pread read
  handling in my_pread;
- error number padding patch for comp_error;
- mutex names in SHOW ENGINE INNODB MUTEX output
- mysqldump --ignore-create-error option.

Whiteboard

Completed porting prerequisites:
- A complete 5.6 tree;
- file-ids resolved;
- 5.6 port review bugs fixed:
-- commented on https://bugs.launchpad.net/bugs/1099764;
- merge-regression bugs fixed (https://bugs.launchpad.net/percona-server/+bugs?field.tag=merge-regression);
-- https://bugs.launchpad.net/percona-server/+bug/1182046
-- https://bugs.launchpad.net/bugs/1182072 (Unused ha_innobase::is_corrupt() method);
-- https://bugs.launchpad.net/bugs/1182050 (Duplicate LIST_PROCESS_HOST_LEN definition);
-- https://bugs.launchpad.net/bugs/1182049 (Make innochecksum fully use InnoDB headers);
-- our mysql-test/include/linux.inc replaced with the upstream version.
- a 5.6/5.7 GCA branch (see the linked branches of this blueprint), check and revert patches to be dropped on that branch. 5.6.16 is fully merged to 5.7.4. 5.6.20 is fully merged to 5.7.5

Processed Jenkins:
- Do not need to define HAVE_purify anymore.

Processed features and bug fixes:
- show slave status nolock (https://blueprints.launchpad.net/percona-server/+spec/nonblocking-show-slave-status-5.7-syntax) (reverted in the 5.6 branch)
- mysql --syslog patch (5.7.1) (reverted in the 5.6 branch)
- bug 1042517 fix (5.7.1) (reverted in the 5.6 branch)
- bug 1132351 fix (5.7.2) (reverted in the 5.6 branch. Oracle 5.7 fix is slightly different, but that's OK, PS 5.7 will be benchmarked and the fix might be adjusted as necessary).
- bug 1068210 fix (5.7.2) (reverted in the 5.6 while keeping the testcase).
- bug 606811 / bug 1186974 (5.7.4) (reverted in the 5.6 branch)
- bug 1255551 (5.7.4) (reverted in the 5.6 branch)
- slow_query_log_timestamp_precision (5.7.2) (reverted in the 5.6 branch)
- bug 1197524 fix (5.7.3) (reverted in the 5.6 branch, testcase disabled temporarily)
- mysqlbinlog --rewrite-db (5.7.2) (reverted in the 5.6 branch, testcase disabled temporarily).
- query timeout. (reverted in the 5.6 branch, will take the 5.7 implementation).
- trx descriptors: bug 1131189 / bug 1131187 / bug 1170103. (reverted in the 5.6 branch keeping the bug 1170103 testcase).
- fusionIO atomic writes (5.7.4 revs 7517, 7518) (reverted in the 5.6 branch)
- bug 1163439 fix. (reverted in the 5.6 branch)
- bug 1182535 fix (5.7.5). (reverted in the 5.6 branch. Will likely need 5.7-specific fixing later).
- NOT REVERTED bug 1279671 fix (5.7.5). Upstream fix incomplete, does not handle autoinc_mutex.
- Kostja's GET_LOCK patch (5.7.5) (reverted in the 5.6 branch).
- NOT REVERTED bug 1218347 fix (5.7.5). Upstream fix very conservative.
- bug 1238039 fix (5.7.5-) (reverted in the 5.6 branch)
- buffer pool mutex split. Sent to Oracle at http://bugs.mysql.com/bug.php?id=75534, reverted in the 5.6 porting branch.
- innodb_log_block_checksum=crc32. Sent to Oracle at http://bugs.mysql.com/bug.php?id=75595, reverted in the 5.6 porting branch.
- HandlerSocket plugin. (reverted in the 5.6 branch).
- fake changes. (reverted in the 5.6 branch).
- http://bugs.mysql.com/bug.php?id=60682 fix (reverted in the 5.6 branch)
- http://bugs.mysql.com/bug.php?id=72475 fix (reverted in the 5.6 branch)
- http://bugs.mysql.com/bug.php?id=73066 fix (reverted in the 5.6 branch, keeping the testcase)
- Subunit MTR patch (reverted in the 5.6 branch; 5.7 Jenkins does it differently).
- bug 1299688 fix (reverted in the 5.6 branch).
- removed HAVE_purify from CMake Valgrind setup
- Move plugin-specific testcases to plugin-specific test suites
- replace ibool with bool in InnoDB patch
- bug 1220544 fix / http://bugs.mysql.com/bug.php?id=70228 (confirmed fixed in 5.7.6)
- Removed secure_file_priv extension (keep the testcase, it tests more) (upstream implemented in 5.7.6)
- Removed innodb_sched_priority_cleaner (upstream implemented in 5.7.6)
- bug 1132350 / http://bugs.mysql.com/bug.php?id=68476 fix (confirmed fixed in 5.7.6)
- bug 1169494 / http://bugs.mysql.com/bug.php?id=68970 fix (confirmed fixed in 5.7.6).
- bug 1351148 / http://bugs.mysql.com/bug.php?id=73979 fix (confirmed fixed in 5.7.6).
- bug 1157037 / http://bugs.mysql.com/bug.php?id=68713 fix (confirmed fixed in 5.7.7).
- bug 1334330 / http://bugs.mysql.com/bug.php?id=70860 fix (confirmed fixed in 5.7.8).
- dropped our log block crc32 feature (http://bugs.mysql.com/bug.php?id=75595) (upstream merged in 5.7.8)
- dropped our super_read_only feature (upstream implemented in 5.7.8)
- bug 1484599 / http://bugs.mysql.com/bug.php?id=78058 fix (confirmed fixed in 5.7.9)
- removed innodb_buffer_pool_populate and mysqld_safe numa_interleave options due to 5.7.9 innodb_numa_interleave option
- attempted to do -DWITH_UBSAN=ON testing
- removed --source include/have_innodb.inc and ENGINE=InnoDB from non-InnoDB-specific testcases
- dropped various -master.opt options that match the changed defaults
- checked whether our testcases can use include/kill_mysqld.inc and include/kill_and_restart_mysqld.inc, include/add|remove_debug_point.inc, include/execute*, no_checkpoint_*, as needed
- made plugins use MySQL plugin services API
- replaced any server fprintf(stderr) with WL #6661 changes http://dev.mysql.com/worklog/task/?id=6661 and ib::error() etc in InnoDB
- added PFS memory allocation annotations (my_malloc, ut_*nokey)
- utility user made compatible with offline_mode
- added testcases for 74900, 74584 / bug 1384656
- tested the compatibility of SHOW CREATE USER with utility user
- tested the compatibility of ACCOUNT LOCK/UNLOCK with utility user
- added testcases for 77580, 76430
- implemented and added testcase for combination of enforced and disabled storage engines

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.