musl-pgo-tests.patch (19258B)
1 From 6146295a5b8e9286ccb8f90818b764c9a0192090 Mon Sep 17 00:00:00 2001 2 From: "R. David Murray" <rdmurray@bitdance.com> 3 Date: Wed, 19 Mar 2025 13:05:09 -0400 4 Subject: [PATCH] gh-90548: Make musl test skips smarter (fixes Alpine errors) 5 (#131313) 6 7 * Make musl test skips smarter (fixes Alpine errors) 8 9 A relatively small number of tests fail when the underlying c library is 10 provided by musl. This was originally reported in bpo-46390 by 11 Christian Heimes. Among other changes, these tests were marked for 12 skipping in gh-31947/ef1327e3 as part of bpo-40280 (emscripten support), 13 but the skips were conditioned on the *platform* being emscripten (or 14 wasi, skips for which ere added in 9b50585e02). 15 16 In gh-131071 Victor Stinner added a linked_to_musl function to enable 17 skipping a test in test_math that fails under musl, like it does on a 18 number of other platforms. This check can successfully detect that 19 python is running under musl on Alpine, which was the original problem 20 report in bpo-46390. 21 22 This PR replaces Victor's solution with an enhancement to 23 platform.libc_ver that does the check more cheaply, and also gets the 24 version number. The latter is important because the math test being 25 skipped is due to a bug in musl that has been fixed, but as of this 26 checkin date has not yet been released. When it is, the test skip can 27 be fixed to check for the minimum needed version. 28 29 The enhanced version of linked_to_musl is also used to do the skips of 30 the other tests that generically fail under musl, as opposed to 31 emscripten or wasi only failures. This will allow these tests to be 32 skipped automatically on Alpine. 33 34 This PR does *not* enhance libc_ver to support emscripten and wasi, as 35 I'm not familiar with those platforms; instead it returns a version 36 triple of (0, 0, 0) for those platforms. This means the musl tests will 37 be skipped regardless of musl version, so ideally someone will add 38 support to libc_ver for these platforms. 39 40 * Platform tests and bug fixes. 41 42 In adding tests for the new platform code I found a bug in the old code: 43 if a valid version is passed for version and it is greater than the 44 version found for an so *and* there is no glibc version, then the 45 version from the argument was returned. The code changes here fix 46 that. 47 48 * Add support docs, including for some preexisting is_xxx's. 49 50 * Add news item about libc_ver enhancement. 51 52 * Prettify platform re expression using re.VERBOSE. 53 --- 54 Doc/library/test.rst | 31 ++++++++++++++- 55 Lib/platform.py | 37 +++++++++++------- 56 Lib/test/support/__init__.py | 39 +++++++++++++------ 57 Lib/test/test__locale.py | 20 ++-------- 58 Lib/test/test_locale.py | 12 ++---- 59 Lib/test/test_math.py | 3 ++ 60 Lib/test/test_os.py | 15 ++++--- 61 Lib/test/test_platform.py | 35 +++++++++++++---- 62 Lib/test/test_re.py | 12 ++---- 63 Lib/test/test_strptime.py | 5 +-- 64 Lib/test/test_support.py | 13 ++++++- 65 ...5-03-17-17-11-41.gh-issue-90548.xSPf_L.rst | 2 + 66 12 files changed, 143 insertions(+), 81 deletions(-) 67 create mode 100644 Misc/NEWS.d/next/Library/2025-03-17-17-11-41.gh-issue-90548.xSPf_L.rst 68 69 diff --git a/Doc/library/test.rst b/Doc/library/test.rst 70 index 46f8975687714b..f27cd55e7271a0 100644 71 --- a/Doc/library/test.rst 72 +++ b/Doc/library/test.rst 73 @@ -246,7 +246,27 @@ The :mod:`test.support` module defines the following constants: 74 75 .. data:: is_android 76 77 - ``True`` if the system is Android. 78 + ``True`` if ``sys.platform`` is ``android``. 79 + 80 + 81 +.. data:: is_emscripten 82 + 83 + ``True`` if ``sys.platform`` is ``emscripten``. 84 + 85 + 86 +.. data:: is_wasi 87 + 88 + ``True`` if ``sys.platform`` is ``wasi``. 89 + 90 + 91 +.. data:: is_apple_mobile 92 + 93 + ``True`` if ``sys.platform`` is ``ios``, ``tvos``, or ``watchos``. 94 + 95 + 96 +.. data:: is_apple 97 + 98 + ``True`` if ``sys.platform`` is ``darwin`` or ``is_apple_mobile`` is ``True``. 99 100 101 .. data:: unix_shell 102 @@ -831,6 +851,15 @@ The :mod:`test.support` module defines the following functions: 103 Decorator for tests that fill the address space. 104 105 106 +.. function:: linked_with_musl() 107 + 108 + Return ``False`` if there is no evidence the interperter was compiled with 109 + ``musl``, otherwise return a version triple, either ``(0, 0, 0)`` if the 110 + version is unknown, or the actual version if it is known. Intended for use 111 + in ``skip`` decorators. ``emscripten`` and ``wasi`` are assumed to be 112 + compiled with ``musl``; otherwise ``platform.libc_ver`` is checked. 113 + 114 + 115 .. function:: check_syntax_error(testcase, statement, errtext='', *, lineno=None, offset=None) 116 117 Test for syntax errors in *statement* by attempting to compile *statement*. 118 diff --git a/Lib/platform.py b/Lib/platform.py 119 index 1f6baed66d3df9..a62192589af8ff 100644 120 --- a/Lib/platform.py 121 +++ b/Lib/platform.py 122 @@ -189,22 +189,25 @@ def libc_ver(executable=None, lib='', version='', chunksize=16384): 123 # sys.executable is not set. 124 return lib, version 125 126 - libc_search = re.compile(b'(__libc_init)' 127 - b'|' 128 - b'(GLIBC_([0-9.]+))' 129 - b'|' 130 - br'(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)', re.ASCII) 131 + libc_search = re.compile(br""" 132 + (__libc_init) 133 + | (GLIBC_([0-9.]+)) 134 + | (libc(_\w+)?\.so(?:\.(\d[0-9.]*))?) 135 + | (musl-([0-9.]+)) 136 + """, 137 + re.ASCII | re.VERBOSE) 138 139 V = _comparable_version 140 # We use os.path.realpath() 141 # here to work around problems with Cygwin not being 142 # able to open symlinks for reading 143 executable = os.path.realpath(executable) 144 + ver = None 145 with open(executable, 'rb') as f: 146 binary = f.read(chunksize) 147 pos = 0 148 while pos < len(binary): 149 - if b'libc' in binary or b'GLIBC' in binary: 150 + if b'libc' in binary or b'GLIBC' in binary or b'musl' in binary: 151 m = libc_search.search(binary, pos) 152 else: 153 m = None 154 @@ -216,7 +219,7 @@ def libc_ver(executable=None, lib='', version='', chunksize=16384): 155 continue 156 if not m: 157 break 158 - libcinit, glibc, glibcversion, so, threads, soversion = [ 159 + libcinit, glibc, glibcversion, so, threads, soversion, musl, muslversion = [ 160 s.decode('latin1') if s is not None else s 161 for s in m.groups()] 162 if libcinit and not lib: 163 @@ -224,18 +227,22 @@ def libc_ver(executable=None, lib='', version='', chunksize=16384): 164 elif glibc: 165 if lib != 'glibc': 166 lib = 'glibc' 167 - version = glibcversion 168 - elif V(glibcversion) > V(version): 169 - version = glibcversion 170 + ver = glibcversion 171 + elif V(glibcversion) > V(ver): 172 + ver = glibcversion 173 elif so: 174 if lib != 'glibc': 175 lib = 'libc' 176 - if soversion and (not version or V(soversion) > V(version)): 177 - version = soversion 178 - if threads and version[-len(threads):] != threads: 179 - version = version + threads 180 + if soversion and (not ver or V(soversion) > V(ver)): 181 + ver = soversion 182 + if threads and ver[-len(threads):] != threads: 183 + ver = ver + threads 184 + elif musl: 185 + lib = 'musl' 186 + if not ver or V(muslversion) > V(ver): 187 + ver = muslversion 188 pos = m.end() 189 - return lib, version 190 + return lib, version if ver is None else ver 191 192 def _norm_version(version, build=''): 193 194 diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py 195 index cef84fd9580c37..11b2c9545a1b43 100644 196 --- a/Lib/test/test__locale.py 197 +++ b/Lib/test/test__locale.py 198 @@ -137,10 +137,7 @@ def numeric_tester(self, calc_type, calc_value, data_type, used_locale): 199 return True 200 201 @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") 202 - @unittest.skipIf( 203 - support.is_emscripten or support.is_wasi, 204 - "musl libc issue on Emscripten, bpo-46390" 205 - ) 206 + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") 207 def test_lc_numeric_nl_langinfo(self): 208 # Test nl_langinfo against known values 209 tested = False 210 @@ -158,10 +155,7 @@ def test_lc_numeric_nl_langinfo(self): 211 if not tested: 212 self.skipTest('no suitable locales') 213 214 - @unittest.skipIf( 215 - support.is_emscripten or support.is_wasi, 216 - "musl libc issue on Emscripten, bpo-46390" 217 - ) 218 + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") 219 def test_lc_numeric_localeconv(self): 220 # Test localeconv against known values 221 tested = False 222 @@ -210,10 +204,7 @@ def test_lc_numeric_basic(self): 223 224 @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") 225 @unittest.skipUnless(hasattr(locale, 'ALT_DIGITS'), "requires locale.ALT_DIGITS") 226 - @unittest.skipIf( 227 - support.is_emscripten or support.is_wasi, 228 - "musl libc issue on Emscripten, bpo-46390" 229 - ) 230 + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") 231 def test_alt_digits_nl_langinfo(self): 232 # Test nl_langinfo(ALT_DIGITS) 233 tested = False 234 @@ -245,10 +236,7 @@ def test_alt_digits_nl_langinfo(self): 235 236 @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") 237 @unittest.skipUnless(hasattr(locale, 'ERA'), "requires locale.ERA") 238 - @unittest.skipIf( 239 - support.is_emscripten or support.is_wasi, 240 - "musl libc issue on Emscripten, bpo-46390" 241 - ) 242 + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") 243 def test_era_nl_langinfo(self): 244 # Test nl_langinfo(ERA) 245 tested = False 246 diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py 247 index 798c6ad62cddd1..528ceef528114c 100644 248 --- a/Lib/test/test_locale.py 249 +++ b/Lib/test/test_locale.py 250 @@ -1,5 +1,5 @@ 251 from decimal import Decimal 252 -from test.support import verbose, is_android, is_emscripten, is_wasi 253 +from test.support import verbose, is_android, linked_to_musl, os_helper 254 from test.support.warnings_helper import check_warnings 255 from test.support.import_helper import import_fresh_module 256 from unittest import mock 257 @@ -351,10 +351,7 @@ 258 259 @unittest.skipIf(sys.platform.startswith('aix'), 260 'bpo-29972: broken test on AIX') 261 - @unittest.skipIf( 262 - is_emscripten or is_wasi, 263 - "musl libc issue on Emscripten/WASI, bpo-46390" 264 - ) 265 + @unittest.skipIf(linked_to_musl(), "musl libc issue, bpo-46390") 266 @unittest.skipIf(sys.platform.startswith("netbsd"), 267 "gh-124108: NetBSD doesn't support UTF-8 for LC_COLLATE") 268 def test_strcoll_with_diacritic(self): 269 @@ -362,10 +359,7 @@ 270 271 @unittest.skipIf(sys.platform.startswith('aix'), 272 'bpo-29972: broken test on AIX') 273 - @unittest.skipIf( 274 - is_emscripten or is_wasi, 275 - "musl libc issue on Emscripten/WASI, bpo-46390" 276 - ) 277 + @unittest.skipIf(linked_to_musl(), "musl libc issue, bpo-46390") 278 @unittest.skipIf(sys.platform.startswith("netbsd"), 279 "gh-124108: NetBSD doesn't support UTF-8 for LC_COLLATE") 280 def test_strxfrm_with_diacritic(self): 281 diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py 282 index 2649be86e5086e..b4f5dd80f55f86 100644 283 --- a/Lib/test/test_math.py 284 +++ b/Lib/test/test_math.py 285 @@ -2772,6 +2772,9 @@ def test_fma_infinities(self): 286 or (sys.platform == "android" and platform.machine() == "x86_64") 287 or support.linked_to_musl(), # gh-131032 288 f"this platform doesn't implement IEE 754-2008 properly") 289 + # gh-131032: musl is fixed but the fix is not yet released; when the fixed 290 + # version is known change this to: 291 + # or support.linked_to_musl() < (1, <m>, <p>) 292 def test_fma_zero_result(self): 293 nonnegative_finites = [0.0, 1e-300, 2.3, 1e300] 294 295 diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py 296 index 0353c2b4866c45..333179a71e3cdc 100644 297 --- a/Lib/test/test_os.py 298 +++ b/Lib/test/test_os.py 299 @@ -2393,14 +2393,11 @@ 300 self.check(os.fchown, -1, -1) 301 302 @unittest.skipUnless(hasattr(os, 'fpathconf'), 'test needs os.fpathconf()') 303 - @unittest.skipIf( 304 - support.is_emscripten or support.is_wasi, 305 - "musl libc issue on Emscripten/WASI, bpo-46390" 306 - ) 307 def test_fpathconf(self): 308 self.assertIn("PC_NAME_MAX", os.pathconf_names) 309 - self.check(os.pathconf, "PC_NAME_MAX") 310 - self.check(os.fpathconf, "PC_NAME_MAX") 311 + if not support.linked_to_musl(): 312 + self.check(os.pathconf, "PC_NAME_MAX") 313 + self.check(os.fpathconf, "PC_NAME_MAX") 314 self.check_bool(os.pathconf, "PC_NAME_MAX") 315 self.check_bool(os.fpathconf, "PC_NAME_MAX") 316 317 diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py 318 index ca73b043d31b7f..6ba630ad527f91 100644 319 --- a/Lib/test/test_platform.py 320 +++ b/Lib/test/test_platform.py 321 @@ -551,6 +551,10 @@ def test_libc_ver(self): 322 (b'GLIBC_2.9', ('glibc', '2.9')), 323 (b'libc.so.1.2.5', ('libc', '1.2.5')), 324 (b'libc_pthread.so.1.2.5', ('libc', '1.2.5_pthread')), 325 + (b'/aports/main/musl/src/musl-1.2.5', ('musl', '1.2.5')), 326 + # musl uses semver, but we accept some variations anyway: 327 + (b'/aports/main/musl/src/musl-12.5', ('musl', '12.5')), 328 + (b'/aports/main/musl/src/musl-1.2.5.7', ('musl', '1.2.5.7')), 329 (b'', ('', '')), 330 ): 331 with open(filename, 'wb') as fp: 332 @@ -562,14 +566,29 @@ def test_libc_ver(self): 333 expected) 334 335 # binary containing multiple versions: get the most recent, 336 - # make sure that 1.9 is seen as older than 1.23.4 337 - chunksize = 16384 338 - with open(filename, 'wb') as f: 339 - # test match at chunk boundary 340 - f.write(b'x'*(chunksize - 10)) 341 - f.write(b'GLIBC_1.23.4\0GLIBC_1.9\0GLIBC_1.21\0') 342 - self.assertEqual(platform.libc_ver(filename, chunksize=chunksize), 343 - ('glibc', '1.23.4')) 344 + # make sure that eg 1.9 is seen as older than 1.23.4, and that 345 + # the arguments don't count even if they are set. 346 + chunksize = 200 347 + for data, expected in ( 348 + (b'GLIBC_1.23.4\0GLIBC_1.9\0GLIBC_1.21\0', ('glibc', '1.23.4')), 349 + (b'libc.so.2.4\0libc.so.9\0libc.so.23.1\0', ('libc', '23.1')), 350 + (b'musl-1.4.1\0musl-2.1.1\0musl-2.0.1\0', ('musl', '2.1.1')), 351 + (b'no match here, so defaults are used', ('test', '100.1.0')), 352 + ): 353 + with open(filename, 'wb') as f: 354 + # test match at chunk boundary 355 + f.write(b'x'*(chunksize - 10)) 356 + f.write(data) 357 + self.assertEqual( 358 + expected, 359 + platform.libc_ver( 360 + filename, 361 + lib='test', 362 + version='100.1.0', 363 + chunksize=chunksize, 364 + ), 365 + ) 366 + 367 368 def test_android_ver(self): 369 res = platform.android_ver() 370 diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py 371 index 5538de60b2a03a..f65b4076aee2c6 100644 372 --- a/Lib/test/test_re.py 373 +++ b/Lib/test/test_re.py 374 @@ -1,6 +1,6 @@ 375 from test.support import (gc_collect, bigmemtest, _2G, 376 cpython_only, captured_stdout, 377 - check_disallow_instantiation, is_emscripten, is_wasi, 378 + check_disallow_instantiation, linked_to_musl, 379 warnings_helper, SHORT_TIMEOUT, CPUStopwatch, requires_resource) 380 import locale 381 import re 382 @@ -2172,10 +2172,7 @@ def test_bug_20998(self): 383 # with ignore case. 384 self.assertEqual(re.fullmatch('[a-c]+', 'ABC', re.I).span(), (0, 3)) 385 386 - @unittest.skipIf( 387 - is_emscripten or is_wasi, 388 - "musl libc issue on Emscripten/WASI, bpo-46390" 389 - ) 390 + @unittest.skipIf(linked_to_musl(), "musl libc issue, bpo-46390") 391 def test_locale_caching(self): 392 # Issue #22410 393 oldlocale = locale.setlocale(locale.LC_CTYPE) 394 @@ -2212,10 +2209,7 @@ def check_en_US_utf8(self): 395 self.assertIsNone(re.match(b'(?Li)\xc5', b'\xe5')) 396 self.assertIsNone(re.match(b'(?Li)\xe5', b'\xc5')) 397 398 - @unittest.skipIf( 399 - is_emscripten or is_wasi, 400 - "musl libc issue on Emscripten/WASI, bpo-46390" 401 - ) 402 + @unittest.skipIf(linked_to_musl(), "musl libc issue, bpo-46390") 403 def test_locale_compiled(self): 404 oldlocale = locale.setlocale(locale.LC_CTYPE) 405 self.addCleanup(locale.setlocale, locale.LC_CTYPE, oldlocale) 406 diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py 407 index 0d30a63ab0c140..fbc43829e22a96 100644 408 --- a/Lib/test/test_strptime.py 409 +++ b/Lib/test/test_strptime.py 410 @@ -544,10 +544,7 @@ def test_date_locale(self): 411 self.roundtrip('%x', slice(0, 3), time.localtime(now - 366*24*3600)) 412 413 # NB: Dates before 1969 do not roundtrip on many locales, including C. 414 - @unittest.skipIf( 415 - support.is_emscripten or support.is_wasi, 416 - "musl libc issue on Emscripten, bpo-46390" 417 - ) 418 + @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390") 419 @run_with_locales('LC_TIME', 'en_US', 'fr_FR', 'de_DE', 'ja_JP', 420 'eu_ES', 'ar_AE', 'my_MM', 'shn_MM') 421 def test_date_locale2(self): 422 diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py 423 index 46d796379fa212..8d5b3440d3bd30 100644 424 --- a/Lib/test/test_support.py 425 +++ b/Lib/test/test_support.py 426 @@ -746,7 +746,18 @@ def test_get_signal_name(self): 427 428 def test_linked_to_musl(self): 429 linked = support.linked_to_musl() 430 - self.assertIsInstance(linked, bool) 431 + self.assertIsNotNone(linked) 432 + if support.is_wasi or support.is_emscripten: 433 + self.assertTrue(linked) 434 + # The value is cached, so make sure it returns the same value again. 435 + self.assertIs(linked, support.linked_to_musl()) 436 + # The unlike libc, the musl version is a triple. 437 + if linked: 438 + self.assertIsInstance(linked, tuple) 439 + self.assertEqual(3, len(linked)) 440 + for v in linked: 441 + self.assertIsInstance(v, int) 442 + 443 444 # XXX -follows a list of untested API 445 # make_legacy_pyc 446 diff --git a/Misc/NEWS.d/next/Library/2025-03-17-17-11-41.gh-issue-90548.xSPf_L.rst b/Misc/NEWS.d/next/Library/2025-03-17-17-11-41.gh-issue-90548.xSPf_L.rst 447 new file mode 100644 448 index 00000000000000..88746c1866f14e 449 --- /dev/null 450 +++ b/Misc/NEWS.d/next/Library/2025-03-17-17-11-41.gh-issue-90548.xSPf_L.rst 451 @@ -0,0 +1,2 @@ 452 +:func:`platform.libc_ver` can now detect and report the version of ``musl`` 453 +on Alpine Linux.