Skip to content

Use --fast-ci/--slow-ci default for parallelism - #812

Open
vstinner wants to merge 1 commit into
python:mainfrom
vstinner:use_cpu_count
Open

vstinner wants to merge 1 commit into
python:mainfrom
vstinner:use_cpu_count

Conversation

@vstinner

Copy link
Copy Markdown
Member

No longer pass a pessimistic -j2 option by default. Instead, rely on --fast-ci/--slow-ci which uses -j0 by default: use os.process_cpu_count() or os.cpu_count() to decide the number of jobs to run.

No longer pass a pessimistic -j2 option by default. Instead, rely on
--fast-ci/--slow-ci which uses -j0 by default: use
os.process_cpu_count() or os.cpu_count() to decide the number of jobs
to run.
@vstinner

Copy link
Copy Markdown
Member Author

--fast-ci and --slow-ci uses -j0 by default: https://github.com/python/cpython/blob/main/Lib/test/libregrtest/cmdline.py#L480-L483

When -j0 is used, libregrtest uses (os.process_cpu_count() or 1) + 2: https://github.com/python/cpython/blob/565f76fee1fe1f3ea39aee203e2ed4887a67bc11/Lib/test/libregrtest/main.py#L539-L541

os.process_cpu_count() was added to Python 3.13 and is different than os.cpu_count(): it takes in account CPU affinity: https://docs.python.org/dev/library/os.html#os.process_cpu_count

Example:

# Run Python on two specific CPUs using taskset
$ taskset -c 5,11 python -c 'import os; print(os.cpu_count())'
12
$ taskset -c 5,11 python -c 'import os; print(os.process_cpu_count())'
2


def create_test_opts(self, branch, worker):
testopts = [*self.testFlags, *get_j_opts(worker, 2)]
testopts = [*self.testFlags, *get_j_opts(worker)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will reduce parallelism for UnixInstalledBuild which invokes regrtest directly with no --slow-ci.

Also, it may be a little risky for sanitisers to have no cap...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I didn't notice that UnixInstalledBuild doesn't use --slow-ci. That's bad! I added --slow-ci to make sure that all buildbots run the test suite the same way: pass same options.

I wrote #815 to add --slow-ci to UnixInstalledBuild and ValgrindBuild.

@encukou

encukou commented Oct 12, 2026

Copy link
Copy Markdown
Member

Should be check with worker owners if it's OK to increase the load?

@vstinner

Copy link
Copy Markdown
Member Author

Should be check with worker owners if it's OK to increase the load?

According to #810 (comment), many builders already configure the number of worker processes.

But I'm not sure how to read these tables. @hugovk: do these tables include buildbot workers which don't configure the number of worker processes? For example, I don't see angelico-debian-amd64 in these tables. If I check a angelico-debian-amd64 job: it has 2 CPUs and uses "2 worker processes". With this change, it will use 4 worker processes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants