Bug-2038705: Migrate get_releases function to PyGithub and update tests by moijes12 · Pull Request #9695 · mozilla/treeherder

moijes12 · 2026-07-17T13:19:51Z

Changes Introduced

Replace the previous REST/JSON-based get_releases implementation with a PyGithub-based implementation that returns standardized release dictionaries (including a datetime object for published_at).
Update codepaths that consume releases (collector and tasks) to handle timezone-aware datetimes and ISO-formatted dates.

Testing

Update existing changelog tests to mock PyGithub objects and add a new comprehensive unit test module (tests/utils/test_github.py) that exercises get_releases behavior across params (since, number).
Manually tested the command time python manage.py updated_changelog --days 12. It takes 23 seconds again 18 seconds taken by the old implementation.

root@00c495b4b5f8:/app# time python manage.py update_changelog --days 12
{"logging.googleapis.com/diagnostic": {"instrumentation_source": [{"name": "python", "version": "3.16.0"}]},"severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "google.cloud.logging_v2.handlers.structured_log"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 151, "file": "/usr/local/lib/python3.13/site-packages/google/cloud/logging_v2/handlers/structured_log.py", "function": "emit_instrumentation_info"}, "httpRequest": {} }
{"message": "Updating unified changelog (days=12)","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 16, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }
{"message": "Found 14 items, 0 existed and 14 where created.","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 34, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }

real    0m23.569s
user    0m4.210s
sys     0m0.344s

Fixes Bug-2038705

@gmierz @Archaeopteryx

- Migrate the `get_releases` function of the `treeherder/utils/github.py` to PyGithub. - Updates tests in tests/changelog/ to update mocking of get_releases - Add tests for get_releases in a new tests/utils/test_github.py file Fixes Bug-2038705

Archaeopteryx · 2026-07-21T13:55:22Z

Exclusively used by perfherder.

The github.get_releases now filters first by published_at and then by the number of items expected. The relevant tests have been updated too.

moijes12 · 2026-07-23T13:12:03Z

@Archaeopteryx @gmierz I have updated the filtering logic to first filter by published_at and then by number of items. The relevant tests have been updated too. I had checked locally that the call to PyGithub's get_releases() returns the GitRelease objects with the oldest releases first. So it is better to first get only those releases that were published after the cutoff date and then get a slice of them. Doing the reverse would be incorrect.

Use a timezone aware datetime object when setting the "since" parameter of changelog.

Add the timezone awareness to the changelog since parameter

Use tz aware timestamps in Changelog

moijes12 · 2026-07-28T04:01:01Z

@Archaeopteryx @gmierz

Update

I have updated the changelog filtering logic to have the "since" date use a timezone aware format. This is because the changelog table was always using a timezone aware data and the get_releases() method of PyGithub seemed to be returning a timezone aware date object.
Also, my changes have degraded the performance of python manage.py update_changelog. Below is the current data on performance. I am working on improving that and hae therefore moved the ticket back to review:

=============== Using code in PR ========================
root@c47636ca32ea:/app# time python manage.py update_changelog
{"logging.googleapis.com/diagnostic": {"instrumentation_source": [{"name": "python", "version": "3.16.0"}]},"severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "google.cloud.logging_v2.handlers.structured_log"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 151, "file": "/usr/local/lib/python3.13/site-packages/google/cloud/logging_v2/handlers/structured_log.py", "function": "emit_instrumentation_info"}, "httpRequest": {} }
{"message": "Updating unified changelog (days=1)","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 16, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }
{"message": "Found 1 items, 1 existed and 0 where created.","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 37, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }

real    0m48.794s
user    0m5.060s
sys     0m0.546s


================= Using Code in master ==========================
root@c47636ca32ea:/app# time python manage.py update_changelog
{"logging.googleapis.com/diagnostic": {"instrumentation_source": [{"name": "python", "version": "3.16.0"}]},"severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "google.cloud.logging_v2.handlers.structured_log"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 151, "file": "/usr/local/lib/python3.13/site-packages/google/cloud/logging_v2/handlers/structured_log.py", "function": "emit_instrumentation_info"}, "httpRequest": {} }
{"message": "Updating unified changelog (days=1)","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 16, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }
{"message": "Found 0 items, 0 existed and 0 where created.","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 37, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }

real    0m18.164s
user    0m4.306s
sys     0m0.452s

Improve the performance of get_releases by lazy loading of GitRelease objects.

moijes12 · 2026-07-28T05:46:55Z

Improved the performance by lazy loading of GitRelease objects. See result below

root@00c495b4b5f8:/app# time python manage.py update_changelog --days 12
{"logging.googleapis.com/diagnostic": {"instrumentation_source": [{"name": "python", "version": "3.16.0"}]},"severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "google.cloud.logging_v2.handlers.structured_log"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 151, "file": "/usr/local/lib/python3.13/site-packages/google/cloud/logging_v2/handlers/structured_log.py", "function": "emit_instrumentation_info"}, "httpRequest": {} }
{"message": "Updating unified changelog (days=12)","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 16, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }
{"message": "Found 14 items, 0 existed and 14 where created.","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "treeherder.changelog.tasks"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 34, "file": "/app/treeherder/changelog/tasks.py", "function": "update_changelog"}, "httpRequest": {} }

real    0m23.531s
user    0m4.139s
sys     0m0.408s

`github.py` now converts the PyGithub objects into native python objects so that downstream modules do not need to handle them.

- Use timezone aware date strings in test comparisons. - Fix unit tests

moijes12 · 2026-07-28T10:12:27Z

Added the below updates too:

Used timezone aware datetime throughout
Fixed unit tests

gmierz

Changes look good to me, thanks for testing locally too :)
Just one minor change below and it'll be good to land.

gmierz · 2026-07-28T15:59:09Z



-def pygithub_get_repo(owner, repo):
+def pygithub_get_repo(owner, repo) -> Repository:


I don't mind having these type hints, but given that we don't have them anywhere else in the file this makes it look odd. Can you remove them? We could add them in a separate PR if you want.

I'd recommend keeping it. I wanted to start introducing them slowly rather than adding it in one big PR which would make it difficult to review. Let me know what you think

moijes12 force-pushed the migrate-get-releases branch from 0ad02f6 to edacce4 Compare July 17, 2026 13:25

Archaeopteryx requested a review from gmierz July 21, 2026 13:55

Bug-2038705: Fixed filtering logic in get_releases

579d21e

The github.get_releases now filters first by published_at and then by the number of items expected. The relevant tests have been updated too.

Bug-2038705: Use timezone aware date in changelog

4047bf7

Use a timezone aware datetime object when setting the "since" parameter of changelog.

moijes12 marked this pull request as draft July 27, 2026 10:44

moijes12 added 2 commits July 27, 2026 11:04

Bug-2038705: Make changelog since tz aware

44e2005

Add the timezone awareness to the changelog since parameter

Bug-2038705: Use tz aware timestamps in Changelog

359109a

Use tz aware timestamps in Changelog

Bug-2038705: Lazily load the git releases

fdf6ab4

Improve the performance of get_releases by lazy loading of GitRelease objects.

moijes12 added 2 commits July 28, 2026 09:26

Bug-2038705: Use native Python dicts

4e572bd

`github.py` now converts the PyGithub objects into native python objects so that downstream modules do not need to handle them.

Bug-2038705: Use tz aware date strings in tests

b031478

- Use timezone aware date strings in test comparisons. - Fix unit tests

moijes12 marked this pull request as ready for review July 28, 2026 13:02

gmierz requested changes Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bug-2038705: Migrate get_releases function to PyGithub and update tests - #9695

Bug-2038705: Migrate get_releases function to PyGithub and update tests#9695
moijes12 wants to merge 8 commits into
mozilla:masterfrom
moijes12:migrate-get-releases

moijes12 commented Jul 17, 2026 •

edited

Loading

Uh oh!

Archaeopteryx commented Jul 21, 2026

Uh oh!

moijes12 commented Jul 23, 2026 •

edited

Loading

Uh oh!

moijes12 commented Jul 28, 2026

Uh oh!

moijes12 commented Jul 28, 2026

Uh oh!

moijes12 commented Jul 28, 2026

Uh oh!

gmierz left a comment

Uh oh!

gmierz Jul 28, 2026

Uh oh!

moijes12 Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		def pygithub_get_repo(owner, repo):
		def pygithub_get_repo(owner, repo) -> Repository:

Uh oh!

Conversation

moijes12 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Introduced

Testing

Uh oh!

Archaeopteryx commented Jul 21, 2026

Uh oh!

moijes12 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moijes12 commented Jul 28, 2026

Update

Uh oh!

moijes12 commented Jul 28, 2026

Uh oh!

moijes12 commented Jul 28, 2026

Uh oh!

gmierz left a comment

Choose a reason for hiding this comment

Uh oh!

gmierz Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

moijes12 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

moijes12 commented Jul 17, 2026 •

edited

Loading

moijes12 commented Jul 23, 2026 •

edited

Loading