Skip to content

Parallelize initiate and complete directory multipart upload#670

Merged
GianlucaFicarelli merged 3 commits into
mainfrom
improve_directory_upload
Jul 20, 2026
Merged

Parallelize initiate and complete directory multipart upload#670
GianlucaFicarelli merged 3 commits into
mainfrom
improve_directory_upload

Conversation

@GianlucaFicarelli

@GianlucaFicarelli GianlucaFicarelli commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Parallelize multipart upload directory initiate and complete.

Implement #669 with threads.


Speeds up directory uploads by running S3 CreateMultipartUpload and
CompleteMultipartUpload calls concurrently across files, instead of
sequentially.

Changes

app/service/asset_helpers.py (new file)

  • Extracted pure, thread-safe S3 helpers out of asset.py:
    initiate_multipart_upload, complete_asset_s3, complete_empty_asset_s3,
    generate_upload_presigned_urls, build_asset_read_with_upload_meta

app/service/asset.py

  • Directory multipart initiate: DB record creation remains sequential; S3
    initiation is now parallelized via ThreadPoolExecutor
  • Directory multipart complete: S3 completion/verification is now parallelized
    via ThreadPoolExecutor
  • Removed the now-redundant entity_asset_empty_upload_initiate and
    entity_asset_empty_upload_complete functions (logic absorbed into helpers)
  • create_entity_asset_unverified now returns Asset (ORM object) instead of
    AssetRead, avoiding redundant serialization in callers

app/utils/s3.py

  • get_s3_client now returns a thread-local client (one per storage type per
    thread) using boto3.session.Session(), avoiding concurrent access to the
    shared default session

app/utils/virtual_lab.py

  • Extracted resolve_virtual_lab_id helper to deduplicate repeated
    virtual-lab-id resolution logic across asset service functions

app/config.py

  • Added S3_DIRECTORY_UPLOAD_MAX_WORKERS setting (default: 32)

@GianlucaFicarelli GianlucaFicarelli self-assigned this Jul 17, 2026
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.26087% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
app/service/asset.py 94.87% 0 Missing and 2 partials ⚠️
Flag Coverage Δ
pytest 97.75% <98.26%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
app/config.py 97.89% <100.00%> (+0.02%) ⬆️
app/service/admin.py 90.38% <100.00%> (ø)
app/service/asset_helpers.py 100.00% <100.00%> (ø)
app/utils/s3.py 88.70% <100.00%> (+0.37%) ⬆️
app/utils/virtual_lab.py 100.00% <100.00%> (ø)
app/service/asset.py 98.93% <94.87%> (-1.07%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GianlucaFicarelli
GianlucaFicarelli force-pushed the improve_directory_upload branch 5 times, most recently from 722f129 to 3b5e157 Compare July 17, 2026 14:33
@GianlucaFicarelli
GianlucaFicarelli marked this pull request as ready for review July 17, 2026 14:43
@GianlucaFicarelli
GianlucaFicarelli force-pushed the improve_directory_upload branch from 3b5e157 to 73c3dcd Compare July 20, 2026 07:51

@mgeplf mgeplf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@GianlucaFicarelli
GianlucaFicarelli enabled auto-merge (squash) July 20, 2026 09:36
@GianlucaFicarelli
GianlucaFicarelli merged commit 14e94b9 into main Jul 20, 2026
1 check passed
@GianlucaFicarelli
GianlucaFicarelli deleted the improve_directory_upload branch July 20, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants