Skip to content

Release/2.8.0#1242

Open
chakravarthik27 wants to merge 15 commits into
mainfrom
release/2.8.0
Open

Release/2.8.0#1242
chakravarthik27 wants to merge 15 commits into
mainfrom
release/2.8.0

Conversation

@chakravarthik27

Copy link
Copy Markdown
Collaborator

This pull request introduces several improvements and fixes across the codebase, focusing on enhanced translation model support, improved HuggingFace embedding loading, bug fixes, and minor refactoring. The most significant changes include a major refactor of the translation model handler to support both encoder-decoder and causal language models, improved caching and loading for HuggingFace embeddings, and updates to dataset and prompt handling for better compatibility and correctness.

Translation Model Refactor and Improvements:

  • Refactored PretrainedModelForTranslation to support both encoder-decoder and causal language models, including improved language handling, prompt support, target language token insertion, and more robust model loading and prediction logic. Added architecture validation and clearer error messages for unsupported models. (langtest/modelhandler/transformers_modelhandler.py, langtest/errors.py) [1] [2]

  • Updated dataset references and field names for SST2 to use SetFit/sst2 and changed the expected field from sentence to text for better compatibility with HuggingFace datasets. (langtest/datahandler/datasource.py, langtest/transform/constants.py) [1] [2]

HuggingFace Embeddings and Paraphrasing:

  • Improved HuggingfaceEmbeddings to cache and reuse model and tokenizer instances, preventing redundant loads and improving performance. (langtest/embeddings/huggingface.py)

  • Refactored grammar transformation to use the paraphrase utility from langtest.utils.hf_utils instead of directly instantiating a pipeline, improving modularity and maintainability. (langtest/transform/grammar.py)

Prompt and Pipeline Handling:

  • Updated prompt template imports to use langchain_core.prompts for improved compatibility, and fixed template sorting to only include valid fields. (langtest/prompts.py) [1] [2] [3]

  • Fixed a bug in NER pipeline training and retraining by passing the tokenizer as processing_class instead of tokenizer. (langtest/pipelines/transformers/ner_pipeline.py) [1] [2]

Bug Fixes and Miscellaneous:

  • Fixed a bug in get_criteria_scores to handle non-string score values safely, and corrected the path for saving CSV files in load_csv from amgea to amega. (langtest/transform/utils.py) [1] [2]

Encoded references:
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

- Changed imports from `langchain.prompts` to `langchain_core.prompts` in `prompts.py`.
- Updated import from `langchain.evaluation.qa` to `langchain_classic.evaluation.qa` in `helpers.py`.
- Updated `poetry.lock` to reflect new versions of `langchain` and `langchain-core` to mitigate security issues.
- Updated `pyproject.toml` to require `langchain` version `1.3.13`.
… translation test to use 'German' as target language
…d enhance translation handling in PretrainedModelForTranslation
fix: update imports and dependencies to address vulnerabilities

@blidiselalin blidiselalin left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants