Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement log_loss Function and Add Comprehensive Test Coverage #28800

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Implement log_loss Function and Add Comprehensive Test Coverage #28800

wants to merge 6 commits into from

Conversation

muzakkirhussain011
Copy link
Contributor

PR Description

This PR introduces the following changes:

Feature Implementation:

  • log_loss Function:
    • Added the log_loss function to the metrics module.
    • The function computes the logistic loss (or binary cross-entropy loss) for binary classification tasks.
    • Supports clipping of predicted probabilities to avoid logarithm of zero and allows for optional sample_weight input to handle weighted samples.
    • Ensures y_true and y_pred inputs have the same shape and includes validation for edge cases such as improper input shapes or invalid values.

Test Implementation:

  • test_sklearn_log_loss:
    • Added a comprehensive test function to validate the log_loss function across various scenarios.
    • The test generates random binary classification data (y_true) and corresponding predicted probabilities (y_pred).
    • Ensures correct handling of sample_weight and validates the function's performance across different backends (e.g., TensorFlow, PyTorch).
    • The test compares the output of the Ivy log_loss implementation with the reference implementation from sklearn.metrics.log_loss to ensure correctness.

Additional Notes:

  • All methods and functions utilized in the implementation (ivy.clip, ivy.log, ivy.mean, etc.) have been cross-checked and confirmed to be available within the Ivy framework.
  • This PR enhances the robustness of the Ivy metrics module, particularly for binary classification tasks.

Testing:

  • Added test cases cover a range of input scenarios, including edge cases, to ensure that the log_loss function performs as expected under various conditions.
  • Tests have been verified across different backends to ensure compatibility and consistency.

Checklist

  • Did you add a function?
  • Did you add the tests?
  • Did you run your tests and are your tests passing?
  • Did pre-commit not fail on any check?
  • Did you follow the steps we provided?

@muzakkirhussain011
Create func_wrapper.py
d3c3de2 
func_wrapper.py is a Python module designed to streamline the integration of Hugging Face Transformers into your natural language processing (NLP) projects. It provides a set of input and output conversion wrappers to simplify the process of passing data between your custom functions and Transformers' data structures.

Input Conversion Wrappers:

inputs_to_transformers_tensors: This wrapper converts input data (text, tensors, etc.) into Transformers-compatible data structures. It is particularly useful when your custom functions expect diverse input types.
Output Conversion Wrappers:

outputs_to_pytorch_tensors: After your custom function returns data, this wrapper ensures that the output data is converted into PyTorch tensors or other appropriate formats.
Usage:

Import func_wrapper.py into your project.
Initialize a Hugging Face Transformers model and tokenizer.
Wrap your custom function with to_transformers_tensors_and_back. This wrapped function can now accept and return Transformers-compatible data.
Here's a simple example of how to use func_wrapper.py:


import torch
from transformers import BertForSequenceClassification, BertTokenizer
from ivy.functional.frontends.transformers.func_wrapper import to_transformers_tensors_and_back

# Initialize the model and tokenizer
model_name = "bert-base-uncased"
model = BertForSequenceClassification.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

# Wrap your custom function using the conversion wrappers
wrapped_function = to_transformers_tensors_and_back(your_function, model, tokenizer)

# Prepare sample input data
sample_input_text = "This is a sample input text."
sample_input_tensor = torch.rand((3, 3))

# Call your wrapped function with the sample input data
output = wrapped_function(sample_input_text, sample_input_tensor)

# The output is automatically converted to PyTorch tensors
print(output)
Please note that func_wrapper.py is still in development, and further enhancements and refinements are expected. Your feedback and contributions to improve its functionality are welcome.
@muzakkirhussain011
Merge branch 'unifyai:main' into patch-2
ea8612d
@muzakkirhussain011
feat(metrics): Implement log_loss function with sample weight support
c01177b
@muzakkirhussain011
test(metrics): Add test for log_loss function with various input scen…
c79effd 
…arios
@muzakkirhussain011
Delete ivy/functional/frontends/transformers/func_wrapper.py
e140942
@muzakkirhussain011 muzakkirhussain011 marked this pull request as draft August 9, 2024 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@muzakkirhussain011