埋め込み

Embedding · Hugging Face Bitsandbytes documentation Embedding Bitsandbytes 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio Transformers Transformers.js Xet smolagents timm Search documentation main v0.49.2 v0.48.2 v0.47.0 v0.46.0 v0.45.4 v0.44.1 v0.43.3 v0.42.0 EN You are viewing v0.43.2 version. A newer version v0.49.2 is available. Join the Hugging Face community and get access to the augmented documentation experience Collaborate on models, datasets and Spaces Faster examples with accelerated inference Switch between documentation themes Sign Up to get started Embedding The embedding class is used to store and retrieve word embeddings from their indices. There are two types of embeddings in bitsandbytes, the standard PyTorch Embedding class and the StableEmbedding class. The StableEmbedding class was introduced in the 8-bit Optimizers via Block-wise Quantization paper to reduce gradient variance as a result of the non-uniform distribution of input tokens. This class is designed to support quantization. Embedding class bitsandbytes.nn. Embedding < source > ( num_embeddings : int embedding_dim : int padding_idx : Optional = None max_norm : Optional = None norm_type : float = 2.0 scale_grad_by_freq : bool = False sparse : bool = False _weight : Optional = None device : Optional = None ) Embedding class to store and retrieve word embeddings from their indices. __init__ < source > ( num_embeddings : int embedding_dim : int padding_idx : Optional = None max_norm : Optional = None norm_type : float = 2.0 scale_grad_by_freq : bool = False sparse : bool = False _weight : Optional = None device : Optional = None ) Parameters num_embeddings ( int ) — The number of unique embeddings (vocabulary size). embedding_dim ( int ) — The dimensionality of the embedding. padding_idx ( Optional[int] ) — Pads the output with zeros at the given index. max_norm ( Optional[float] ) — Renormalizes embeddings to have a maximum L2 norm. norm_type ( float , defaults to 2.0 ) — The p-norm to compute for the max_norm option. scale_grad_by_freq ( bool , defaults to False ) — Scale gradient by frequency during backpropagation. sparse ( bool , defaults to False ) — Computes dense gradients. Set to True to compute sparse gradients instead. _weight ( Optional[Tensor] ) — Pretrained embeddings. StableEmbedding class bitsandbytes.nn. StableEmbedding < source > ( num_embeddings : int embedding_dim : int padding_idx : Optional = None max_norm : Optional = None norm_type : float = 2.0 scale_grad_by_freq : bool = False sparse : bool = False _weight : Optional = None device = None dtype = None ) Parameters norm ( torch.nn.LayerNorm ) — Layer normalization applied after the embedding. Custom embedding layer designed to improve stability during training for NLP tasks by using 32-bit optimizer states. It is designed to reduce gradient variations that can result from quantization. This embedding layer is initialized with Xavier uniform initialization followed by layer normalization. Example: Copied # Initialize StableEmbedding layer with vocabulary size 1000 , embedding dimension 300 embedding_layer = StableEmbedding(num_embeddings= 1000 , embedding_dim= 300 ) # Reset embedding parameters embedding_layer.reset_parameters() # Perform a forward pass with input tensor input_tensor = torch.tensor([ 1 , 2 , 3 ]) output_embedding = embedding_layer(input_tensor) Methods: reset_parameters(): Reset embedding parameters using Xavier uniform initialization. forward(input: Tensor) -> Tensor: Forward pass through the stable embedding layer. __init__ < source > ( num_embeddings : int embedding_dim : int padding_idx : Optional = None max_norm : Optional = None norm_type : float = 2.0 scale_grad_by_freq : bool = False sparse : bool = False _weight : Optional = None device = None dtype = None ) Parameters num_embeddings ( int ) — The number of unique embeddings (vocabulary size). embedding_dim ( int ) — The dimensionality of the embedding. padding_idx ( Optional[int] ) — Pads the output with zeros at the given index. max_norm ( Optional[float] ) — Renormalizes embeddings to have a maximum L2 norm. norm_type ( float , defaults to 2.0 ) — The p-norm to compute for the max_norm option. scale_grad_by_freq ( bool , defaults to False ) — Scale gradient by frequency during backpropagation. sparse ( bool , defaults to False ) — Computes dense gradients. Set to True to compute sparse gradients instead. _weight ( Optional[Tensor] ) — Pretrained embeddings. < > Update on GitHub ← 4-bit quantizer