torch_to_nnef.tensor.quant.base

Classes:

Name	Description
QScalePerGroupF16	f16 scale only per group.
QScheme
QTensor	Common interface for all Compressed storage.
U8Compressor	Abstract class to add u8 compression methods.

Functions:

Name	Description
qscale_per_group_f16_min_max_calibration	Build QScalePerGroupF16 and calibrate requested float tensor.

QScalePerGroupF16

QScalePerGroupF16(group_size: int, scale: Tensor, n_bits: int)

Bases: QScheme

f16 scale only per group.

Tract aligned using negative scales.

QScheme

Bases: ABC

Methods:

Name	Description
to_device	Specific device handling.

to_device

to_device(new_device)

Specific device handling.

Each QScheme may implement support for specific device switching for internal quant/dequant (like GPU, ...) allowing faster computation

QTensor

QTensor(fp_tensor: Tensor, qscheme: QScheme, dequant_to_dtype=torch.float32, u8_compressors: Optional[List[U8Compressor]] = None)

Bases: OpaqueTensor

Common interface for all Compressed storage.

Methods:

Name	Description
to_device	Specific device handling.
write_in_file	Called at NNEF write time.

to_device

to_device(new_device)

Specific device handling.

write_in_file

write_in_file(dirpath: Union[str, Path], label: str)

Called at NNEF write time.

Each specific inference engine format should implement the file dump prefered.

U8Compressor

Abstract class to add u8 compression methods.

This can be used to

Apply bitpack elements bellow 8bit Apply classic compression algorithm

Warning !! .shape of u8_tensor compressed must be same as .shape once decompressed

Methods:

Name	Description
compress	Compress a u8 tensor (into u8).
decompress	Decompress an u8 torch tensor (into u8).
to_device	Specific device handling.

compress abstractmethod

compress(u8_tensor) -> torch.Tensor

Compress a u8 tensor (into u8).

Parameters:

Name	Type	Description	Default
u8_tensor		tensor to be compressed with dtype torch.uint8	required

Return: compressed tensor with dtype torch.uint8

decompress abstractmethod

decompress(u8_tensor) -> torch.Tensor

Decompress an u8 torch tensor (into u8).

Parameters:

Name	Type	Description	Default
u8_tensor		compressed tensor with dtype torch.uint8	required

Return: tensor decompressed with dtype torch.uint8

to_device

to_device(new_device)

Specific device handling.

Each compressor may implement support for specific device (like GPU, ...)

Allowing faster computation

qscale_per_group_f16_min_max_calibration

qscale_per_group_f16_min_max_calibration(fp_tensor, n_bits: int, group_size: int, percentile: float = 1.0) -> QScalePerGroupF16

Build QScalePerGroupF16 and calibrate requested float tensor.

Return

Tuple( QScalePerGroupF16 qscheme, torch.Tensor[uint8] )