Refs: #239976

• anyOS @lramos15
• anyOS @aeschli
• anyOS @hawkticehurst

Complexity: 4

Create Issue

Create Issue

This iteration, we are adding a new proposed api that allows users to attach images and send them to the LLM from the extension.

The shape of the API is as follows:

export class LanguageModelDataPart {
       value: ChatImagePart;
       constructor(value: ChatImagePart);
}

export interface ChatImagePart {
       /**
       * The image's MIME type (e.g., "image/png", "image/jpeg").
       */
       mimeType: string;

       /**
       * The raw binary data of the image, encoded as a Uint8Array. Note: do not use base64 encoding. Maximum image size is 5MB.
       */
       data: Uint8Array;
}

Example usage:

const messages = [
       vscode.LanguageModelChatMessage2.User([new vscode.LanguageModelDataPart({ 
              data: imageData, 
              mimeType: 'image/png',
       })]),
       vscode.LanguageModelChatMessage2.User('Tell me about this image. Start each setence with "MEOW"'),
];

const chatResponse = await request.model.sendRequest(messages, {}, token);

Testing

1. Create an extension, or use use this chat sample
2. Register a command or a particpant and slash command.
3. You can send requests to the LLM via:
   

   vscode/src/vscode-dts/vscode.proposed.languageModelDataPart.d.ts

   Line 9 in 45c88a5

   sendRequest(messages: Array<LanguageModelChatMessage | LanguageModelChatMessage2>, options?: LanguageModelChatRequestOptions, token?: CancellationToken): Thenable<LanguageModelChatResponse>;

4. When calling this command, ensure that the model can answer questions about the image.

General notes

1. Image types must be png, jpg, gif, webp, or bmp.
2. The image data at the point of attachment is assumed to be non-encoded base64 binary data.
3. One easy example was using the screenshot ⬇

import screenshot from 'screenshot-desktop';
const imageBuffer = await screenshot({ format: 'png' });
const imageData = Uint8Array.from(imageBuffer);