CoCo AI

CoCo AI is our code completion service inside Visual Studio Code (VSCode) via the Codestral LLM (large language model). It utilizes a VSCode plugin called Continue to provide code assistance easily using keyboard shortcuts. For all below commands, Ctrl can be substituted with Cmd for Mac users.

Setup

Visual Studio Code (or Jetbrains) is required as your IDE (Integrated Development Environment) to use CoCo AI. Install the continue.dev extension from the extension marketplace in your VSCode. Continue is an open-source AI code assistant plugin that can query code snippets or even entire repositories for a chosen model. Continue provides a short introduction to their product upon installation which is easy to follow. Continue might create a directory and file “.continue/config.json” depending on whether you followed their introduction or not. If you do not have this file and directory, create it in your home directory. Open the file within an editor of your choice and write/replace the following attributes:

"models": [
{
"title": "Meta Llama 3 8B Instruct",
"model": "meta-llama-3-8b-instruct",
"apiBase": "https://chat-ai.academiccloud.de/v1",
"completionOptions": {},
"provider": "openai",
"apiKey": "<api_key>"
},
{
"title": "Meta Llama 3 70B Instruct",
"model": "meta-llama-3-70b-instruct",
"apiBase": "https://chat-ai.academiccloud.de/v1",
"completionOptions": {},
"provider": "openai",
"apiKey": "<api_key>"
},
{
"title": "Codestral-22B",
"model": "codestral-22b",
"apiBase": "https://chat-ai.academiccloud.de/v1",
"completionOptions": {
"temperature": 0.2,
"topP": 0.1
},
"provider": "openai",
"apiKey": "<api_key>"
}
],

Note that only a subset of all models available are included above. Furthermore, the openAI GPT 3.5 and 4 models are not available for API usage, and thus not available for CoCo AI. Other available models can also be included as above. Make sure to replace <api_key> with your own API key (see here for API key request). All available models are:

  • “meta-llama-3.1-8b-instruct”
  • “meta-llama-3.1-70b-instruct”
  • “llama-3.1-sauerkrautlm-70b-instruct”
  • “codestral-22b”
  • “qwen-2-72b-instruct”

To access your data stored on the cluster from VSCode, see our Configuring SSH page or this GWDG news post for instructions. This is not required for local code.

Basic configuration

Two important concepts to understand among completion options is temperature and top_P sampling.

  • temperature is a slider from 0 to 2 adjusting the creativity, with closer to 0 being more predictable and closer to 2 being more creative. It does this by expanding or flattening the probabilities of the next token (response building block).
  • top_P is a slider from 0 to 1 which adjusts the total population of probabilities considered for the next token. A top_P of 0.1 would only mean the top 10 percent of cumulative probabilities is considered. Variating top_P has a similar effect on predictability and creativity as temperature, with larger values considered to increase creativity.

Predictable results, such as for coding, require low values for both parameters, and creative results, such as for brainstorming, require high values. See the table in the current models section for value suggestions.

Our suggestion is to set the above completion options for each model according to the table in Chat AI and switch between the models based on your needs. You can also store the model multiple times with different completion options and different names to refer to, such as below.

    {
      "title": "Creative writing model",
      "model": "meta-llama-3-70b-instruct",
      "apiBase": "https://chat-ai.academiccloud.de/v1",
      "completionOptions": {
        "temperature": 0.7,
        "topP": 0.8
      },
      "provider": "openai",
      "apiKey": "<api_key>"
    }
    {
      "title": "Accurate code model",
      "model": "codestral-22b",
      "apiBase": "https://chat-ai.academiccloud.de/v1",
      "completionOptions": {
        "temperature": 0.2,
        "topP": 0.1
      },
      "provider": "openai",
      "apiKey": "<api_key>"
    },
    {
      "title": "Exploratory code model",
      "model": "codestral-22b",
      "apiBase": "https://chat-ai.academiccloud.de/v1",
      "completionOptions": {
        "temperature": 0.6,
        "topP": 0.7
      },
      "provider": "openai",
      "apiKey": "<api_key>"
    }

Another completion option to consider setting, particularly for long responses, is max_tokens. It is a value less than the context-window that specifies how many tokens may be considered per prompt, plus generated for the response to that prompt. Each model has a different context-window size (see the table in current models for sizes). Similarly, each model has a default max_tokens length. This length is optimal for most tasks, but could be changed for longer tasks, such as “Name the capital of each country in the world and one interesting aspect about it”.

The context-window wants to look at system info, chat history, its training memory, the last prompt and the previous tokens from the current response. Therefore max_tokens limits the response generation in order not to risk degenerating the quality of the response by allocating context away from the other context sources. This is why it is recommended to split a large task into smaller tasks for the requirement of smaller response generation. It could be that this is difficult or unachievable however, in which case the max_tokens would be preferred to be increased (with the risk of degradation). See API Use Cases for an example of how to change max_tokens.

Further configuration options can be found at the Continue configuration page.

Functionality

The three main abilities of the Continue plugin is to analyse code, generate code and resolve errors. A new useful ability is a tab autocomplete key shortcut that is still in Beta. All below code examples use Codestral.

Analyse code

Highlight a code snippet and press the command Ctrl+L. This will open the Continue side bar with the snippet as context for the question of your choice. From this side bar you can also access any file in your repository, as well as provide different types of context, such as entire package or language documentations, problems, git, terminal, or even your entire codebase in your chosen repository. This can be done either by pressing @ or clicking the + Add Context button. Typical functionality is provided, such as opening multiple sessions, retrieving previous sessions and toggling full screen. Models can be changed easily to, say, a model with a creative configuration, to which prompts without context can be sent, the same way the web interface of Chat AI works.

Generating code

Highlight a code snippet and press the command Ctrl+I. This will open a dropdown bar where your prompt of choice about this code can entered. When entered, this will generate further code based on the code snippet, or edit the code snippet itself. These edits can range from correcting faulty code, generating in-line documentation, renaming functions, etc. The generated code and potentially deleted code will show in a format reminiscent of a git merge conflict with Accept and Reject options. Bear in mind that there is no clear indication within VSCode whether cluster resources are available for code generation or whether code generation is not being performed actually for some other reason. We suggest to wait a short moment before trying again.

Before code generation with prompt:

Generation Prompt Generation Prompt

After code generation:

Generation Result Generation Result

Notice from the example that the code completion model is capable of more than just generating what seems like functional code. It also has all the benefits and knowledge that is expected from an LLM: semantics, grouping and linguistic reasoning. There are still limitations to this knowledge based on the date until which model training was performed, which for the most models these days is at least the year 2021.

Resolve errors

If errors have been encountered in your VS Code Problems,Output or Terminal, press the command Ctrl+Shift+R to place the errors in-context in the Continue side bar and prompt a solution for them. The results of this would explain the errors in great detail and possibly provide solution code for the identified faulty code. The same could be done manually from the Continue sidebar by providing the error as context and requesting it to be fixed.

Tab Autocomplete

Continue repetitively analyses the other code in your current file, regardless of programming language, and provide suggestions for code to fill in. To enable this function, write/replace the following attribute in your config.json file.

"tabAutocompleteModel": {
"title": "GWDG Code Completion",
"model": "codestral-22b",
"apiBase": "https://chat-ai.academiccloud.de/v1",
"completionOptions": {
"temperature": 0.2,
"topP": 0.1
},
"provider": "openai",
"apiKey": "<api_key>"
}

If the model selected is not a model particularly well-trained for code completion, Continue will prompt you accordingly. Now you should receive code suggestions from the selected model and be able to insert the suggested code simply by pressing Tab, much like the functionality of the default code suggestions VS Code provides when inspecting the packages loaded. Both suggestions could appear simultaneously, for which pressing Tab would prioritise the VS Code functionality over Continue. It may happen also that there is a conflict of hotkeys between tabAutocomplete and tab spacing, in which case the tab spacing hotkey needs to be disabled or remapped in your VS Code settings. In Settings, go to Keyboard Shortcuts and search ’tab’, then disable or replace the keybinding of the tab command. You can disable tabAutocomplete with the commands Ctrl+K Ctrl+A. Unfortunately there is no way to change the keybind of tabAutocomplete.

It is also possible to step through an autocompletion suggestion word-by-word by typing Ctrl + RightArrow. Note that Ctrl + LeftArrow does NOT undo any steps. The below code example was almost entirely generated with tabAutocomplete only from initally typing def plus, beside the need for correcting some indentation.

Partial Tab Autocompletion

Partial Tab Autocompletion Partial Tab Autocompletion

Full Tab Autocompletion

Full Tab Autocompletion Full Tab Autocompletion

More information about tabAutocomplete, including further configuration options, can be found at the Continue documentation.