Background Information

Project overview

Setup

Introduction

Prompt template

LCEL

Introduction to prompt engineering

Prompts

Memory

Conversation memory

Supabase

Restoring memory

Vectorstores

Supabase vector

Ingest

Retrieval

Generation

Retrieval Augmented Generation

Output Parsing

Tools

Agents

Langsmith

Farewell

( the primaryDescriptionMarkdown is long - several lines, include images, as long as you want )

In this course we <headline of what we do>. <why this technology is good> but <here's how learning it is problem>. 

In <the length of these lessons> we cover <three salient points about what we cover. Your BEST points>. Its <however long it is> and comes with <these bonus features, and reasons to buy. code, etc.>

It's taught by <author name> who <credibility, reason you should listen to them>. <social proof, if you have it yet>

<then you can list other supporting stuff you want to add, anything you want here>

-------
Example:
-------

We created a free course to create your first GraphQL server (from scratch) using Node and TypeScript. TypeScript and GraphQL are a powerful combo, but there's a lot of confusing edges if you're getting it setup the first time.

In this course we cover the basics of getting TypeScript setup with Node, how GraphQL works, and how to set up a GraphQL server using Apollo server. It's two hours long, it comes with complete code examples, and we walk through the code step-by-step.

It's taught by my friend, Shopify-engineer Hassan Djirdeh and he's integrated what he's learned using GraphQL in production as an engineer at Shopify. We've already had over 1,300 students enroll and it's been getting great reviews so far.

// ... and so on

In this course we <headline of what we do>. <why this technology is good> but <here's how learning it is problem>. 

( the summary is short - one or two lines )

Langchain.js Bootcamp

Before we ingest data, let's set up some helper functions.

import { createSupabaseClient } from '@/lib/serverUtils';
import { NextResponse } from "next/server";

export async function GET() {
  const supabase = createSupabaseClient()
  const {data: {session}} = await supabase.auth.getSession()

  if (!session?.user?.id) {
    return new Response('Unauthorized', {
      status: 401
    })
  }

  const { data, error } = await supabase.from("dataset").select()

  if (error) {
    return NextResponse.json({error})
  }

  return NextResponse.json({data})

}

export async function POST(req: Request) {
  const supabase = createSupabaseClient()
  const {data: {session}} = await supabase.auth.getSession()

  if (!session?.user?.id) {
    return new Response('Unauthorized', {
      status: 401
    })
  }

  const { tableName } = await req.json()

  const { error: createError } = await supabase.rpc('create_documents_table', { table_name: tableName })
  if (createError) {
    return NextResponse.json({createError})
  }

  const { error: indexError } = await supabase.rpc('create_hnsw_index', { table_name: tableName })
  if (indexError) {
    return NextResponse.json({indexError})
  }

  const { error: collectionsError, data } = await supabase.from("collections").insert({ collection_name: tableName, user_id: session?.user?.id }).select()

  if (indexError) {
    return NextResponse.json({collectionsError})
  }

  return NextResponse.json({message: 'success', data}, { status: 201 })

}

 SQL functions that we set up in the past section by using:

supabase.rpc(functionName, { arguments })

 table that we set up to track user Collections.

ingest

Document Loaders read Documents from file or Blob and transform them into a universal langchain Document class. After we load all the files we can treat them the same.

. We'll add more Splitters in a second, but first, let's consider a Text Splitter.

We only need to import and initialize the Loader on a file.

import { TextLoader } from "langchain/document_loaders/fs/text";

Document Loaders are often third party integrations and you may need to install a dependency depending on which ones you choose.

 to split documents into smaller chunks. This is useful to avoid confusing the model with unneccessary information and also to keep our context small.

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";

One parameter to consider is "overlap". Overlapping can help retain meaning in text while creating more chunks. Let's look at an example to see how it can be useful.

overlap

---
code: https://gitlab.com/fullstackio/books/newline-course-apps/jeffery-wang-langchain.js-bootcamp-app/-/tree/main/Code/module_04/lesson_04.04/protected/src?ref_type=heads
---

# Ingest

1.  Document Loader
2.  Text Splitter
3.  SupabaseVectorStore
4.  vectorstore.addDocuments

## Helpers

Before we ingest data, let's set up some helper functions.

Create a `api/vectorstore/collection/route.ts` file.

```ts { actualFilename=./protected/src/ingest.ts endLine=2 startLine=1 statedFilename=ingest imports url=https&#x3A;//gitlab.com/fullstackio/books/langchainjs-bootcamp/module_04/lesson_04.04/protected/src/ingest.ts }
import { createSupabaseClient } from '@/lib/serverUtils';
import { NextResponse } from "next/server";
```

Add a `GET` to recieve all user Collections.

```ts { actualFilename=./protected/src/ingest.ts endLine=23 startLine=5 statedFilename=ingest get url=https&#x3A;//gitlab.com/fullstackio/books/langchainjs-bootcamp/module_04/lesson_04.04/protected/src/ingest.ts#L5 }
export async function GET() {
  const supabase = createSupabaseClient()
  const {data: {session}} = await supabase.auth.getSession()

  if (!session?.user?.id) {
    return new Response('Unauthorized', {
      status: 401
    })
  }

  const { data, error } = await supabase.from("dataset").select()

  if (error) {
    return NextResponse.json({error})
  }

  return NextResponse.json({data})

}
```

Add a `POST` to create and index a new Collection.

```ts { actualFilename=./protected/src/ingest.ts endLine=57 startLine=27 statedFilename=ingest post url=https&#x3A;//gitlab.com/fullstackio/books/langchainjs-bootcamp/module_04/lesson_04.04/protected/src/ingest.ts#L27 }
export async function POST(req: Request) {
  const supabase = createSupabaseClient()
  const {data: {session}} = await supabase.auth.getSession()

  if (!session?.user?.id) {
    return new Response('Unauthorized', {
      status: 401
    })
  }

  const { tableName } = await req.json()

  const { error: createError } = await supabase.rpc('create_documents_table', { table_name: tableName })
  if (createError) {
    return NextResponse.json({createError})
  }

  const { error: indexError } = await supabase.rpc('create_hnsw_index', { table_name: tableName })
  if (indexError) {
    return NextResponse.json({indexError})
  }

  const { error: collectionsError, data } = await supabase.from("collections").insert({ collection_name: tableName, user_id: session?.user?.id }).select()

  if (indexError) {
    return NextResponse.json({collectionsError})
  }

  return NextResponse.json({message: 'success', data}, { status: 201 })

}
```

This route calls the `create_documents_table` and `create_hnsw_index` SQL functions that we set up in the past section by using:

```sql
supabase.rpc(functionName, { arguments })
```

Then it inserts a row to the `collections` table that we set up to track user Collections.

## Diagram

![ingest](https://d2uusema5elisf.cloudfront.net/courses/langchainjs-bootcamp/module_04/lesson_04.04/public/assets/ingest.png "ingest")

## Route

First let's create a separate `POST` route for ingesting at `/api/vectorstore/route.ts`.

## Document Loaders

Document Loaders read Documents from file or Blob and transform them into a universal langchain Document class. After we load all the files we can treat them the same.

Our application accepts `.txt`, `.pdf`, and `.csv`. We'll add more Splitters in a second, but first, let's consider a Text Splitter.

We only need to import and initialize the Loader on a file.

```ts { actualFilename=./protected/src/ingest.ts endLine=65 startLine=65 statedFilename=txt loader url=https&#x3A;//gitlab.com/fullstackio/books/langchainjs-bootcamp/module_04/lesson_04.04/protected/src/ingest.ts#L65 }
import { TextLoader } from "langchain/document_loaders/fs/text";
```

```ts { actualFilename=./protected/src/ingest.ts endLine=83 startLine=83 statedFilename=txt loader url=https&#x3A;//gitlab.com/fullstackio/books/langchainjs-bootcamp/module_04/lesson_04.04/protected/src/ingest.ts#L83 }
    const loader = loaderMap(file)
```

:::note
Document Loaders are often third party integrations and you may need to install a dependency depending on which ones you choose.
:::

## Splitting text

We can use a `TextSplitter` to split documents into smaller chunks. This is useful to avoid confusing the model with unneccessary information and also to keep our context small.

```ts { actualFilename=./protected/src/ingest.ts endLine=66 startLine=66 statedFilename=import splitter url=https&#x3A;//gitlab.com/fullstackio/books/langchainjs-bootcamp/module_04/lesson_04.04/protected/src/ingest.ts#L66 }
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
```

One parameter to consider is "overlap". Overlapping can help retain meaning in text while creating more chunks. Let's look at an example to see how it can be useful.

![chunk overlap](https://d2uusema5elisf.cloudfront.net/courses/langchainjs-bootcamp/module_04/lesson_04.04/public/assets/overlap.png "overlap")

-   **The example text is a simple phrase: "Jack saw Jill run to see him sing."**
-   **And our constraint is that we have to split the text into 3 words each.**
-   **The chunks returned are "run to see" and "Jill run to".**
-   **The question is: who is running?**