Additional Discussion on RAG: Emphasis on Systemic Way of Approaching/Experimenting RAG Application

[00:00 - 00:30] Yes, Julia. Yeah, so I have two questions. Yes. So I started looking at the second mini project. And it's honestly a lot and overwhelming for me because we had only one election or lecture on rock. And this is the whole project about rock and stuff that we didn't discuss. Yeah, I know I'm a developer. I can go and start reading called documentation.

[00:31 - 02:33] It mentioned multiple different tray works and away how to do it. And then one that if you can spend my day like 20 minutes, go to planning mode, how to approach with project step by step and places what we are sure, because what is a deliverable of each step. And how to pick like I understand you can use different chanting strategy and different. What tools I need to use for chanting should I just try to use different tools and do with comparison, or I can start at least and go pick one tool, for example, and structure it and try to finish with one structure and calls that because it's also talking about I've said about be it too. So again, it's just was really overwhelming for me to read. I need to have a bet. I need to have a plan. Yeah, so that's a very good question. But the reason I included all the tools was primarily to introduce that these are all the tools that you can basically use with regards to any rag application. It does not mean you have to forcefully integrate these tools like in the project. You can use your own intuition of using any tools you want. So for example, let's start from very first step. So these are all the tools. Okay. And then if you go downwards, you can see I have provided two details. And this is like any PDF from cable. It doesn't have to be like on the PDFs. And then there is another PDF that is basically, I think this is one is the aerospace one. Yeah. So basically you can use one of the PDF for the data set. Yes, Julia, do you have a question?

[02:34 - 02:48] No, with the question, sorry, with my creations that I need to have a plan. Yes . Yeah. I thought you're like you, your hands was raised. So I was okay. Okay. I will remove it. Yeah. Yeah.

[02:49 - 04:06] But on that PDF, so we actually are only assessing one PDF. We don't have to do it for PDFs of different layouts. Yes. Just one. Okay. Because this is just a starting off, right? This is the purpose of this project is to get you familiarize of the workflow that Julia mentioned because like I see there are information that are overwhelming and I kept it on purpose . The reason for that was these are some things. There are some things that we are going to use in advanced red. So I want you to like just skim through all the things that I have. But the plan over here is you just use one of the PDF. It can be like this one or any single PDF from these data. Now what you do is you use that PDF and then you use like pypd or any PDF passer library. You can go over to the notebook from a neck unit and we are using the passer PDF to extract the text from the PDF. Right now we are not worrying about the structure and stuff. We just want a simple rack to be working. And then what we do is the chunking size and the overlap and the overlap can be defined different parameters.

[04:07 - 05:41] The combination of these parameters I can show you later because I have the project ready and it's like an end to it. And I will also release these mini projects that I have created just for the sake that if people have not completed the mini project, they can look at the templates that I have on my end. And then what you do is for example, let's say you have four different combination over here. It can be the chunking size as 100 and overlap as 50, similarly for other chunking size and overlap size. So you have four combination of parameter over here. And then what you do is you choose any open EI embed ding model. So this is not the LLM model. It's an embedding model over here. So this is what I'm by junk model because to use. Yes. So these are all the. So from I will, I will talk with one of my developer to make it like more formatted friendly, but from here, maybe I 'm versus another poet. I didn't want to mention this, but maybe meanwhile he is working on better for much and you can not allow us to remove with struggle and put with like an word or comment. It's really. I just took all this stuff and put on work and was tried to format it. I see. Yeah, I do that. I can do that. That's not a problem. I can do that.

[05:42 - 06:03] So some things like in the project, I have explained on why you should do those things. The approach that I have mentioned in the project. So for example, here, the combination of chunk size and chunk overlap, like what chunking size and what rule of size you should use and why.

[06:04 - 06:46] So the reason of why is here and then the what depends on the what depends on if you are summarizing or if you are extracting the key points from the, yeah, doing quick set of interruptions and just follow up. You mentioned that what work in conflict with text in this assignment, but this is that you was talking about, for example, heads and tables. So it's layout parts of document. So for example, it's a chunk by heading paragraph or table. It's not only about the text. It's for now, what you can do is the table is the tax itself.

[06:47 - 09:59] So when I mentioned by chunking by adding paragraph or table, what I was thinking, like basically, you can choose the chunking parameter in such a way where you have a structured output of text after chunking. But for now, what you can do is you can just ignore these points if it's confusing because we are going to do these anyways in advance. So there , that's the reason there are many optional things. So what slide related this suggest to your sniff PGF, but will small amount of which particular shiny. You can use any, you can use any tool that's the purpose of exploration. You can use any tool, whichever you are comfortable on. You can use PDF plumber, or you can use PY, MUPDF, or you can use on structure. But whenever, whatever you use, you have to like then use that throughout the entire workflow of the mini project. And then basically, what you do is once you have passed the text from the PDF, you have this full combination parameters of chunking size and the whole website. And then further, what you do is you choose an embedding model. So damn meeting model. So there are three types of embedding model, the text embedding small, that you can use by API. There is one large and another one is ADA. Yeah. So these are the three embedding models. These are the three embedding models that you can basically use. Once you have those four combination of different like chunking size and overlap, you use first you use like one embedding model, and then you do embedding on those chunked text. Then you store those in vector DB, similarly for other threes for different vector DBs. And then similarly for large, and similarly for EDA. Like, now I have one question for you. Like, how many DB or new species we will have at this point, if we are using full combination of different parameters, and three embedding models. Anyone when you poor, how is it 24? Can you explain me? I was just doing the explanatory of where you multiply four times, three times two times a one to get the, I don't know, I'm not really good at the math sides of the stats. That was one of those problems. Yes. Or it's just 12. Yeah. It's 12. Okay. Yeah . It's 12 because we have four name species for each, because we have four different four parameters, four combo parameters of chunking and overlap size. And then we have three embedding models.

[10:00 - 10:59] So we will have, so three embedding models. So we will have 12 these. And then what you do is you can, you generate synthetic data set out of those 12 name species. And you generate like different question and answer. Let's say you will have three synthetic data for each chunk size. So for chunk, like each chunk. So for example, chunk equal to one, we'll have three questions, chunk two, we'll have three questions. And so how data breaks will help us to generate Quaysh directory? Yeah. So you said, sorry, I was like , I had to pick up my dog and I get distracted. So you said that we are doing chunking and talk to that we are using which three embedding young model. And we put with in that vector database. Yes. And after that, we are generating questions in that equation.

[11:00 - 11:56] Exactly. How we generate this question? Why we need a vector database in order to generate questions? You just retrieve sequential chunks from the vector DB. And then you convert those embeddings back to text. And then when it's converted back to text, you use a prompt on generate me three variation of question from this text. So how based on what we are retrieving with this data on from vector DB, just randomly, or how? It's sequential. It's sequential retrieving. Okay, I never work with vector DB, so we just get dump of all data base and that's exactly similar to pointer. Imagine it as pointer, like three three zero , then it's going to give us all the data from zero. And then you will have like bunch of attacks.

[11:57 - 12:12] And then you use a prompt to generate three questions. And then also with those three questions, we have to store the goal label, which is the zero goal label equal to zero. So now you have this synthetic goal label. Sorry, I didn't understand. Yes.

[12:13 - 13:03] What is goal label? It's the position from where you are generating your question. Okay. So in here, the position is basically starts from zero. And then you set about goal label here. Yes. Okay. And then what you do is basically you. So there are some optional things that you can do. But what you do is once you have this synthetic data set for each DB, then you use this synthetic data set to evaluate each of the namespaces or DBs to check if it's retrieving the true goal label. So for example, now you have one. Now you have a synthetic data.

[13:04 - 14:15] So you start with a question for that namespace and the retriever answers as Ch unk ID maybe it answers as Chunk ID top five is like one zero and all the other things. Then you have these different reasons for each chance. And from that, you can calculate the recall and the appreciating. So you are you have a vector database and always vector database. Your sky and Chris tables and you're creating a synthetic equation here in L alone. And you are keeping that with question was generated based on with goal label. Yes. On this channel that you are taking with questions and you're started to test your vector database asking questions and you're checking that L alone is that you are a river is that reading the correct goal label as a shade that we have with particular question. Exactly.

[14:16 - 14:28] Yes. Okay. And then you must and call this I want on the challenge to tell. Yes. No, you are that's fine. That's the purpose of today's call. You can we can go in as much detail as possible.

[14:29 - 15:47] My my motive is for everyone to understand and do all the mini projects and complete their learning curve with you. So once you have these results, you will have combination of 12 reasons because we have 12 variations with three embedding models and four combo parameters which is basically chunk size and old access. Then you choose the best one out of those 12 named species and then basically that represents that these are the best parameters for that specific media. Oh, you bet me your best chance and or overlay. Yes. Yes. So your co-coaching quota is the best. Okay. Yes. And then once you have this best result with the help of recall and MRI and precision, you attach an LLM with the retriever and using prompt engineering. For example, in here the prompt engineering will be always answer users question from the context and that context is being retrieved by the retriever.

[15:48 - 16:41] So one that's set attach. Okay. And then for and then and now at this point, we are taught shall allow them to work the database. You don't have to attach to vector DB. You have to attach an LLM with the data that is retrieved from the retriever. Let's say don't understand. So we have a data in vector database already. We put all with chunks in vector database. Yes. So how we that what so ways our retriever like where is a vector database and this then from election vector databases, one of the part of the tree with can be multiple three different searches. So how we attach LLM to vector data? You do.

[16:42 - 17:29] So you don't have to attach the LLM with vector database. So what happens in the pipeline is so now you understood at this point, right? Now we have the best. Yeah, I understand. I want to ask how to calculate with metrics. Yeah, I think in the one who's a notebook, they was doing some calculation. I don't like that clip. Yeah, you use the existing you use the existing library and then you can basically take the code snippets from the notebook or reference it from the notebooks and then it will basically help you calculate these different schools. It's not that. Yeah, so because of that, I was not asking, but I don't understand with a touch LLM we bring three were using prompt engineering. Can you like?

[17:30 - 18:54] Yes. So what are it a little bit? Yes. So now what happens is at this point, we know that. Okay, so these are some different combination of parameter and these are, this is the embedding model that works the best for this PDF. Now what we do is we create a pipeline where if you ask an equation, it directly goes to retriever. The retriever fetch is the chance based on your question and then these chunks are converted back to text. Then further when it's retrieved from when it's retrieved from the vector DV, you feed this to another LLM inference or API. Okay. And then it should be fitted with the prompt engineering technique. Okay. Okay. Yes. So we are going we are creating pipelines that when we ask a question, it's getting to the best vector database retrieve based on this question retrieve information retrieve a blob of text related to this. And after that, we are taken with as the system prompt and creating a request to LLM sending with the system and user prompt as a question as a user print and all that to get the complete text out. Okay. Okay. I understand soda. Okay.

[18:55 - 19:14] Then what you have is basically then what you do is basically measure the pit fullness relevance and ground truth with the context that is being retrieved from the vector DV. To check if LLM is actually answering from the chunk that is retrieved from the vector.

[19:15 - 19:42] Okay. Because what happens is usually it hallucinates and it just answers from its phone memory from one LLM memory. So now you measure these and how they share it. Who similar to recall MRR and precision. There is a library for all of these. You don't have to measure like all three. You can just use like fit full as our relevance to of the schools. And then what you do is basically you provide the context and the LLM generated answer.

[19:43 - 20:36] The context over here is the vector DV chunk. And then it calculates the score for you. Now what you have is you calculate that relevance and field fullness score. So imagine you have lower score in that. Now it's very common. So what is the high score and what is the low score? High score is something where basically what I usually do is in order to evaluate the high and low score. What you do is evaluate each things and randomly check by yourself on what should be the threshold of yours in this use case. So what I do is basically you check chunk and the generated answer by yourself. And then if you see the relevance or if you see the connection between those two and you see the score is let's say 30%.

[20:37 - 21:22] But you are fine with the connection. Then the 30% is your threshold. Once you let's say imagine if you have like lower school in here. Then this is the question for all the audience. If you have lower score at this point. What are some things or what are some variation or things that you would evaluate. In what stage like where are you going to go? What are you going to do if you have lower score at this stage? I will do and prompt the cost for me if I have a good matrix on a retrieval. It means that it's and it's three of us pretty good.

[21:23 - 21:35] But LLM cannot use proper returns formation. That means that prompt is need to be improve. Yes, exactly. Proms can be improved or it can be LLM that needs to be changed.

[21:36 - 24:33] It can be anything from here. Now, let me share very scratch or very first. These are the things that I'm using for a these are the different combination and parameters that I 'm using for a similarly different libraries. And then what I use in the different libraries, you have to multiply all six already noted you agent one more extra. It's yes. But so that's what I wanted to disclose. So at this point, like for example, if you had one more library then it's going to be like 24. But instead of doing that, what I did was basically calculated these things. I think I don't remember if I have time. So what I when I was doing the mini project, I will release all these after everyone has done all the mini projects. But what I was doing is basically I calculate the time. And then time was my constraint for all of these. So when I have the time, then what I do is I only choose specific things from here. Time much means how long it took you to do extraction. Yes, exactly. Time the best one. Yes. Yes. Time and time and the maximum characters because of context. So for example, if you're using GPT three point, you want to keep your budget lower. And if you 're using like GPT three point five, it will have lower context window. Now what you do is you can basically calculate the context window out of this. And then only choose that aligns with your specificity how to calculate context in the auto bus. Basically, you can just copy these different combination. You don't have to follow what I did. I will basically add more things in the advancement because I don't want to overwhelm with information. But what you do basically is you copy this data. And then you just use chat GPT to ask considering these evaluation. What are the best evaluation for using like GPT three point five three point five turbo. And then it will just give you like best parameters. But you don't need to follow that. Basically, you can just follow like one library and different combination of parameter. Now, what you have over here is you have these different reasons. I have more results like MAP and the there are many kind of reasons that you can include. But the main results are basically these three and the average time. Now you basically you can see the first combination is the best one because it has the highest result . But at the same time, the time is too much like it's 0.47. In my case, what I would basically do is I would go with lower time with a little lower accuracy for the part of any project.

[24:34 - 25:55] You don't need to think about it. You just pick the highest accuracy. And then you can address the other model after you get the highest. Okay. So now I'm only using PDF plumber for text extraction. For chunking, I'm using these different parameters for chunk ing. And then for embedding models, I think I'm only using two embedding model in this case, which is basically text embedding three small and large. And then for vector DB, I'm like managing everything on the cloud, which is turbo buffer. You can use like local vector DB, which is FAIS or you can use other vector DB's like Lance DB. There are lots of other vector DB's that are available locally. For retrieval method, I'm using VM 25 for retrieving the chunks versus send don't understand, but it will match the clock. So what does it? So there are different types of retrieval methods that you can use. Basically, one is like lexical keyword based search. Another one is like vector search. But you already saying Jim quiz parameter to when you are really I never work with just like that. Maybe I'm knowing. So you're using quiz parameters when you are getting data from vector DB. Yes.

[25:56 - 26:09] Yes. Exactly. Okay. So basically, it will have something like retriever and then you can in the bracket, you can specify method equal to VM 25 method equal to vector search like in that direction.

[26:10 - 27:46] Okay. And then what you do is basically, you can also use like combination of these two. It is basically a hybrid search. In my case, what happened was basically it under performed then vector search. And then basically now you have these different options and then different options of embeddings and different options of retrieval method. And then you basically use name species for each of the possible options. And then you calculate scores by the scores over here are basically all of these. This is similar to average score of everything. Can you show how was goal label look like? Yes. So this is the goal label or we just see you know, yes, chunk ID. So these are all the different synthetic data. What for your learning, what you can do is basically you can vector DB usually under performs on the overlap. So what you do is basically you generate synthetic data set on with the emphasis is of under the overlap, but it's too much complication.

[27:47 - 29:15] So you can just follow Q and a sequentially. And now once you have these synthetic data, you can use this synthetic data and then basically it retrieves some chunk ID. Now if the chunk ID matches like top five, for example, I have like top equal to five. So top five retrieval chunk IDs. So if top five key has this chunk ID, which is 75 something, then if it's on top, then I will have good score on recall at the rate one. If it's a second position, recall at the rate two, good score on recall at the rate two, similarly, three, four and five. But what we do is we just remove this complication and we just keep it as equal one, three and five. Okay, why we are removing position for a position to why we are doing only one stream by the reason is like the reason is the recall at the rate five will have higher score. Even if the chunk ID is at top position, it just the vice versa. So recall at the rate one will have lower score if the position of the chunk ID is at three. Oh, okay . So these scores are usually to find where most of the chunk IDs are positioned at what position.

[29:16 - 29:34] Okay. So for example, let's imagine like a retriever is retrieving this chunk ID at six. Then recall at the rate five will have lower score, recall at the rate three will have lower score, recall at one will have lower score. That means it does not fall in the top five.

[29:35 - 29:51] But if recall at the rate five has higher score, recall at the rate three has mid score and recall that the rate one has lower score. That means the chunk ID is at fourth position. Because recall that the rate three is on weeks. It's not.

[29:52 - 30:28] Yeah. So that's how the recall works. Okay. Can you show like you was mentioning cultural like different types of by being easy tools. Can you show how you use them or your friendship? Which tools? I think he was mentioning several log files and chicken, what your llamas doing. I need to see brain trust to motivate and company system performance. You mentioned. I'm not using brain trust now. I will use brain trust in the advanced.

[30:29 - 30:39] But you can basically let me introduce you to the brain trust. So this is how brain trust is a production tool. It's a good tool. Can you guys see my screen? Yes. Okay.

[30:40 - 31:15] So this is brain trust. There are these different projects that I have in here to log things from the user. Let's go into the text to SQL. So this is one of the project that I will release where I will introduce brain trust. Now, over here, this what this project does is like a user has user ask and input, which is basically what are my largest expense category. LLM answers the and then it executes that query from the database to show the data to the end user.

[31:16 - 32:37] So now what happens is I give you two things on the end user thumbs up and thumbs down. Now, if LLM is working as it's supposed to be working, the user will read it as thumbs up because it's retrieving the data that user asked, which is what are my largest category ? If it does thumbs down, then basically what happens is we have to check this thing, which is the SQL that is being generated from them. And if something is wrong in here, that means there is something wrong in the AI pipeline. So this is brain trust. Now, this is just for one application. It can be used like with, it can be used with multiple application. It can be used with react to. So for example, so in here, I think I have no feedback, but let me see if I have no. Yeah, there are no other feedbacks on thumbs up. But if it goes like thumbs down, that means there is something wrong in LLM inference or EI pipeline. And you can basically see how things work. For example, this is another locking system that I have in the text right text work asking. So in here, what I'm recording is basically when a user asks something, I'm recording the execution success. I'm recording the time.

[32:38 - 33:44] I'm recording the query complexity. What in your case, what you will have is if you're using brain trust, then this will be the row as MRR. It can be this can be the other matrix, which is basically recall and all the other metrics. So this is like an logging system. So that's now how you use brain trust is basically, it's like a library. You do people install brain trust. And then you can basically read the documentation of how you can evaluate things on real time. But it's not mandatory. This is a production tool. For example, if you have implemented EI application, in your case, the tennis EI application that we were discussing on our call, then once it's published, you would have to integrate like it's mandatory to integrate brain trust, because without logs, you won't know if the EI application that you have published to the users or to the internet is actually a good application.

[33:45 - 34:08] You mentioned that's optional. But so cohere ranker is basically, it's an optional assignment, because people were asking a lot of things about reranking and stuff in the early rag. So early rag usually doesn't have cohere ranking or any open source reranking.

[34:09 - 35:22] We removed the reranking from your notebook. So now your notebook is like updated notebook, and we transferred the reranking the advance. What happens in cohere, like, this is the cohere ranking. Basically, what you do over here is you install the same install co here, and then you can basically read the documentation on how to use it. But what happens in the coding level is, let's say you have higher recall at five. Most of your gold ID are being retrieved at chunk five, but it's not retrieved at chunk one. So in this case, now you have cluster of things that are on chunk five, but it's not for some reason, it's not being retrieved at chunk one. So what you do in this case is you use reranking. You train the reranking from your rag system such that those specific retrieved chunk IDs now are pushed to top one. So this is the case where you use a cohere ranking. Think of it as creating a simple machine learning model where it recognizes some pattern.

[35:23 - 35:45] And with the help of this pattern, it pushes your position of retrieved chunk ID at five to increase your recall at the rate one score. So on what step it is used for user, I don't understand in my pipeline, where I will use it.

[35:46 - 36:26] So this is usually used after you have this core, which is basically the score that I showed you. So once you are at this stage, which is basically, you have all these scores. Now, if we dive deep into the scores, you might see some scenario where, for example, take this combination of parameter in this case. Now you have the best score as this. So what in here is top 10 retrieved chunk ID is performing better than top one.

[36:27 - 36:38] The reason for that is the goal label are being assigned anywhere in the top 10 . So in this case, what you do is you train the reranking after you have this combination of parameter.

[36:39 - 37:04] And then you train the, you train a reranking model, and then you use the re ranking model after the retrieval. How to train the reranking model, like we are not trained in TA. So yeah, it's not, it will be covered in advanced, right? So you don't know. That's why I mentioned its option. If you are interested, some people were interested in reranking and stuff.

[37:05 - 37:34] So that's why I introduced in the RAG itself, where if they are interested, they can just choose to do reranking, but it will be covered in the advanced. But that's where the reranking is used. It's used after the retrieval or at the stage where you have, where now you have the best combination of parameter, you have the highest result, and there is no area of improvement by changing those parameters or anything. So now you use reranking.

[37:35 - 37:43] To push that evaluation, to push the evaluation on, on, on a little higher. Yes, go ahead, Kevin.

[37:44 - 38:09] Hello, Deepan. I have a question. Let's go back a little bit to when we are pretty much working with different retrievers and different embed ding models. You mentioned, I'm going to borrow your previous example. You mentioned that we retrieve chunks sequentially, and from each chunk, we start creating synthetic questions .

[38:10 - 38:36] Is there any reason why you are suggesting to pretty much retrieve one chunk at a time and then generate synthetic questions? Is there any benefit on doing that? Yes, the reason for that is whenever a user is, so this is where the question comes from. Like, why not just create synthetic data out of past text? So that's what you are asking, right?

[38:37 - 39:04] The reason for that is, if you generate synthetic question out of each chunks, it will have diversity and variation and emphasis is of question for each chunk . Now, let's say if you have just past text, you can generate like 100 questions from past text, but at that point, now you are storing like chunks. But what happens usually is the other might be able to answer correctly, but it's not targeting the right chunk.

[39:05 - 39:45] Okay. I didn't see. I was not expecting to have that kind of answer, but thanks for providing this kind of answer, because my question was from a different angle, but it's useful, really useful, your explanation now. So let me explain what angle I was, I was pretty much framing my question. What if we grab two chunks at a time, or maybe three chunks at a time sequentially still with a sliding window technique, right? And start generating synthetic questions out of it. Is there any benefit compared to one chunk at the time? Yes. So that's like drilling down on the evaluation. That's like advanced evaluation.

[39:46 - 40:53] Basically, what you are trying to do over here in this case is you are trying to generate questions out of two different chunk IDs. Yeah. And in this case, this is like an advanced electropic and advanced synthetic data generation. The one that you are mentioning is in addition to what in what we are doing with the synthetic data. What you say it is like a state step later on evaluation. Yeah. Yeah. I was curious about what are the implications and requirements for doing that? Yes. So in that, like for example, if you are like generating synthetic data from two chunk IDs, then in that case, what you do is now like you would have to specify the attachment of context at the LLM stage or the generation stage. So you don't just attach the context for one chunk ID, because now we are using like multiple chunk IDs. And it should be like a combination of chunk IDs.

[40:54 - 41:48] So for example, let's say you generated a question from zero and one. Now, what you do is on the generation side, you also have to attach all the time or most of the time to chunk IDs, which is basically called as query routing and stuff. So that are on the advanced interact. So in this case, what happens is, for example, a user is asking this question, what's this versus this? So in this case, the query will be routed to retrieve two chunks because it's a versus question. So it might have two chunks. And it retrieves like those two chunks from the vector DB. And then you attach those two chunks in the generation piece. And then we generate answer based on those two chunks.

[41:49 - 42:42] This is where you use combination of chunks on generating synthetic data in addition to generating synthetic data for single chunk. I'm safe. So at this point, once you do that, like once you do query routing and all the advanced chunk, now you are having good results, but still it's not enough. So now comes the knowledge graph, which is basically mapping down the relationship with the different chunk IDs or the position of relationship between different things. And then you use these knowledge graphs to answer based from the relationship. Okay. So my intuition is telling me that I forgot a name . Diversity is going to be affected, right, dramatically. No, we are creating the diversity , right?

[42:43 - 43:03] So we are adding like diverse questions. So basically we are right now at the stage of increasing the diversity by adding more synthetic data question with all of that. Yes. So it shouldn't be just that. It should be like both. You see, okay.

[43:04 - 43:09] These hypothetical questions helps me to build to boost my intuition. Thanks. Yes.

[43:10 - 43:23] Yeah. It's a good question. It's one of the important research. It's one of the important area that is being worked on in the reg application. It was really helpful today.