How To Install Local Llm's In 97 Minutes (Free)

0:00In this video is all about self-hosting and AI inference. What you're going to get out of this if you stick around to the end is you are going to learn how to install local models and we have my friend Vamsi teaching the whole class. We do this a weekly inside of our community how to install local models from simple Q&As to deep dive tutorials. So, this is going to be a long one, but feel free to comment down below what are your thoughts and if you join the community you can get access to Vamsi to help scale out your local LLMs. So, let's get right into it. Okay. So, what we're going to do now is I'm going to start with my introduction, who am I and currently what I'm working on and how it is relevant to self-hosting in

0:46general, right? So, we're going to talk about a lot of theory so you can understand, um, you know, what hardwares to buy if you're interested in self-hosting and what are the different software stacks that are available to you and what are the different AI runtimes that is available to you. And this will all, uh, come together once I finish this, um, presentation, right? So, these are some of the topics that we're going to talk about, right? What are the pillars of local hosting? Why do you want even want to self-host? In what situations we are better off with, um, you know, frontier models or you're just using frontier models and we're going to talk about some basics as what is what LLM is and what quantization is and what KV cache is, you know, we're going to get into

1:31the weeds as we progress. Um, so, this is me. Um, my name is Vamsi Acharya. Um, I have 15 years of experience, um, in dealing with enterprise security. Um, I've been working with IT professional services firms and tech consulting firms and large enterprises, um, and my area of expertise has been identity and access management. If you if you don't know what identity and access management is, it's basically managing electronic identities for enterprises, right? It's all about giving right access to right people at right time. You know, the crux of it just boils down to that. Now, on regular day-to-day basis you must be using a lot

2:17of these, you know, let's say your banks, right? So, if you want to you're doing e-banking and you want to access your account and what are you that you're going to do? You're going to enter your credentials. You need to authenticate first and there will be a second factor authentication, right? And this process of authentication and authorization is something that I do for them at the IT, right? And there are different sub-disciplines within this. We have the customer identity which deals with customer facing solutions and there is workforce identity which deals with the lifecycle management of this electronic identities and if you own a business you're more aware of these use cases. Right? When an employee joins, leaves or maybe in leave

3:03of absence, how do we deal with their accesses? And then we have privileged access management that is dealing with non-human identities. It is becoming more and more relevant, um, as with especially with, um, you know, AI in picture, AI using these service accounts, non-human non-human accounts to connect to disparate systems, right? So, this is something that I do. This is my background and I think a year ago, um, I really got into AI, um, engineering in general. Um, I'm I was really fascinated by the agentic systems. Um, I honestly always wanted to start my own company, but I had certain, you know, financial and personal

3:48constraints and that made me like that made it impossible for me to go out, network with people, you know, hire people and start a company. And with AI it seems everything's impossible. I can do it from my bedroom. Right? I can just design an AI system and I can instruct it to do certain tasks so I don't have to go to the weeds of everything and I can essentially be the orchestrator which I enjoy being. So, when I first met Bow, I was working on a pet project, essentially. Um, so, let me show you what I was working on. So, this is the system I'm working on, right? So, what it what I'm going to break down what this system exactly does and how it is

4:33relevant to what you might be doing in the future, right? So, throughout the course of, um, time I've been observing these, um, AI systems and how it's been the evolution of AI systems and I noticed that So, I asked myself a question, right? What makes a particular AI service attractive to people? Um, it essentially boils down to two factors, right? Um, number one, of course, is cost, right? Everyone is very cost-sensitive. Um, you need to be able to provide these services without breaking your bank. And number two, the quality of the output, right? If you have a better quality, you're willing to pay, you know, extra buck to achieve that quality service. So, so

5:19what influences these factors, right? And again, there are two principles that influence these factors. Number one is to get a superior output, you need to have um, better context, okay? And to get to to be more efficient in terms of cost, your context window is to be um, you need to better manage your context window, okay? These are the two things that I noticed over the course of time. Um, so, what I built is a common inference and memory systems for disparate agents, AI agents, right? So, as you can see, there are multiple AI agents connecting to this common system that has come that share the same memory so that if Claude could is working on

6:05something, Gemini, open Claude, Hermes agents, no matter what they're working on, they have they all have a context context common memory module that they're feeding into and receiving from, right? And then I built a cascading inference system so this is very relevant to what you might be doing, right? So, this these are nothing but, um, just bunch of open models, right? Interacting with each other. So, what's happening here is that there is a router system. Depending on use cases, it routes to different models within the ecosystem and every time each model cascades through, that means they the basic let's say we have a situation where we you you are basically working on coding a system, right? So, what it

6:51does is it first hits a classifier which tries to determine the use case that you're working on, right? And then it just routes to a fast model that is going to do some scaffolding and it's going to pass it on to the reason and to the core, right? So, at each step each model is improving the context and summarizing it for the other one. At the end of this model resides a cloud a frontier model, but you barely have to use it because majority of the cases the local model should be able to produce the output that you're looking for, right? And and at the bottom I have this is the observability stack is basically keeping an eye on this system and giving me the metrics so I can fine-tune the system further. So, by building this

7:37common inference and memory module, um, I would essentially get similar output of what a frontier model is giving me for most of the use cases. Um, any any questions on this? I'll have a hundred, but at the end. >> [laughter] >> Okay. Yeah, maybe at the end maybe at the Q&A. Um, all right. So, how we how is it relevant to local host in general, right? So, imagine you're you're an interviewer and you're interviewing an expert and you're trying to elicit some response from the expert. Let's say you're a journalist interviewing maybe a a famous astrophysicist, right? Brian Greene, Neil deGrasse Tyson, um, and you are interviewing them but when a journalist interviews an expert, they

8:22come with an agenda, right? They're purposeful, correct? So, that's exactly when So, they're trying to elicit some kind of a response. So, they need to be very careful with how they're going to frame the response to an expert to get back the, um, the response that they're looking for, right? So, so that number one, there are three pillars of AI inference in general, not just local hosting, AI inference in general, period. Number one is prompt engineering. Prompt engineering is all about how you frame your question, right? Um, to the AI so you can get an accurate answer. So, don't if you're too vague, AI AI AI doesn't know what to answer and it can't answer. So, you have to be very specific and purposeful, just like when you when a journalist

9:07interviews a subject matter expert, right? Um, number two is context engineer. Context engineering is something I just, you know, showed you, right? Um, getting the answers from so from a a historical memory module, right? What I mean by that is when you when you ask a question to an expert, in the expert's mind he digests that question and he retrieves some kind of information from the past, right? Based on his expertise, from the long-term memory and puts it in the short-term memory and understands the context and responds, right? That's what is going on in their mind. Real quick, before we keep going, if you're watching this and you want to actually build with some of these tools, not just

9:52watch videos about them, then you're going to want to check out our community down below, Shipping School. We have a full cloud code course, a full open claw course, and four live boot camps every single week where we actually help you get set up from scratch. Like actually set this thing up, not just watch the tutorial and figure it out by yourself. And we also provide one-on-one coaching so you could book a call with me, we could share screens, and I can help you get cloud code or open claw running on your machine. That's it, no fluff. I built this community because watching YouTube's only gets you so far. We launched it just 3 days ago and we have over 55 members. You need people around you who are actually building, people who hold you accountable, and coaches who can help you when you get stuck.

10:38I'll put the link in the description down below. Get in now before the price goes up. So, essentially context engineering is something Oops, sorry guys. My mouse is too sensitive. Okay, I'll just use my keyboard. Okay, so context engineering is just that, right? Retrieving the right context um for relevant to the question. And how can you achieve it through a system called rag? Rag means retrieval augmented generation. It's basically um ingesting relevant context into what you already asked based on your historical behavior, right? The different kinds of memory I can get into more detail on it, but I don't want to get too technical up front. I just want to give like a high-level picture as of now, but if you have any questions I'm happy to answer.

11:25And the third pillar is the model itself, right? So, if you if you invite a political expert in your show and ask him something about astrophysics, he's not going to be able to answer it and vice versa, right? So, the right model for the right use case is always essential to elicit the right response. Similarly, so model has models have been fine-tuned to answer um to respond to certain use cases. Sure, the frontier models mostly can a single model can answer a lot of these use cases, but that's why they are like 1 trillion uh parameters heavy. And often when you're hosting models locally, you don't have memory to basically store such a huge humongous

12:11model. So, you need to know what model to use for what use case, right? And what is the weight of the model and what is the quantization um offering of them. So, you need to be aware of what this is and that is that's our third pillar, which is the model. And why do self-host, right? Um so, what what do you guys think? So, what do you think? I have two reasons, right? I can think of why self-hosting is essential. Any idea? My From my perspective, the reason I'm interested is uh using open claw just my, you know, cost per day was going up a lot. So, if I can do a lot of the basic work with the local model and just save Anthropic or GPT or whatever for the higher thinking things, then I can substantially reduce the cost. Right.

12:56Yes, cost is yeah, one of the component, but it may not be like exactly it may not give you a competitive edge um if you don't have the critical mass. Like for example, if you're not using like to if you're not using your model to the full extent, then you're wasting most of the capabilities, right? And you're not utilizing its memory and the as you can as you know, like these hardware can get very expensive and um most of the time it doesn't even reach the critical mass. And energy costs are not going down anyway, right? So, they're only going up. So, if you have an always-on node that's running an agent and you're not utilizing your local model, essentially can get expensive to you. You know, you know, maybe on over the long term you're basically paying more

13:42for hosting, right? Um for me though, I think two important reasons that one should self-host is because of the privacy and independence, right? If you're running a business, you cannot have your business depend on the corporate strategy of Anthropic or Google or Open AI, right? You can't let your business strategy dependent on them because they might increase prices, they can throttle the usage, there can be a lot of scenarios where you're essentially dependent on them. And the privacy is of course the most important thing. In the majority of the cases, you're sending all your data to the cloud, right? And you don't know what they're doing with your data. They might be training it, I don't know if someone else is creating you like PII information. You never know what's going

14:28on behind the scenes. And if you are in a regulatory industry that like um you know, health care or legal, you don't want your patients or client data leaking outside. So, these are the primary motivations for people to self-host basically. And of course, there are other like benefits of self-hosting like, you know, latency and then if you if you host locally and you'll have a better performance than a model being kept somewhere in the cloud. And you can even access it offline. Um this morning I was I just saw a post somewhere in the Reddit that someone just host hosted 23.6 model locally and was in a plane, was in a airplane mode and could still be able to work offline with a local model. That's all

15:14that's convenience that everyone like to have, right? And customization, you can fine-tune your model to your needs. And in order if you want to do the same level of customization with uh a frontier model, it will cost you millions and billions of dollars because they won't allow you to customize their model. And enterprises some enterprises do that, but it's very expensive in that case, right? So, other benefit is self-hosting. And of course, compliance, it's auditable, right? Most of the um data if the if your data is hosted locally, then it's easily auditable. If it's somewhere in the cloud, you don't know what's happening. And there can be situations where frontier models um definitely has an upper hand um

15:59when it comes to quality. Um of course, like they have, you know, best-in-class quality, but even though even though I think the gap is closing really fast. Um if you're following news at all recently, there is the the deeps Dixie version 4 that came out and it's very close to frontier models in terms of benchmarking. And it's very cheap. I mean, it it's a very huge model, but even if the cloud if you even if you go to the cloud providers, the model itself is cheap to use. And of course, prototyping, right? If you're prototyping um a solution, you know, local model is basically building all that is going to be an overkill. And when the load is very unpredictable. I think this this cloud services in general has an has an edge when you

16:46don't know the demand of your business. If your demand fluctuates, clouds have an edge for elastic usage. So, elasticity is a trait of cloud um hosting in general. And of course, if you don't have appetite for infrastructure, if you don't want to put extra effort, right? So, hosting locally is not exactly plug-and-play experience. It takes a lot of um what do you say? Um it it takes learning curve, right? Um so, these are the primary reasons why you may want to go with frontier model. But I think in in the long term um self-hosting is going to be beneficial and we going to discuss >> Right. we going to talk about why that is the case. So, what exactly is an LLM? So,

17:31LLM is a basically an a sophisticated algorithm that can predict the next token. And I don't know if you want me to go deeper into how the LLM works, but at the high level I'll give you a picture. So, basically when you give an LLM a prompt, it converts the text into tokens. Like it breaks down text into meaningful tokens and it converts them into vectors. Vectors are basically floating point numbers, right? And then there is uh so, the current LLMs are based on transformer architecture that was um invented by Google in 2017. So, basically what they do is they read all the tokens at once and try to predict the the next subsequent token, right?

18:17And then it will assign some kind of probability to the next token and it will pick a token based on the highest probability. Like it's going to pick at highest probable token and the loop and then there is a loop, it repeats. The cycle repeats. Okay, any any questions so far, guys? No questions. All right. There are some non-transformer ones, right? Yeah, first were non-transformer ones. So, what happened in non-transformer architecture is the the algorithm used to scan through each word and look for relevance against each word, which wasn't exactly scalable because it takes a lot of time to do that. The transformer architecture, what it did is it took the entire context and would able to um find relevance at once. So, it could

19:02able to inject the entire context at once basically and able to predict the next token instead of predicting token by looking at each word one at a time. So, next thing is quantization, right? This is this is very important especially if you're hosting locally because often times you don't have a very large um you know, you you cannot host a very large model because of limitations in RAM that you um when you're loading a model, you need a massive amounts of RAM depending on how big the model is, right? So, what it helps is because we we learned that, you know, weights of a model are nothing but floating point values. So, floating point values are decimals, right? Look at this example. So, let's

19:48say you're you're basically building a website and it has certain color palette, right? And each number of that color is represented in multi-dimensional matrix, such as this one. And it is a decimal and for the LLM runtime to inference this this values in a floating format as is with full precision, it's going to take a lot of memory, right? Instead, what we do, we round down the numbers to something like this. Let's say integers. But the outcome will be very similar, and human eye cannot discern the difference in the color palette just because we truncated few digits, okay? So, this is essentially the quantization is. So, let's say if we quantize uh

20:33let's say if we quantize a model to four-bit uh precision. Um so, like full precision models are either 16-bit or 32-bit usually, right? And if you let's say if you quantize it down to four-bit, the chances are you save 75% memory at 3 to 5% of quality loss, right? And at eight-bit, you essentially the quality loss is unmeasurable and you save 50% of storage. So, it's very valuable. So, today even the production in production systems, all the models are pretty much quantized at a very little expense. And quantization all also makes the model faster. AI think for instance much faster because we round down the numbers. Any question about quantization? Yeah, so I have a question. I I understand uh with like

21:18with colors, I can see how basically you can you can scale down and and remove a lot of digitals you know, digits and kind of end up with a color that's very similar. In the context of knowledge, right? I mean, I there's still there's still compressing the knowledge inside. So, >> Right. when you go from like Q8 [snorts] to Q6 to Q4, the size goes down a lot. My question is, what is what is actually being uh kind of removed from from the model? Is it based on kind of like how often different things interact with each other and like how often different sort of data points are what are they doing? Yeah. >> we learned what LLM is.

22:05LLM is basically a prediction engine. It predicts the next token based on the proximity of what the next token should be. So, what it does is it's basically compresses these numbers. So, the accuracy of proximity might little be um compromised. But as we seen the real world, that compromise is very small, right? Where does the precision usually matters? Majority of the cases, it doesn't matter. If you're doing like very sophisticated math or coding, that's when the precision more than anything, right? So, at the FP8 quantization, you essentially you're not even noticing any difference, and you have 50% of memory gains. That's huge. Because RAM is expensive. We all know that. Especially after today's

22:51supply chain constraints, you can feel the heat. RAM prices are so inflated, so it become more relevant as of now. Does that answer your question? Yeah, thank you. All right, let's go. So, these are different um quantization formats. The most popular one is GGUF. Um you know, if you have any questions regarding these, I can go more in detail with you. But yeah, so GGUF is the most common quantized format um for your local hosting, and it can be used with Llama, oobabooga, and LM Studio. These are the inference engines, basically the runtimes for AI inference. And there are native quantization methods. So, FP8 and MP4 are popular with Blackwell, which is

23:37Nvidia architectures. And we have MLX 4-bit is native to Apple silicon. And these are different quantization formats. GGUF is usually uh preferable in testing environments. You cannot use that in production. Um because it doesn't um support continuous batching. We we're going to dig more into that. So, KV cache, what is Yeah. Sorry, I'll be quick. Um I've been playing with um LM Studio and oobabooga. And I what I had read was that for Apple silicon, you needed to get GGUF or MLX. I downloaded a ton of GGUF models, and it kept saying model does not load. Is that cuz I'm on a Mac? I'm download I should not be loading GGUF? I don't think so. GGUF loads on every platform that I can think of. Cuz I'm a 32 gig MacBook Pro M1, and

24:23even like uh small models are like one they're really small like TTS models are not loading. I just Every time I try to load a model, maybe get a gamma worked. Um you know, I find >> does it say exactly? 2.5 I think worked. Model failed to load. I download it, and then I go to load the model to start a chat, and it just says model failed to load. No idea why. So, you said you said you're using LM Studio? Yeah. Okay. I'm I'm I'm loading models on the oobabooga. Okay. All right. Yeah, I mean I I I can't see why it shouldn't work. There might be some reason. Um I mean, every model is different. You have to look at whether you downloaded the right model for your architecture for your hardware precisely. I mean, there could be a lot of factors. Um yeah, next thing is basically um it's a KV cache. Um this is

25:10like short-term memory for LLMs. Um LLM, as you know, right? So, it goes through loops while during inference, and it needs to have some kind of context at a runtime, so it doesn't have to regenerate every token each time. So, that's why we have KV cache to offer that context to LLM runtime. Um and KV KV stands for key value pairs. So, this is a vectors. Um this is a cache that LLM uses to retrieve tokens real time um during the inference. And KV um the reason why I wanted to talk about this is because of the memory overhead for hosting a model in your local system, right? So, everything

25:55every model requires vast amount of memory, right? For you to host um for you to be able to host it locally. But how do you know which model is right for your hardware limitations, right? Considering your hardware limitations, what size model should you be hosting, right? So, there is roughly a math that you can do to see whether your hardware supports that model or able to load that model, right? So, I'll give you an example, right? Let's say you are you want to host a full precision 16-bit model into your memory, and you have 70 billion parameter model. And by definition, 16 16-bit is two bytes. So, you just multiply 70 by two, that gives

26:40140 gig of RAM. That is usually the math that you need to make in your head to see if you can even um if you if your hardware can even host that model. So, the formula is simple. It's the um it's the precision bit byte times your number of parameters. So, I think I must have heard um formula somewhere. Apparently not. Okay. So, it's number of param parameters times number of bytes per each parameter. So, 16-bit is two bytes, right? So, you just multiply it. So, let's say if you're if if you want to use a quantized model, right? That will be eight-bit model, and that is one byte. So, 70 billion parameter model would take 70 gigabytes in both your memory and storage plus additional KV

27:27cache overhead. So, KV cache will have some certain overhead like two to four gigabytes. So, let's say if you want to host a 70 gigabyte model um at FP8 like quantized eight-bit quantization, you need at least 70 gigabytes plus maybe you would take two gigabytes KV cache overhead. That will that will bring it to 70 gig gigabyte 72 gigabytes of memory at least you should have to be able to host that model. Does that make sense? Well, so one of the questions that I had about this [snorts] when I was trying to set up oobabooga with uh OpenClaw, there was something about like OpenClaw needing at least 64 uh 64K context

28:12window. And then oobabooga also kind of has a in the graphical interface, it has a slider for context, and then the model is going to have some maximum context. And I got like I don't necessarily like if I'm trying to like there are some models that are 27 that 27B, so like what 27 billion per 27 Correct. 27 billion. Yeah, that's a practical. That's what I was told, yeah. So, like 27B Q4, and I'm like I was trying to figure out how much context to give it because it seems like your math is on 32,000. It seems like OpenClaw requires at least 64. So, that's double that, right? Or not? So, the context context window is basically defined by the the the

28:58creators of the model. They tell you how much context window that you need to be used for that particular model. Context window, so the max tokens is the limit. Sorry, it's not the limit. The context window itself is the limit of how many tokens that you can provide you through single prompt to that model. So, that is the limit. And max tokens is something you set for your settings. So, you can fine-tune it. You can set it to maximum limit, or you can set it to whatever is comfortable to you, right? So, so you can ask like if if you have a first grader, you can't ask any questions that you would ask a 10th grader to, right? They wouldn't know because they're not trained on it. So, each model is trained at certain context window, trained with certain context window. And the the

29:44model provider will usually give you what that context window is. But doesn't it doesn't it increase does does the maximum size increase the tokens going to and from? Meaning like it would slow down your own machine or it would cost you more if you're using host hosted LLM's? Uh it won't cost you I mean of course I mean if you have more context it's going to cost you more in terms of compute and energy but um but I I'm not sure I understand exactly your cost. If you increase the size of the context window Mhm. on a local model does that slow down the response? It depends on how much RAM you have and what is your firepower. It can it can

30:30slow down yes. And then for for cloud models does that increase the cost? For cloud models yes yes as I mentioned initially context window determines the the cost of using your models because that's how the you know the the the cost is priced per token processing in both inbound and outbound. Right so that would increase the number of tokens right when that they're going >> Right if you have large context if you have very large context window that will increase the number of tokens and that means that that will also increase your cost of inference in general. Yeah thanks. Yeah so there's both right there's input and output right so if you input a lot of context window necessary it doesn't necessarily mean that you will have a large output but you you can

31:17basically fine tune the model to have smaller context windows both sides. That might reduce your cost and that's where the prompt engineering and context engineering comes into picture and how well you do that is is going to determine your cost and usually you have more leeway with local models than context than 28 months. Um this is VLLM so let me go back to talk about it later. So yeah so um the inference itself has four steps right inference process AI inference process itself has four steps. Number one is loading serving prefill and decode okay. So loading is loading the model into the memory it happens the very first time basically prepares your model needs to be ready. Serving is

32:03basically making the LLM model available to the you know AI runtime okay and prefill and decode is probably the most important steps in AI inference. Prefill is where the actual processing of your prompt happens right. So this is this this particular step is very um compute heavy. So you need to have very strong GPU to be able to um perform this action faster right. So typically you you must have heard about flops floating point operations. This determines how fast the computer is going to happen in your hardware. So this is part of the prefill process. So here it it really doesn't matter in this particular step it doesn't matter how much memory you have. In this particular

32:49step what matters is um how fast how strong is your chip. So this particular step determines the time to first token. So how soon it processed your prompt and delivered first token is what matters. And the decode is memory intense um step. So it takes so once prefill basically takes your entire prompt and generates entire dumps entire KV cache okay into the cache and decode takes that cache and generates tokens out of it right. So it reads the weight and the entire KV KV cache and computes the token. So here the compute is trivial the compute is like it's very trivial. It doesn't matter how strong your GPU is what matters is the the

33:36memory bandwidth. So how soon can your hardware retrieve the tokens from GPU and serve it to the memory that is the memory bandwidth. And this particular step determines tokens per second. So how many tokens it can generate okay. So in an ideal production setup Nvidia is undisputed king because Nvidia has arguably the best GPUs available for AI inference because of its speed okay. And in the production sense it also has a very high memory like if you're using a workstations production grade workstations they also have very high memory bandwidth. So together um it will it will basically generate tokens at faster pace. As you can see this is an example of different hardware that is

34:22available right. So Nvidia H100 is production grade workstation um this this is bandwidth is 3350 gigabytes per second this is bytes not bits okay. And compare that with your Mac Studio M3 Ultra it's 819 gigabytes per second. This is very good for a consumer grade product. However it's still very slow compared to you can see a workstation for Mac OS. So one of the popular hardwares for local inference is RTX 406000 um as you can see the memory bandwidth is about like 1800 gigabytes per second. So it can the inference is much faster in this than on Mac Studio. But this is a standalone GPU it doesn't come with the entire system and it costs you around $10,000. Um another best

35:08alternative in my sense for local hosting is in my opinion is Nvidia DGX Spark it's it's currently for 128 gigabyte of RAM um it will cost you close to 3500 to 4000 dollars which I think is pretty uh good deal for personal computing personal AI inference. And you already put the uh 70 put the M5 Max 128 gigabytes in there. There's no M5 Max here yet I I don't know what the bandwidth on that yet. Do you do you know that? I I think the the M5 Max M5 Max is about 612 um but it's only it's only on laptops right now. You don't have uh Mac Studio You don't have a studio. Correct M5 Ultra is

35:55yet to come and and I think that is going to be game changing in my opinion because Mac also have higher memory um capacity compared to Nvidia. You see this Nvidia H100 the production grade system has only 80 gigabyte of RAM. You have to basically cluster multiple GPUs to be able to host a bigger model. Yeah the the predictions that I've seen for M5 Ultra are somewhere about 1200 in in the in the gig in the in that bandwidth area if it's going to be roughly two times than the max um and uh you know but but it is supposed to somehow speed up that first feed to to the first token at least based on M5

36:40benchmarks but I don't know it's it's uh who knows when it's going to come out. So maybe they have more powerful chips. You you need to understand that there is a trade-off right. Um so Nvidia have powerful chips that definitely can do prefill um much efficiently than uh Apple silicon chips can but the trade-off is that the the power consumption right. So Nvidia chips are energy consumers compared to Mac. That's why you see in Mac even though the prefill is low and the energy it's very energy efficient at the same time. So that's a trade-off. Do you want a faster inference or do you want to host a larger model um at a better like better efficiency. That is the trade-off that you need to and you can see the different tiers tiers of

37:26hardware data center level we just discussed H100 workstation. This is still very much affordable RTX 406000. This is this is a very powerful GPU if you can get your hands on it and at the personal AI level we have DGX Spark Mac Studio and strict you know and consumer level these are like regular RTX um you know gaming gaming GPUs but you can also probably use it for AI inference. Okay so what can you buy today what is practical for you right. So these are the options that you have. Mac Studios as you know is a is a hard find these days they're however DGX Sparks are still available in the market. A lot of people are very fixated on Macs they forgot

38:11that DGX Sparks are quite useful machines as well for local inference. And they definitely have stronger chips compared to Mac. But you have to run those on Linux now? Uh so this is what I do right. So I have a Mac M4 Max and a DGX Spark and I use my Mac as a control plane okay and SSH to DGX Spark. So I get the best of both worlds in that sense. I didn't know you could do that okay. >> that's what I do. Um but you still use the commands. You can just use DGX Spark as headless machine right. Um if you are hosting models locally then you need to be convenient with certain level of command line administration um and because uh OSX is Unix like operating system and it has Z

38:58shell and Linux uses like this particular this Debian based OS for DGX uses bash um bash bash terminal so the commands are pretty much very similar. So you won't notice a difference in terms of administration when you use it as headless machine. And the third economical option is Ryzen AI Max um I personally am not a big fan of it because I mean it doesn't have as much software maturity as the other two platforms and also If you're struggling to keep up with content what I'm about to save you about 40 days worth of work. I [music] built something called content machine. It's 10 AI agents that run on the open claw orchestration and they handle everything scripts thumbnails X

39:43posts blogs outreach, clips, [music] newsletters, all of it. So, I went from 1,000 subscribers to 4,000 subscribers on YouTube in 7 days using this exact system. Every single morning I wake up >> [music] >> and the content's already done. I spend maybe 15, 20 minutes reviewing and approving them and I move on with my day. It works [music] for any niche, fitness, finance, real estate, marketing, whatever you're building and it is 100% completely customizable to your use case. So, you get the mission control dashboard, all of the cron jobs, everything I've built over the last 40 days helping me gain more and more people to subscribe [music] and join the community. So, you plug in your own thing and it molds it to you,

40:28it learns how you talk and it writes [music] so it doesn't sound like AI slop. $97 one-time, it's not a subscription. I'll put the link down below and you'll thank me later. The memory bandwidth is really low. So, both prefill and decode is slow with Ryzen. And if you want to if you want a cost-efficient system and you just want to play around, I think Ryzen is a good choice. But, you can't run on it anything that's in my opinion very useful. Based on what I know, um there is not even support for quantization models. So, that's a bummer. And this is, you know, side-to-side comparison of the three competitors, Spark next to Leon and Ryzen and As you can see, Mac has better um memory bandwidth, but um

41:14DJX Spark has better compute. And Ryzen is leader in none, basically. It's a mediocre machine, but it's cheap. And these are the inference machines, as I mentioned earlier. Inference machines are basically runtime for um AI inference. Um llama.cpp or llama is basically the most used inference engine because it's easy to set up, it's easy to use, easy to configure. Um However, uh traditional llama engine run used to run on llama.cpp and it is not very good at um continuous batching. What that means is like in multi-user system, when you scale the system up, it most likely it cannot keep It's not a right engine for production systems because it can't

42:00handle multi-user scenario. If you need concurrent inferencing, llama is not a right choice. But, for your local inference, if you're running models and sequentially, then llama should be fine. And it's easier to use. vLLM is most used production inference engine because it can do paged attention and continuous batching. So, basically what it means is that it can it can vertical scale up and support like higher loads at more efficiently compared to the other inference engines. SG Lang is also getting very popular in the productions because it's very agent friendly. It has to do with how they structured their KV cache and that way it's very efficient in running agentic models on it. TensorRT-LLM is Nvidia

42:46native just like how MLX is for Apple silicon. The problem The only problem with llama is when it's built on llama, um it's essentially a wrapper around llama.cpp um and it's it doesn't use the local hardware to its full capacity because it's not a native engine. It doesn't use the native library. Even though I think llama recently started supporting MLX. So, if you're using it on Mac, I think they're all I don't think it supports all the models, but if you use llama with MLX, it's going to be much faster than using like default llama with llama.cpp as the back as the back end. So, what are you using on your Mac? I'm currently just using MLX directly. Okay. Uh um and on my Nvidia, I'm using vLLM. Even

43:31though I didn't tune it a lot, uh I need to start tuning it. Um but, I'm planning to install SG Lang. Seems like it's more efficient these days for my workloads. Again, coming back to same thing. So, llama is very beginner friendly, but it's single user. For first-time users, this is very beginner friendly, you can start with that. For but, in the production, you might want to look into vLLM. What What do you mean by single user? Um what What I mean by that is um you know, in a production environment, barely your system will be accessed by single user. You're multiple users accessing your system concurrently. So, like if I if I have two If I have a Hermes Hermes agent and uh uh an Open Claw, can they both reference

44:17and use llama? They can use llama, but it's going to be slower because the processing is sequential. It has to wait for the other system to release the um utilization. Uh It has to It has to release the resources for the other system to use. With vLLM, they both can use it concurrently. That's the difference. And vLLM works on Mac? Um vLLM Yes, they recently I think they have they started supporting Mac, but even though um I I don't think it's very mature in the Mac environment. It's mostly in the Nvidia. So, on on the on the Mac, [snorts] what am I after besides llama? You can do MLX. MLX is native and llama MLX is recently gaining traction. And is MLX faster? Yeah, definitely MLX is

45:03faster because it's native to Mac. It uses native Mac libraries. And What is that? I'm sorry. SG Lang is only on Nvidia? No, SG Lang does text, too, if I'm not wrong. Yeah. SG Lang is essentially an alternative to vLLM. It's gaining traction, it's newer hardware. But, my point is you can't run it on a Mac You can't run it on a Mac. SG Lang, I believe you can. I never tried it. Okay. Yeah. And these are as of 2026, beginning of 2026, these were like popular open-source models. As you can see, like in past month or two, things have changed. We got GLM 5.1. We have Deep Seek version four now. We have Qwen 3.6. Lot has changed ever since then. And we have Qwen four. Um yeah, so it's really crazy how fast the

45:50industry is moving. So, these were some of the models that I could think of that are popular that you can leverage in your local setups. What is What does a mixture of agents actually mean? I don't I never quite understood that term. So, mixture of agents is basically a way for the inference to make inference faster because what happens is it activates only certain parameters that's relevant to your use case for that particular prompt during runtime instead of activating all the weights of the model. So, in a dense model, let's say if you have a uh 140 billion dense parameter model, no matter what user prompt you give, no matter the use case, uh it always activates all the parameters, all the

46:35data weights, the neural pathway, the entire neural pathway. With mixture of experts, it only activates certain pathways that is relevant to your use case that particular point. So, during runtime, it's much faster. Got it. So, so for a local model, I want MOE. Mhm. Then I mean, MOEs are much better. Yes. If you can use MOEs, they're much faster. At least theoretically, they should be faster. Yes. I think at scale, you'll notice difference. Okay, thanks. And, you know, these are different parameters. Um these are different models based on your hardware constraints. These are different models that you can post. As I mentioned, you can use the formula that I given you earlier, right? So, an 8-bit model at Q4 Q4 is basically 4-bit, right? That's

47:20like the um 16 like a quarter of a byte. It will take like 2 billion parameters. Two sorry, two gig two gigabyte of RAM plus maybe two gigabyte of KV cache will give you probably around 4 gigabyte of utilization for that. So, it's comfortably you can host this model at Q4 on 8 GB VRAM. I would just I would just mention that for for like anybody who's trying to do this on a Mac, uh the math changes because Mac is using uh that RAM for everything and I can tell you that like I tried running 27B Q4 model on a 32 gigabyte machine and you very quickly go into like severe pressure and it just quits on you. Correct. Like even

48:06Yeah, even 27B on 32 gig Mac doesn't work. Well, yes. Um that's because um you're you're also using So, this is it only works if you use the system as purely AI inference engine and you're not doing any other work on it in most cases because you have you're preserving most of the RAM for inference, right? But, if you're using the system for other needs for other needs, then yes, like, you know, those applications will also need to have access to the same memory pool and it will tremendously slow down your system. So, in most cases, that's why I would recommend not use your MacBook for inference. I mean, you can use it if you just want to use it for special purpose, like if you have a Qwen three

48:52folder, for example, and you have 128 gigabyte of RAM and you're just posting one model, that's probably that's going to be okay. But, if you're hosting multiple models in your Mac and pushing it to its limits, it's going to slow down. So, the best way of doing it is to get a Mac Studio and dedicate it for AI inference. Or maybe get a DJX Spark and dedicate it for AI inference. That's what I'm doing now. I'm planning to get another DJX Spark and cluster them together so I can host um larger models. It's called tensor parallelism. You can do that with Mac Studios as well. I just I wanted to mention if anybody is looking to buy like a Mac Studio or a Mac Mini right now, um you can you can also ask Claude Claude um There is like an eight There is like a

49:37special code or special key that Mac that Apple uses for like to be able to check the inventory at local stores. So, if you give your Mac if you give your photo prompt like use the you know use the find which Macs are which Mac Studios and Mac Minis are available in my local stores. I was able to get Claude basically tell me what's available where like rather than having to kind of click around and figure out for each store. I like it depends on it depends on the day but there are still some that are available even though like it's sold out in 99.9% of cases. You you can even do I think now they have this what is that routines. There's a feature

50:22called routines I believe. So, you can schedule you can schedule a search. It will look for you every day and basically give you a report of Mac inventory available in the internet. I wasn't I wasn't able to find any anything any studios with more than 96 gigabytes of RAM anywhere. Yeah. Yeah. Right. So, this this is what I would advise, right? Look into DGX look into Nvidia systems. Right now because everyone's after Mac, they forgot about you know amazing Nvidia supercomputers. So, on that 128 gigabyte what what's the max model you can run on that then? As I said it depends on the quantization. You have to do the math and see Okay. You can you can probably run 70 billion model quite comfortably on. For me though I I host multiple models and

51:07orchestrate them based on depending on the use case. I found that much more efficient for my agentic systems. So, I would say you you can even it's very what do you say very cheap compared to Macs currently the DGX box. You can get it like you can get like I have a gigabyte version of DGX box and I bought it for 3500. So, I can essentially get two and cluster them together for 7000 bucks. And I'm like and they're not even like as capable as Macs right now. And I would advise that you wait until M5 comes out. I think M5 is going to be game changing. You're basically paying premium for same amount of RAM that you could get with Nvidia on Mac. I don't see a point in that. I know Macs are

51:52different and Macs are better bandwidths but still I don't think it's it's worth it in my opinion. And if it's up to you if you prefer Macs but if you already have a MacBook or something you can use it as a control plane and just as a suggestion to these systems as headless use it as headless machines and still have the same experience. Can I ask where you found that Spark? Where did you find it at that price? Uh, Newegg is the vendor. Newegg. Newegg, yeah. Yeah. Yeah. Even Amazon I think has Sparks. So, this Spark I I could find it for 3500 because it's a 1 terabyte storage. I think they ran out of it. Now they have 4 terabyte ones at like 40 4200 or something but but I think it's still cheap. >> Okay. Okay. So, I because storage is cheap. If you can get like minimum storage just

52:38get that and buy storage separately. It'll be much cheaper. Yeah. Hey can you hear me guys? My mic's kind of Okay. Yeah. It's working out. Do you you mentioned you orchestrate to multiple [clears throat] local models in your hardware? Yes. That that's because you have the the storage I mean the capacitor at the memory capacity to run them all like at same time. Yes, I have Mac M4 Macs at 128 gigabyte and the DGX Spark at 128 gigabyte. Okay. >> So, I run models between them. Yes. Okay cuz I'm actually my local model I don't have the storage like that. I I have like a 390 on my I have like an isolated >> storage do you have? Uh, that one the VRAM 24 gigs and RAM's like 100 200 almost. It's my it's

53:26like a proxmox server. I never I I just utilize it cuz I never prepared to use it for AI. I just got into AI like two months ago. So, I'm just kind of leveraging whatever hardware I have but the constraints or the the thing I've been kind of thinking about was you know I'm I'm limited to running just one model on that hardware on that VM but you know it'd be nice if I could figure out a way I guess to set up in my workflows where I could dynamic my orchestrator could dynamically call a different model on that endpoint and then trigger it turn it on use it and then you know turn it off and I know it's going to be like a slower process but it's just kind of experiment. I'm just trying to figure out like what I could do based on what I

54:11have because I could only run one model at a time. Well, yeah I mean it really depends on what your use case is what what your workflows are and how complicated they are. Yeah, exactly. Yeah, I I I don't really have a use case for it. I'm just trying to like hypothetically think of stuff but I I don't know. Just random just just ask just kind of bring it up. I mean you can you can probably So, if you're interested in self-hosting I think it would wouldn't hurt for you to invest in some hardware. I know initially it's it would be expensive getting a 128 gigabyte you know a machine but I think on the long run it would be much more useful because the open models are getting more and more efficient at this. Mhm. And

54:57there So, eventually I think you know self-hosting is going to be the future. I was going to get to it but you know whatever I say I think you know it's my opinion take it with grain of salt. I think we are in a AI bubble right now um and all the hype that we are seeing about these frontier models is just that. Like it's it's hype. See what Anthropic did before it was 4.7. They throttled 4.6 make made it dumber so that we realize we think that Open is 4.6 is better. I know on the benchmarks it performs little better but there's not like really doesn't matter. And a lot of people have actually did research on this. Turns out you don't need a model that is trained on Shakespearean

55:42Shakespearean literature to be able to be useful for you. In 100 languages maybe and like you you don't need all that, right? So, all those parameters that they talk about the weights of the model like 1 trillion plus weights most of the weights you would never use it anyway. So, the that weight comes from the knowledge base and 80 80% of the knowledge base you would never use. Everyone has their own use case. That is why I think local hosting is going to be more powerful because you can fine-tune your own model according to your use case. And if you engineer your architecture well, I guarantee you you get like frontier level quality on your local system. I really guarantee you that because I've done it and it's currently in beta testing my solution.

56:29I think Bob used it for few of his use cases and he told me that he couldn't discern difference between he was using I think GPT 5.4 and he he said he couldn't notice a difference. And he could do that for free using my architecture for his workflows. Does that make sense? I think you're right. That's why I'm here. So, >> [laughter] >> There you go. And yeah, so I think before we wrap >> this this whole thing the whole thing with the mythos and the zero day stuff is is It's not substantially better. Yeah. It's not. I think it's just they're trying to see bubbles when burst. I think >> to they're trying to list for 800 billion dollars. That's why, right? They're like trying to give all these [laughter] reasons. Yeah. They're going to inflate the valuation. They're going to screw the investors. That's what

57:15they're going to do. Pardon my language but that is what is happening right now. And a lot of large enterprises are yet to realize the ROI on the investment that they've made. Is AI like absolute bubble? I think we are in a bubble but it doesn't mean that AI is going to be useless. It's going to be very useful but it's not what they're saying it is. We just learned it's just a you know pattern recognition engine. It doesn't have it it it gives you an illusion of agency but it doesn't have the agency in the sense that humans have. So, all this BS about you know what is that AGI and all that? I mean I don't buy it. Invest in your local hardware if you

58:01can. I think it will pay you off long term. And another one thing is I think Do you Do you know who who won the AI race essentially? Who do you think won the AI race? And I have my own opinion. I'll share it with you but if you have yours share it. Nvidia. I don't think so. I mean Nvidia [clears throat] in a sense yes. Nvidia in a sense yes but Apple because they didn't spend all their money on it they're going to buy one of the bankrupt players. They're going to buy a bankrupt player with their hundreds of billions in cash. There you go. And you know what Apple did, right? They focused on building the hardware for it at the consumer level. So, everyone can host their own models. Even Nvidia in that sense is going to lose. Apple is better at lot of things. I mean Nvidia does have like years of you know software

58:46development community. I mean they have they're better at production as of now but if you think about it Nvidia GPUs weren't designed for security. They were designed for speed. They were originally when they envisioned it they envisioned it for gaming not for AI inference. Apple edges security in their hardware Nvidia doesn't. For Nvidia it's optional. And Apple is doing a lot of things right and it's going to be a secure environment for you to host your models. I mean you can still use Nvidia and you know make it secure but it's you know on top of that it's just a feature of Nvidia that you need to turn on to make your hardware secure. But in Apple it's by default. Everything's encrypted. You need process level authentication to access the resources in Mac. Nvidia

59:32doesn't have it. You can turn it on, but it's not default. So, I think if anyone's going to burst the bubble, I think Apple, in my opinion, is going to be the pin that's going to burst the bubble, and they're going to come out victorious. Because even Nvidia's going to lose when people realize that. If everyone's hosting their own models in Apple hardware, then what happens to Nvidia? I mean, of course, this is all speculation. It's just based on my observation, but I think it's very likely. Yes, buy Apple stock. Yeah, I think I think Apple stock is already going up. I was thinking the same thing, but it I did that analysis on it, and if you look at what percentage of their sales make up revenues for the Mac, it's like 5, 6% of sales. So, you know, that would have to that would have and it could very well, but, you know, does that double or triple, quadruple?

1:00:17Because they're so dependent on the iPhone, right? And and if you look at the Nvidia chip, Nvidia went up in price 10x after everybody discovered that it was good for AI. If they're going to discover that Apple is good for AI, like Apple has not gone through 2x or 3x type thing. That's true. I think they're they're going to do it. I think they're going to reach there within 5 or 10 years. It's going to happen. But the biggest constraint with Apple is the supply chain issues that they're facing right now. That is where the constraint. That's why the sales are still low. The demand is so high. There are like scalpers everywhere ripping people off on Apple Apple hardware. Imagine if Apple comes up with enough supply, what's going to happen? People are going to grab it. Yeah, the 4,000 Mac is being sold on Swappa for 6,000.

1:01:02>> Yeah. I think like one the 5 TB Mac Studio was sold for over 20 grand on eBay, which is crazy. People are buying it. They want it because it's a rarity now. You can't get 5 TB Mac. But I so I still I'm I'm still coming back to the same question. So, I have a 96 GB Mac Studio that I picked up, and I'm trying to like I still I don't really understand. So, the models that I would practically run, like I'm never going to run anything similar to like Anthropic type stuff or like really big models. It seems like most models at quantization like of Q4 like above there's 27 and 35, and that's about where it caps off. So, Yeah. what is the like I don't really

1:01:48is there really a reason unless you're planning to run two different models at the same time, is there really a reason for more than 96 at the moment? >> Not not necessarily. I think majority as I said, right? So, you can get Qwen 3.6 comfortably at I think 4-bit quantization. You can run it in 96 gig comfortably. Okay, and and I do have a question on the on Ollama stuff. You can get this community community pre pre-built stuff, or there are some community models where they remove where they remove some restrictions, right? Like or some guardrails so that you're getting more of you whatever whatever Yeah, whatever the Chat GPT and

1:02:34Anthropic have to kind of block the the models from responding. It seems like these models are going to do. Right. Yeah. Yeah, they remove guardrails. I don't know what's going to be like the implications of that. I mean, I don't know the ethical implications of removing guardrails. I don't know what kind of guardrails they've removed, but so so what is your question though? Well, so I you know, I kind of I there's a Qwen there's a Qwen model that's like available I forget what it's called when when they basically remove the guardrails off of it. And and I don't know if it's worth trying that as a local model cuz you can't get that anywhere else. So, I would say if you don't have you can try Qwen 3.6. It it did great on

1:03:19benchmarks. I'm sure it would be good for most of your use cases. And if you want multi if you have any multimodal use cases, then you can do Qwen 3.5. I think Omni it's called Omni. Okay. Yeah. And there are smaller models like text to speech, speech to text, and all those. You can run it on basic 10, 12 GB RAM. So, I've had like old hardware as well. I Basically, what I did was I I created a mesh system where I have a Mac in four Macs, and I have DGX Spark, and I have an old gaming PC with RTX 5070, and I had an old MacBook Pro. So, I I created a mesh system, and I'm installing, you know, different components in different systems. It's causing me a lot of 2x to my energy bill, but

1:04:04>> [laughter] >> but I think it's worth the experiment. I'm I'm learning a lot. Yeah. Thank you. Any more questions? Did you run any benchmarks on your hardware to kind of see what models No. No, I mean, I haven't run any benchmarks per se. You know, it's I want to eventually, but you know, I have too many things on my plate right now. I I don't know if benchmarks will tell any story, to be honest. It really I my I'm going to rely on my personal benchmark, right? Whether my use cases are being catered to with the local models. That's the only benchmark I care about. Yeah. But I compare different year models. Yeah, that's fair. I I

1:04:49haven't done it myself. It's just something I never I haven't explored. It's like you said, there's so much, and you don't have time for everything. So, then you know, then I was wondering, should I you know, is that is that something I don't know if you did that before you picked your models, but Yeah, it makes sense. You kind of figure out how it works with models. >> Yeah, I did pick my model based on the third-party benchmarks. But you know, I mean, benchmarks doesn't translate to production performance. Okay. So, benchmarks are just that. It's going to tell you how it is performing in relation to other models. So, I mean, I just tried I I tried to use different models. I I tried to stick with Qwen family for most, and I experimented with Gemma 4, but I think

1:05:35Qwen is better. And now I want to I have Qwen 3.5 as a reasoning engine. I have Qwen 3 coder for coding, and I have another MoE Qwen 3.5 MoE model that I'm using for RAG. So, all of these systems come together, right? And like I'm actually essentially using it for my development now. I'm using antigravity, combination of antigravity and this. I just downgraded my cloud subscription from max to pro. I don't think I'll be needing that anymore because most of the scaffolding I can do it locally, and antigravity is very cheap. I can get for 20 $20 I get like a lot of limit compared to that. And what I did was I I used my local system. I added an MCP to Zed, and I'm use I'm

1:06:21basically emulating cursor features on Zed using my local models for agentic agentic inference, and also like the code completion features. They're working pretty well for me so far. Nice. Nice. Sounds interesting. But by any chance, did do you have any um Do you have any like documentation or like have you published your kind of like workflows, environments at a high level so that you're not exposing, you know, personal stuff? Um I mean, I haven't prepared any documentation. Most of it is still in my mind. It's just most of them are like my pet projects that I'm doing. But I am I have a small business that I own co-own with my wife, and I'm trying to automate workflows. Um and I want to leverage the local models. So far, what I did was purely

1:07:08research. I was trying out different things, and I think the time has come that I apply these models to the real world use cases. Yeah, definitely. Sounds interesting. Thanks for sharing. My my system, I'm planning to once So, this is in beta testing right now. So, I want to put this system in cloud, and going to invite more users for testing. If you guys are interested in that product, let me know, and I can add you as a beta tester. Now, like as of now, we're doing I you can call it alpha testing. Yeah. Yeah. So, as I said, like so, I'm basically using a unified memory module and inference system for disparate agents. So, you can connect all your agents to single system, and it will

1:07:54maintain the context, and it will compress your context window. It will add more context, and it will only use frontier APIs when necessary. So, majority of the use cases, the locally hosted models should be able to fulfill. In that case, it will reduce your monthly cost from industry for local hosting. My focus has been on context engineering and cascading for inference. Yeah. So, you must already know, you know, these are the different databases for downloading models. Most of you seem like already hosting models locally, so you must be knowing all the databases from which you can download. Another thing though, I don't know if you guys are doing it, you can still

1:08:39fine-tune your model using these arguments here. You can play around with, you know, maximum GPU utilization. I think someone mentioned it's slowing down their system. You can virtually put a limit on GPU util. So, it doesn't slow down your system, right? So, these are the arguments you can use with vLLM or, you know, Ollama. These are examples of configuration. These are the knobs that you can use to essentially fine-tune your model. And tensor parallel. So, tensor parallel is when you're basically you say if you're using multiple nodes in a cluster, you can use tensor parallel to make it work as a single unit. So, that will increase your overall GP capacity. And cache KV type is like you can you can even Um, you can even

1:09:26quantize the KV cache for better memory management. So, when you host locally, you can do all that and manage your system. You can tune your fine-tune your models. If you're not doing that, you're missing out on a lot of things. And again, so this is the self-hosting mindset. You own it, you play with it, you iterate, and stay strategic. Know your hardware, know your limitations, and fine-tune your models accordingly, and be strategic about it. Okay, thanks a lot. Yeah, that's all I have to say as of now. Any more questions? Do Do you guys find Did you guys find it useful? Oh, yeah. Really useful. I've been trying to follow hardware-related content for the last few weeks, and this kind of just brings things, you know, back to memory. And although I don't get

1:10:11it all, it's it's kind of slowly, you know, come together. It's It's a I mean, there's definitely learning curve. You'll get there. So, so keep keep researching. Um, I think that's the beauty of what self-hosting is. You learn a lot, too, in the process. Oh, yeah. And if you don't want to do all this work, man, and you can still have I mean, you can hire someone to do it for you. But end of the day, um, if you have a business and you want to automate your workflows using AI, um, self-hosting is a better investment long-term than, you know, investing in cloud. Just because you you have that independence from those companies. You don't always need to be on leading edge to have a workable system. In fact, most of the stable production systems are not leading edge. They're stable for a reason. So.

1:10:56So, I think we're way over the hour. So, if you have any questions, please let me know, or Oh, thank you very much. That was Thank you for your time. We really appreciate it. Thank you, Vamsi. >> Yeah, thank you very much. It's been great. Yeah, very helpful. Thanks. Take care, guys. Yeah, nice one. Yeah, if you have any question, you can reach out to me, you know, in the community. Yeah, I sent you a DM in the community. Okay, sounds good, man. Have a good one. Bye-bye. So, I know that that was a lot to digest, but like I said, if you join the community, you can have free Q&As, you have unlimited one-on-one calls, and you get access to Vamsi if you choose to help him, you know, work with you to be able to build out your local setup. So, I'll put the link to the community down below, and like I said, let me know your thoughts, and we'll see you in the next video. Take

1:11:41care.