Reply to thread

Message: <blockquote data-quote="NEMISIS" data-source="post: 30512194" data-attributes="member: 494668">If you have a mac you can do this easily, it's processor and unified memory is very good for this kind of workloads, it may not be good as running on a GPU but should be good enough. i am thinking buying a M4 Macbook next week with 48GB Ram, Just for this purpose. 70b model is about 43GB, General rule is, you need to have 1.2*model size of RAM, 48GB should be good enough.btw i played around with both 1.5b and 7b models, i was happy with the results given the size of the model, didn't expect GPT4 level results from this. Having a conversation with a locally running AI is something we never thought of.</blockquote>

Top Bottom