Show HN: Shoehorn – Quantize any model down to run on your machine

3 comments

• hmokiguess 20 hours ago
  Reminds me of https://github.com/AlexsJones/llmfit
  [-]
  • rhgraysonii 13 hours ago
    LLMFit tells you what can run on something. I built something quite similar to their search into Shoehorn now.
• mbuchel-hn 20 hours ago
  does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?
  [-]
  • rhgraysonii 13 hours ago
    Yes that is exactly what this does.
    [-]
    • kennywinker 10 hours ago
      Could you explain what happens when you try to shoehorn a 2.4T parameter model into a 24gb m4 mac?
• jaylane 17 hours ago
  tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running
  [-]
  • rhgraysonii 13 hours ago
    If you could post an issue if you still have the error around that would be awesome.