llama-conductor
codeberg.org
Route workflows, not models. Glass-box, not black-box. Squash LLM nonsense.

[deleted by user]

  • PolarKraken@lemmy.dbzer0.comEnglish
    1·
    4 months ago

    This sounds really interesting, I’m looking forward to reading the comments here in detail and looking at the project, might even end up incorporating it into my own!

    I’m working on something that addresses the same problem in a different way, the problem of constraining or delineating the specifically non-deterministic behavior one wants to involve in a complex workflow. Your approach is interesting and has a lot of conceptual overlap with mine, regarding things like strictly defining compliance criteria and rejecting noncompliant outputs, and chaining discrete steps into a packaged kind of “super step” that integrates non-deterministic substeps into a somewhat more deterministic output, etc.

    How involved was it to build it to comply with the OpenAI API format? I haven’t looked into that myself but may.

      • PolarKraken@lemmy.dbzer0.comEnglish
        0·
        4 months ago

        The very hardest part of designing software, and especially designing abstractions that aim to streamline use of other tools, is deciding exactly where you draw the line(s) between intended flexibility (user should be able and find it easy to do what they want), and opinionated “do it my way here, and I’ll constrain options for doing otherwise”.

        You have very clear and thoughtful lines drawn here, about where the flexibility starts and ends, and where the opinionated “this is the point of the package/approach, so do it this way” parts are, too.

        Sincerely that’s a big compliment and something I see as a strong signal about your software design instincts. Well done! (I haven’t played with it yet, to be clear, lol)

  • floquant@lemmy.dbzer0.com
    1·
    4 months ago

    Holy shit I’m glad to be on the autistic side of the internet.

    Thank you for proving that fucking JSON text files are all you need and not “just a couple billion more parameters bro”

    Awesome work, all the kudos.

  • recklessengagement@lemmy.world
    0·
    4 months ago

    I strongly feel that the best way to improve the useability of LLMs is through better human-written tooling/software. Unfortunately most of the people promoting LLMs are tools themselves and all their software is vibe-coded.

    Thank you for this. I will test it on my local install this weekend.

  • sp3ctr4l@lemmy.dbzer0.comEnglish
    0·
    4 months ago

    This seems astonishingly more useful than the current paradigm, this is genuinely incredible!

    I mean, fellow Autist here, so I guess I am also… biased towards… facts…

    But anyway, … I am currently uh, running on Bazzite.

    I have been using Alpaca so far, and have been successfully running Qwen3 8B through it… your system would address a lot of problems I have had to figurr out my own workarounds for.

    I am guessing this is not available as a flatpak, lol.

    I would feel terrible to ask you to do anything more after all of this work, but if anyone does actually set up a podman installable container for this that actually properly grabs all required dependencies, please let me know!

      • sp3ctr4l@lemmy.dbzer0.comEnglish
        01·
        4 months ago

        Oh I entirely believe you.

        Hell hath no wrath like an annoyed high functioning autist.

        I’ve … had my own 6 month black out periods where I came up with something extremely comprehensive and ‘neat’ before.

        Seriously, bootstrapping all this is incredibly impressive.

        I would… hope that you can find collaborators, to keep this thing alive in the event you get into a car accident (metaphorical or literal), or, you know, are completely burnt out after this.

        … but yeah, it is… yet another immensely ironic aspect of being autistic that we’ve been treated and maligned as robots our whole lives, and then when the normies think they’ve actually built the AI from sci fi, no, turns out its basically extremely talented at making up bullshit and fudging the details and being a hypocrite, which… appalls the normies when they have to look into a hyperpowered mirror of themselves.

        And then, of course, to actually fix this, its some random autist no one has ever heard of (apologies if you are famous and i am unaware of this), who is putting in an enormous of effort, that… most likely, will not be widely recognized.

        … fucking normies man.

    • Fmstrat@lemmy.world
      01·
      4 months ago

      No promises, but if I end up running this it will be by putting it in a container. If I do, then I’ll put a PR on Codeberg with a Docker Compose file (compatible with Podman on Bazzite).

      @[email protected]

        • FauxLiving@lemmy.world
          0·
          4 months ago

          AI Horde has a OpenAI compatible REST API (https://oai.aihorde.net/). They say that it doesn’t support the full feature set of their native API, but will almost assuredly work with this.

          OP manually builds the oapi JSON payload and then uses the python requests library to handle the request.

          The fields they’re using match the documentation on https://oai.aihorde.net/docs

          You would need to add a header with your AI Horde API key. Looks like that would only need to be done in router_fastapi.py - call_model_prompt() (line 269) and call_model_messages() (line 303) and then everything else is setup according to documentation

            • db0@lemmy.dbzer0.com
              0·
              4 months ago

              Very impressive. The only mistake on the third one is that the kudos are actually transferrable (i.e. “tradable”), but we forbid exchanges for monetary rewards.

              Disclaimer: I’m the lead developer for the AI Horde. I also like you’ve achieved here and would be interesting if we can promote this usage via the AI Horde in some way. If you can think of some integration or collaboration we could do, hit me up!

              PS: While the OpenAI API is technically working, we still prefer people to use our own API as it’s much more powerful (allowing people to use multiple models, filter workers, tweak more vars) and so on. If you would support our native API, I’d be happy to add a link to your software in our frontpage in the integrations area for LLMs.

  • UNY0N@lemmy.wtf
    0·
    4 months ago

    THIS IS AWESOME!!! I’ve been working on using an obsidian vault and a podman ollama container to do something similar, with VSCodium + continue as middleware. But this! This looks to me like it is far superior to what I have cobbled together.

    I will study your codeberg repo, and see if I can use your conductor with my ollama instance and vault program. I just registered at codeberg, if I make any progress I will contact you there, and you can do with it what you like.

    On an unrelated note, you can download wikipedia. Might work well in conjunction with your conductor.

    https://en.wikipedia.org/wiki/Wikipedia:Database_download

  • ThirdConsul@lemmy.zip
    0·
    4 months ago

    I want to believe you, but that would mean you solved hallucination.

    Either:

    A) you’re lying

    B) you’re wrong

    C) KB is very small

          • ThirdConsul@lemmy.zip
            0·
            4 months ago

            The system summarizes and hashes docs. The model can only answer from those summaries in that mode

            Oh boy. So hallucination will occur here, and all further retrievals will be deterministically poisoned?

              • ThirdConsul@lemmy.zip
                0·
                4 months ago

                Huh? That is the literal opposite of what I said. Like, diametrically opposite.

                The system summarizes and hashes docs. The model can only answer from those summaries in that mode. There’s no semantic retrieval step.

                No, that’s exactly what you wrote.

                Now, with this change

                SUMM -> human reviews

                That would be fixed, but will work only for small KBs, as otherwise the summary would be exhaustive.

                Case in point: assume a Person model with 3-7 facts per Person. Assume small 3000 size set of Persons. How would the SUMM of work? Do you expect a human to verify that SUMM? How are you going to converse with your system to get the data from that KB Person set? Because to me that sounds like case C, only works for small KBs.

                Again: the proposition is not “the model will never hallucinate.”. It’s “it can’t silently propagate hallucinations without a human explicitly allowing it to, and when it does, you trace it back to source version”.

                Fair. Except that you are still left with the original problem of you don’t know WHEN the information is incorrect if you missed it at SUMM time.

            • PolarKraken@lemmy.dbzer0.comEnglish
              01·
              4 months ago

              Woof, after reading your “contributions” here, are you this fucking insufferable IRL or do you keep it behind a keyboard?

              Goddamn. I’m assuming you work in tech in some capacity? Shout-out to anyone unlucky enough to white-knuckle through a workday with you, avoiding an HR incident would be a legitimate challenge, holy fuck.

    • Kobuster@feddit.dk
      0·
      4 months ago

      Hallucination isn’t nearly as big a problem as it used to be. Newer models aren’t perfect but they’re better.

      The problem addressed by this isn’t hallucination, its the training to avoid failure states. Instead of guessing (different from hallucination), the system forces a Negative response. That’s easy and any big and small company could do it, big companies just like the bullshit

  • 7toed@midwest.socialEnglish
    0·
    4 months ago

    I really need this. Each time I try messing with GPT4All’s “reasoning” model, it pisses me off. I’m selective on my inputs, low temperature, local docs, and it’ll tell me things like tension matters for a coil’s magnetic field. Oh and it spits out what I assume is unformatted LATEX so if anyone has an interface/stack recommendation please let me know

  • rozodru@piefed.socialEnglish
    0·
    4 months ago

    soooo if it doesn’t know something it won’t say anything and if it does know something it’ll show sources…so essentially you plug this into Claude it’s just never going to say anything to you ever again?

    neat.

  • FrankLaskey@lemmy.mlEnglish
    0·
    4 months ago

    This is very cool. Will dig into it a bit more later but do you have any data on how much it reduces hallucinations or mistakes? I’m sure that’s not easy to come by but figured I would ask. And would this prevent you from still using the built-in web search in OWUI to augment the context if desired?

  • 7toed@midwest.socialEnglish
    0·
    4 months ago

    Okay pardon the double comment, but I now have no choice but to set this up after reading your explainations. Doing what TRILLIONS of dollars hasn’t cooked up yet… I hope you’re ready by whatever means you deam, when someone else “invents” this

  • pineapple@lemmy.mlEnglish
    0·
    4 months ago

    This is amazing! I will either abandon all my other commitments and install this tomorrow or I will maybe hopefully get it done in the next 5 years.

    Likely accurate jokes aside this will be a perfect match with my obsidian volt as well as researching things much more quickly.