For the last four months I have been experimenting a lot with LLM models in an attempt to simplify the creation of reasoning chains without using an auxiliary programming language. The idea comes from the control unit in classical processors. I see LLM as a more complex ALU, but the models lack a unit that can perform more complex sequences of prompts, including loops. In psychoanalysis there is a concept of the central executive system, which is somewhat similar to the control unit in CPUs.
> The central executive is responsible for controlled processing in working memory, including but not limited to, directing attention, maintaining task goals, decision making, and memory retrieval.
My first thought was that it would be possible to implement a prompt for the LLM that would perform the functions of the central executive. As a result, I came up with this prompt structure:
The data in the green blocks changes depending on the active prompt. Prompts can refer to each other using a reference table. This makes it possible to build graphs of prompts where prompt is a node. The idea can be extended to dynamically created prompts that are added to the initial graph and form new graph segments (in the process of experimentation).
The main benchmarks alredy close to 100:
- common sense reasoning (WinoGrande)
- arithmetic (GSM8K)
- multitasking (MMLU)
- sentence completion (HellaSwag)
- common sense reasoning 'challenge' (ARC)
The only thing is if the Transformers architecture changes or if there are new benchmarks that measure the performance of models in new properties.
What's next? Increasing performance and decreasing token cost has the potential to open up more complex use cases.
This will lead to the emergence of LLM processors and models will run entirely locally. This is a likely development scenario.
Just implemented on gpt-4o a team of 4 agents that was able to build a 2d platformer. The only model that managed to accomplish the task.
4 agents:
- Root agent: task description, Game Design Document generation.
- Project manager based on ReAct framework (actions: bash command, software developer and QA)
- Software developer works in a loop task by task (final code 250 lines).
- QA runs the program, looks at logs, asks feedback from the user.
Imagine if Apple's or Google's cloud went down and all your apps on iPhone and Android were broken and unavailable. Absolutely all apps on billions of phones.
Cloud =! OpenAI
Clouds store and process shareable information that multiple participants can access. Otherwise AI agents == new applications. OpenAI is the wrong evolution for the future of AI agents
Asked GTP4-32k to make an article summary of 10 important points.
And then applied the summary to the software architecture
---
Public and Shared Nature of Software Architecture: Just like physical architecture, software architecture is inherently public and collaborative. A well-designed software system should be intuitive and user-friendly as it is used and experienced by a broad range of people.
Traditionalist vs. Modernist Debate: In software architecture, the debate between traditional monolithic systems and modern microservices architectures echoes the debate in physical architecture. This leaves some developers ideologically homeless as they see the benefits of both approaches.
Misleading Term of 'Traditionalism': The term 'traditionalism' can be misleading. In the context of software, it might suggest sticking to older technologies or methods, even though modern approaches have become traditions in themselves, with their own pioneers and best practices.
'Goodism' and the Value of Good Software Architecture: The concept of 'goodism', favoring good works over bad ones, applies to software architecture. However, the sheer quality of the work is not all that matters; considerations such as scalability, maintainability, and usability are also crucial.
Easy vs. Challenging Architectures: Easy software architectures are those that can be understood and utilized without much effort, whereas challenging architectures require significant work to comprehend and navigate. This distinction is orthogonal to the good/bad dichotomy.
Rise of Complexity in High-Status Software: Just as there's been a rise in difficulty in high-status arts, software systems have become increasingly complex, with architectures like microservices and event-driven architectures challenging to comprehend and implement effectively.
Software Architecture as a Background Art: Software architecture often serves as a 'background art' to the user's primary task. Therefore, it should be designed in a way that enhances user experience without unnecessary complexity.
What Makes an Easy Software Architecture?: An 'easy' software architecture is one that is intuitive, scalable, and maintainable, with clear documentation and well-defined interfaces for interaction. It involves a combination of architectural patterns, use of appropriate technologies, and following best practices.
The Role of Simplicity in Software Architecture: Similar to the concept of 'tonality' in architecture, 'simplicity' plays a crucial role in software architecture. The software system should be as simple as possible to meet its requirements, enhancing usability and maintainability.
Flexibility in Choosing Software Architectural Styles: There's no strict need to choose between traditional monolithic architectures or modern microservices. There are many strands of modern software architectures that are accessible, and there's no general reason why we have to stick to old styles or always adopt new ones.
I found that the approach of template processing at large prompts leads to difficulty in reading programs. Their attractive part is that control flow is not separate from prompt as in langchain, which allows you to write prompts as classical programs. But the problem remains in unintuitive syntax for large programs
I'm shocked. I have the same mindset. For example, if I need to predict how technologies can potentially evolve, I visualize them in different planes based on the history of technologies, such as centralization/decentralization cycles, compute networks growing, and the shift to centralization. Also, I visualize how some technologists like Kubernetes go into decentralization (like this is some of primitive form of life, where Datacenters/Bigdata some sort of prebiotic soup for the first vertebrates (Edge Computing devices)). And decentralized solutions such as blockchain are superimposed on this infrastructure. In my head, it looks like a huge visual picture with different levels of abstraction. It constantly evolves
I see how human sins are the drivers of technology development
In our project, we have a lot of Raspis and configure them all with Ansible. From time to time we are adding new features and fixing bugs in our Ansible scripts and that makes them more complex. We are using GitLab to review our code, but what is missing is a validation of the syntax and that the code will actually work on production’s hardware.
Our Pis are an integral part of our customers. And that’s not the definition for CD(for me :)). Our pis are not easily accessible in a matter of minutes and are distributed across several hundreds of kilometers.