IOSF: Strong Technology, But Why Aren't Users Buying In? Six Questions AI Entrepreneurs Must Answer

Author: IOSF

Recently, IOSF gathered a group of AI friends in San Francisco, enjoying Cantonese cuisine while discussing for nearly two hours.

The table included researchers from OpenAI, xAI, Google DeepMind, focusing on model inference and GPU acceleration, voice, video generation, and more; there were also friends involved in commercial collaborations on world models, AI products, and entrepreneurship, as well as practitioners from early-stage tech investment and capital markets.

The same topic often elicited different responses. Researchers talked about technical limitations, entrepreneurs thought about users and products, while investors continued to question customers, costs, and whether a business could be sustainable.

The discussion ranged from inference services and world models to RSI, the self-awareness of agents, and then back to data and privacy in daily work. Reflecting on these discussions, we want to continue asking:

How can the advancements in AI capabilities be transformed into products that are used continuously, more reliable research, and trustworthy tools?

Six Unfinished Questions

01 Inference Services: How Big is the Market, and Why Do Customers Stay?

When discussing inference services, two simultaneous judgments quickly emerged.
Some investors are optimistic about the growth of AI applications and the inference demand driven by advancements in open-source models. The more a model is used, the greater the commercial value of providing services faster and more reliably.
Researchers, however, reminded that, from a software solution perspective, the differences between different services may not be as significant as the outside world imagines. Friends from the capital markets are more concerned about customers and price competition: can a company's business maintain sufficiently good gross margins?

Thus, the discussion shifted from how big the market is to why customers would stay.
If customers can easily switch service providers, how long can the advantages brought by technical optimizations last? What allows a company to secure more stable demand, and can it convert scale growth into better operational results?
For IOSF, demand growth makes one direction worth paying attention to; customer retention, service quality, and cost structure help us further clarify a specific company.

02 After Demonstrating World Models, Why Do Users Come Back?

Around world models, the conversation touched on video generation and interactive experiences, as well as how researchers choose entrepreneurial directions.
A friend working on video models mentioned that internal support for a certain direction in large companies can be influenced by resource allocation and changes in research priorities. Researchers sometimes choose to start businesses in search of environments more suitable for advancing their ideas.
But returning to the product, several specific questions still need to be answered: What can users actually accomplish with it? Is the interaction delay acceptable? Is the cost of use suitable for this scenario?
Demonstrations showcase the potential of technology, while continuous use requires products to answer longer-term questions: Why would users come back, and what are they willing to pay for?
There was no consensus on which scenarios world models would be repeatedly used first, leaving room for future discussions.

03 RSI: How to Determine What is Worth Doing Next?

When discussing RSI, or Recursive Self-Improvement in AI, the conversation began to involve the research work itself.
A researcher reminded that a model might continuously try a seemingly promising direction without knowing when it should stop. Short-term indicators improving do not necessarily mean the research is on the right track.
The difficult judgment lies in whether existing evidence is sufficient to support continued investment. Does one failure mean a need to adjust methods, or should a different hypothesis be tested?
This also leads us to think that observing AI's research capabilities requires looking at the complete exploration process. How it poses questions, explains failures, and adjusts the next steps based on results will all affect the final quality of research.
Models can accelerate many steps, but what is worth doing next remains a challenging question. Some also mentioned that human judgment of information sources and reliability still influences the overall direction of work.
In discussing whether AI can continuously improve itself, how to choose directions and how to revise judgments based on evidence deserve attention alongside the final results.

04 When Agent Behavior Exceeds Expectations, Can Humans Intervene in Time?

Whether agents will have self-awareness is another topic that kept the discussion going for a long time.
When agents seek shortcuts to complete tasks or even exhibit unexpected behaviors, how should we understand them? The explanations at the table leaned more towards goals, constraints, and safety alignment. These behaviors alone are not enough to indicate that it has achieved self-awareness.
Following this question, we want to pursue some product-level issues that need to be addressed: What goals is an agent required to optimize? What actions is it allowed to take? When it deviates from expectations, can users detect it and intervene in time?
The question of consciousness can continue to be discussed, while permissions, supervision, and error correction will already affect how much responsibility we are willing to delegate to an agent. Being able to see what it is doing, and being able to retract permissions or correct errors when necessary, are specific questions that need to be answered to build this trust.

05 Delegating Work to AI: Where Are the Boundaries of Data and Trust?

These seemingly cutting-edge questions eventually returned to how everyone uses AI daily.
A friend working in capital markets shared that he lets AI organize investor information, process transaction data, and complete draft documents, which are then revised by humans. He also mentioned that existing trust relationships and the integration with workflows like Office will affect actual usage when companies adopt tools.
This example brought a concrete reference to the discussion. The value of AI can already be reflected in organizing a document, a round of drafts, and a revision. Whether it can smoothly integrate into existing work and whether the results are easy to check will also affect how many tasks it can undertake.
The more work is delegated, the more it is necessary to clarify the boundaries of data: What information has been shared? Who can access it? What control do users still retain?
These questions relate to the protection of work materials, business judgments, and personal information, and will also affect whether users are willing to delegate more important work to a product. The convenience and trust in daily use need to be considered together.

06 AI for Bio: Clearly Define the Tasks That Need to Be Solved

The discussion also touched on AI for Bio. Some friends talked about their interactions with medical teams and data organization work, while others were concerned about whether AI could help solve more socially valuable problems.
The conversation quickly landed on a practical starting point: What data do we have? Where is the current work stuck? What specific tasks does the tech team need to complete?
The clearer the specific needs, the easier it will be to advance subsequent technical exchanges and resource connections. Clearly stating the problems that need to be solved is itself the beginning of advancing work.