Open Proposal for Interest-Driven Artificial World Modeling in Robotics

I am not sure if this is the right area to drop this. This is an idea I had that could have a lot of research implications especially as it get down into anticipation and fine details of observations. Aspects in robotics and Ai with 3D modeling. I am imagining an array that operates as separate nodes but data processed and analyzed in layers.

Open Research Proposal

An Observation-First, Interest-Driven Artificial World Model

A Proposal for Open Collaboration in Active Perception, Continual Learning, and Predictive 3D World Modeling

Status: Open concept / collaborative research proposal
Intended participants: Robotics researchers, computer-vision researchers, AI/ML researchers, students, makers, open-source developers, and hobbyists

Abstract

This proposal explores whether an artificial system can develop an increasingly complex model of its physical surroundings primarily through continuous live visual observation, rather than beginning with extensive semantic knowledge about the world.

The proposed system uses binocular vision to obtain depth and scale, constructs persistent representations of observed entities and spaces, distinguishes direct observations from its own reconstructions and assumptions, and gradually attempts to predict how its environment will change.

A central feature is autonomous interest-driven attention. Rather than distributing computational resources evenly across everything in view, the system may assign higher “interest” to objects, regions, events, or behaviors that are novel, uncertain, difficult to predict, changing, or otherwise informative. Increased interest may cause the system to allocate additional observation time, camera attention, memory, reconstruction quality, and computational resources to that subject.

This creates a potential learning cycle:

observe → reconstruct → become interested → focus → model → predict → observe again → compare → revise

The project’s primary question is not whether an AI can generate a realistic representation of the world. It is whether a system can incrementally construct and correct an empirically grounded internal model of its surroundings through sustained observation.

This direction overlaps with active perception, world models and continual learning, which are established and rapidly developing research areas. Current research describes active perception as dynamically changing viewpoints to gather useful information, while contemporary world-model research studies predictive representations of environmental evolution. 12

1. Research Question

The central question is:

How sophisticated and predictive a model of its surroundings can an artificial system develop when sustained observation is treated as its principal source of truth?

Several secondary questions follow.

Can a system discover persistent entities without initially knowing their conventional names?

Can it construct increasingly detailed 3D representations from repeated binocular observations?

Can it discover regularities in the behavior of objects through repeated exposure?

Can it recognize when one of its assumptions contradicts subsequent observations?

Can prediction error direct subsequent learning?

And perhaps most unusually:

Can the system learn what is worth looking at?

Rather than treating visual attention as simply an optimization problem imposed by a developer, this project proposes allowing the system’s accumulated experiences to influence how it allocates its future observational and computational resources.

2. Motivation

Modern perception systems can recognize, reconstruct and track many aspects of an environment.

However, this project proposes deliberately restricting the amount of externally supplied knowledge available to the learning process.

The distinction is important.

Instead of beginning with:

“This object is a chair, and here is what chairs normally do.”

the system might initially store:

“Persistent Entity 184.”

Its understanding of Entity 184 would then develop through experience.

For example:

• It appears rigid.
• Its shape remains consistent.
• It is usually stationary.
• Certain other entities repeatedly approach it.
• Those entities undergo recurring posture changes near it.

The objective is to allow the internal model to develop from those observations before, or even without, attaching the human semantic category chair.

The experiment therefore emphasizes the difference between perceiving something and being told what it is.

3. Relationship to Existing Research

This proposal does not claim invention of stereo vision, 3D reconstruction, world models, object tracking, active perception, or continual learning.

Instead, it proposes integrating these ideas under an unusual experimental constraint.

Recent surveys characterize world models as predictive representations of how environments evolve under actions and describe applications including planning, simulation, evaluation and robot learning. 14

Active-perception research explicitly investigates systems that change viewpoint to acquire information when passive observation is inadequate. For example, AP-VLM: Active Perception Enabled by Vision-Language Models investigates robots selecting viewpoints and orientations to resolve perception problems. 2

The idea of continually updating world models is also an active research question. The Continual World Models workshop describes the challenge in terms particularly relevant to this proposal: noticing prediction failures, incorporating new evidence, and revising an evolving model of the world. 3

There is therefore considerable existing research on the constituent problems.

The experimental contribution proposed here is their combination around observation-first learning, explicit epistemic provenance, autonomous interest, and resource allocation.

4. Observation-First Principle

Every piece of internal knowledge should maintain information about where it came from.

A proposed minimum distinction is:

OBSERVED

Directly supported by sensory information.

RECONSTRUCTED

Derived geometrically or temporally from multiple observations.

INFERRED

Not directly observed but currently considered likely.

PREDICTED

A proposed future state that has not yet occurred.

EXTERNALLY IDENTIFIED

Knowledge supplied by the permitted external identification mechanism rather than learned through local observation.

UNKNOWN

Information for which the system currently has insufficient evidence.

This division is fundamental.

An inferred surface on the unseen side of an object should not silently become an observed surface simply because the rendered model looks convincing.

Likewise:

visual realism ≠ empirical confidence

5. Physical Perception

The initial physical implementation is proposed around a commercially available stereo/depth camera rather than designing custom camera hardware.

Two synchronized viewpoints provide binocular disparity and therefore a basis for estimating depth and scale.

Rather than attempting distortion-heavy instantaneous 180-degree observation, the stereo camera could ultimately be attached to a controllable pan/tilt mechanism.

The resulting sensor becomes analogous to a movable head:

the system decides where to look.

This converts perception from a passive stream into an action.

6. Persistent World Model

Incoming frames should not be treated as independent images.

Instead, observations contribute to a continuously maintained representation:

Physical world
      ↓
Stereo observation
      ↓
Depth / geometry
      ↓
Spatial registration
      ↓
Persistent entities
      ↓
┌────────────────────────────┐
│ Persistent world model     │
│                            │
│ Geometry                   │
│ Appearance                 │
│ Position                   │
│ Motion                     │
│ Observation history        │
│ Confidence                 │
│ Assumptions                │
│ Predictions                │
│ Interest                   │
└────────────────────────────┘

An entity may begin as little more than a region exhibiting spatial or temporal consistency.

Additional observations can progressively refine it.

7. Progressive Representation

Not every observed object requires the same representational complexity.

A possible progression is:

Level 0: transient visual region

Level 1: persistent 2D entity

Level 2: sprite or representative appearance

Level 3: approximate depth and bounding geometry

Level 4: simplified 3D geometry

Level 5: refined geometry and texture

Level 6: articulation or internal movement model

Level 7: learned behavioral representation

Level 8: predictive behavioral simulation

These levels should not necessarily form a rigid sequence.

An important research question is whether the system itself can learn when richer representation produces enough predictive value to justify its computational cost.

8. Artificial Interest

A major proposed contribution is an interest mechanism governing attention and resource allocation.

The system would continuously estimate how valuable further observation of an entity or event might be.

Potential contributors to interest include:

Novelty

Is this unlike previously observed entities or events?

Uncertainty

Does the system have low confidence in its existing model?

Prediction error

Did reality significantly contradict what the system expected?

Change

Has something unexpected changed?

Complexity

Does the current representation repeatedly fail to explain observations?

Interaction

Does the entity appear to influence other entities?

Persistence

Does an apparently important entity repeatedly reappear?

These factors are hypotheses for experimentation, rather than predetermined definitions of artificial curiosity.

The eventual objective is to investigate whether useful attention policies can emerge from prediction and information gain rather than from hard-coded lists of interesting objects.

9. Interest as Computational Resource Allocation

Interest would have practical consequences.

For example:

Entity 12
Interest: 0.08
→ coarse geometry
→ infrequent observation
→ low memory allocation

Entity 53
Interest: 0.94
→ camera follows entity
→ increased observation frequency
→ additional viewpoints
→ higher reconstruction budget
→ greater memory retention
→ behavior modeling enabled
→ prediction experiments enabled

This produces a limited-resource architecture that does not attempt maximum understanding of everything simultaneously.

The system instead develops unevenly according to what its own experience indicates deserves investigation.

This feature could be particularly valuable for implementing the project using consumer or hobbyist hardware.

10. Active Observation

Interest should eventually affect the physical sensor.

If an entity receives sufficiently high priority, the camera system could:

1. maintain it near the useful stereo region;
2. follow its movement;
3. seek alternative viewing angles;
4. revisit partially observed geometry;
5. observe interactions involving it;
6. determine whether additional observations reduce uncertainty.

Active perception is already an established research direction. AP-VLM: Active Perception Enabled by Vision-Language Models, for example, describes adjusting viewpoints to gather information in partially observable environments. 2

The proposed experiment asks a somewhat different question:

What should make an observation worth acquiring when there is no externally assigned task?

11. Temporal Learning

The system should retain not merely objects but events.

Given repeated observations:

State(t-2)
     ↓
State(t-1)
     ↓
State(t)

it can begin predicting:

Predicted State(t+1)

The future observation then supplies:

Actual State(t+1)

The discrepancy becomes a learning signal.

prediction
     ↓
reality
     ↓
prediction error
     ↓
model revision
     ↓
new prediction

The objective is therefore not simply next-frame generation.

A visually convincing predicted frame could still represent an incorrect understanding of the environment.

The more meaningful measurement is whether the internal model’s predictions become increasingly consistent with subsequent observations.

World-model research already uses predictive environmental representations for areas including planning and robot learning. 14

12. Continual Learning

The world model should not become permanently frozen after an initial training stage.

New observations may:

• reinforce existing models;
• weaken assumptions;
• introduce new entities;
• reveal previously hidden geometry;
• expose prediction failures;
• create new behavioral hypotheses.

This is closely connected to current continual-world-model research. The Continual World Models workshop explicitly identifies adaptation from new experience, detecting change, revising internal state and retaining useful knowledge over time as relevant research problems. 3

Long-term robotic adaptation remains an open research area. A current Frontiers research topic on world models and continual learning highlights challenges involving changing environments, updating representations, knowledge retention and long-term autonomy. 5

13. Controlled External Knowledge

The proposed experiment substantially restricts Internet access.

Normal object understanding, physical reconstruction and behavioral learning should not receive answers from Internet searches.

A narrowly isolated external identification pathway could eventually be investigated for people and animals, subject to privacy, consent and applicable legal requirements.

Any externally supplied result should have explicit provenance:

LOCAL OBSERVATION:
Entity 72

SYSTEM INFERENCE:
articulated
self-propelled
four primary ground-contact limbs

EXTERNAL IDENTIFICATION:
candidate identification supplied externally
confidence: X
source recorded separately

An externally supplied label must not retroactively convert inferred characteristics into observations.

A privacy-preserving implementation should also allow external identification to be completely disabled.

14. Suggested Experimental Phases

Because the project is intended to accommodate hobbyists and researchers with different resources, it should be modular.

Phase I: Live stereo world reconstruction

Establish calibrated live binocular observation and persistent geometry.

Question: Can the system maintain a stable spatial representation from continuous observation?

Phase II: Persistent entities

Introduce object segmentation/tracking without requiring semantic naming.

Question: Can Entity 37 remain Entity 37 after movement, partial occlusion and viewpoint change?

Phase III: Progressive reconstruction

Allow repeatedly observed entities to accumulate richer representations.

Question: Does additional observation measurably improve the entity model?

Phase IV: Interest

Introduce resource limitations and interest scoring.

Question: Does selectively spending resources improve useful environmental modeling compared with uniform allocation?

Phase V: Active vision

Allow interest to control a pan/tilt stereo sensor.

Question: Does autonomous viewpoint selection reduce uncertainty?

Phase VI: Prediction

Introduce short-horizon temporal prediction.

Question: Does prediction improve through repeated observation and correction?

Phase VII: Continual world modeling

Run increasingly long experiments in changing environments.

Question: Does the system retain useful previous knowledge while incorporating contradictory new evidence?

15. Evaluation

The project should resist evaluating success simply through impressive demonstrations.

Suggested measurements include:

• stereo/depth accuracy;
• 3D reconstruction error;
• persistent identity accuracy;
• prediction error over different time horizons;
• uncertainty calibration;
• computational cost per entity;
• improvement attributable to active observation;
• prediction improvement over repeated encounters;
• representation quality relative to computational cost.

One particularly useful comparison would be:

uniform resource allocation

versus

interest-driven resource allocation

If the interest-driven system learns more predictive representations using the same computational budget, the proposed attention mechanism has demonstrated practical value.

16. Failure Is Useful Data

This project should explicitly document incorrect assumptions.

Mirrors, screens, windows, shadows, reflections, transparent objects, partial occlusions and unfamiliar movement may all cause misleading interpretations.

Rather than simply treating these as perception failures, they provide valuable experiments.

For example:

What observation caused the model to form an incorrect hypothesis?

What subsequent observation contradicted it?

Did the system recognize the contradiction?

What changed internally afterward?

A useful world model should not merely accumulate beliefs.

It should be capable of discovering that it was wrong.

17. Safety and Privacy Principles

Live observation makes privacy a foundational requirement rather than an afterthought.

The project should favor:

• local processing where practical;
• visible indication when cameras are operating;
• configurable exclusion/private zones;
• explicit retention policies;
• minimal storage of unnecessary identifiable imagery;
• disabling external identity lookup by default;
• consent-based human identification experiments;
• complete logging of information obtained externally.

The objective is environmental learning, not surveillance.

18. Open Collaboration Model

A contributor should not need to implement the entire project.

Independent modules could include:

• stereo perception;
• calibration;
• pan/tilt control;
• persistent object tracking;
• 3D reconstruction;
• sprite/texture generation;
• memory architecture;
• novelty detection;
• uncertainty measurement;
• interest scoring;
• resource scheduling;
• temporal modeling;
• prediction;
• prediction-error evaluation;
• visualization;
• privacy controls.

This architecture makes participation possible for both researchers and hobbyists.

A contributor with only a stereo camera might investigate persistent tracking.

Someone without hardware might develop the world-model database or simulation components.

Someone with robotics expertise might implement active camera control.

19. Requested Collaboration

This proposal is being shared because its originator has limited resources and does not currently have the hardware, computing resources, or specialist expertise required to construct the entire experimental system independently.

The immediate objective is therefore not funding or commercialization.

It is to find people interested in determining whether the hypothesis is worth testing.

Contributions are welcomed from people who can:

• identify relevant prior research;
• challenge the assumptions of the proposal;
• recommend existing open-source components;
• design small reproducible experiments;
• contribute software;
• test individual modules on existing hardware;
• propose meaningful metrics;
• identify conceptual or technical failures early.

Negative results are valuable.

If part of the concept is already known not to work, documenting why is itself useful.

20. Core Hypothesis

The proposal can ultimately be reduced to one hypothesis:

A resource-limited artificial observer may develop a progressively richer and more predictive model of its physical environment by combining continuous binocular observation, persistent memory, explicit separation of observation from inference, prediction-error-driven learning, and autonomous allocation of attention toward what it finds informative.

The system does not need to understand everything.

It needs to:

look, remember, hypothesize, test, discover mistakes, and decide what deserves another look.

That is the experiment.