Project ideas – MAS962 course project

Wednesday, October 09, 2002

 

Misc ideas

meaning as consensus???

language communities etc.

 

use Wordnet???

 

questions

 

how about: agents that babble, where the babble-content is some transformation (that has neural projections back from the speech production system to the underlying concept system) of the contents of some part of their mind, so that as the vocabulary emerges/self-organises, it constrains the concepts, and vice versa (i.e. as each agent’s conceptual framework matures, it affects how he uses the corresponding/transformed word-utterance).

could you use kohonen nets for the concept formation???

better still, couldn’t you use the kohonen nets for the concept-babble transformation??? hmmm. am not so sure about this.

 

Reading – Steels (1996), ‘Synthesising the origins of language and meaning using co-evolution, self-organisation and level formation’

Abstract

The paper reports on experiments in which robotic agents and software agents are set up to originate language and meaning. The experiments test the hypothesis that mechanisms for generating complexity commonly found in biosystems, in particular self-organisation, co-evolution and level formation, also may explain the spontaneous formation, adaptation and growth in complexity of language.

Excerpts

By a language, I will mean an adaptive system of representation used by distributed agents for communication (and possibly other things). As a communication system, language allows the transmission of meanings from one agent to another using some physical medium such as sound. The agents are distributed in the sense that there is no central controlling agency that defines and imposes a language. Each agent can only gain knowledge of others by interaction. A language is adaptive when it expands or changes in order to cope with new meanings that have to be expressed. Moreover new agents should be allowed to enter the group and agents may leave.

Meaning is here equated with a distinction relevant to the agent. Some meanings, like colors, are perceptually grounded. Others, such as hierarchical relations are socially grounded. Still others, like intentions or descriptions of current actions, are behaviorally grounded. Meanings may arise in any kind of domain and task setting, and need not necessarily be expressable in the language. The set of meanings that humans have access to is ever expanding as new environments are entered and new interactions take place. This should be the case for artificial agents which operate in open dynamically changing environments as well.

My main hypothesis is that language is an emergent phenomenon. Language is emergent in two ways. First of all, it is a mass phenomenon actu­ alised by the different agents interacting with each other. No single individual has a complete view of the language nor does anyone control the language. In this sense, language is like a cloud of birds which attains and keeps its coherence based on individual rules enacted by each bird. Second, language is emergent in the sense that it spontaneously forms itself once the appropri­ ate physiological, psychological and social conditions are satisfied. The main puzzle to be solved is how.

…

This paper does not focus on the origin of cooperation or the origin of communication in itself, although these are obviously prerequisites for language. … For example, Dawkins [7] has argued that two organisms will cooperate if they share enough of the same genes because what counts is the further propagation of these genes not the survival of the individual organism. Axelrod [2], Lindgren [18], and others have shown that cooperation will arise even if every agent is entirely selfish. McLennan [22] and Werner and Dyer [34] have experimentally shown that communication arises as a side effect of cooperation if it is beneficial for cooperation. The emergent communication systems discussed in these papers do not constitute a language in the normal definition of the word, however.

The number of agents is small and fixed. The repertoire of symbols is small and fixed. None of the other properties of a natural language such as multiple levels, synonymy, ambiguity, syntax, etc. are observed. The main target of the research surveyed in this paper is to study the origins of communication systems that do have all these properties.

Before starting, an important disclaimer must be made. This work does not make any empirical claim that the proposed mechanisms are an explanation how language actually originated in humans. … Here, I only propose and examine a theoretical possibility. If this possibility can be shown to lead to the formation of language and meaning in autonomous distributed artificial agents, then it is at least coherent and plausible. Thus, if meaning creation mechanisms enable agents to autonomously construct and ground meaning in perception, action, and interaction, then it is no longer self­evident that meaning has to be universal and innate [10]. And if the proposed language formation mechanisms enable artificial agents to create their own language, then it is no longer self­evident that `linguistic knowledge' must for the most part be universal and innate [6] or that language can only be explained by genetic mutation and selection [26].

 

“None of the other properties of a natural language such as multiple levels, synonymy, ambiguity, syntax, etc. are observed” – what are ‘multiple levels’???

Reading – Steels (1995), ‘A self-organising spatial vocabulary’

is this is as crap as it seems???

see email note 021023

is Steels the state of the art??? is Steels’ later work better???

as far as I can tell, they have the relevant concepts (i.e. ‘meaning sets’???) hard-wired, right??? moreover, they have the referential gestures (the means by which the meaning is anchored hard-wired too, right???

am I more interested in language formation or language acquisition???

perhaps I’m interested in language formation in a community, but given a certain innate/genetic language device/predisposition…???

is that actually different from just language formation from scratch??? would it ever have happened in evolutionary history??? i.e. would there ever be a time where a community without a shared language would come together and have to agree on a vocabulary etc. from scratch, given an already evolved, shared innate/universal grammar device???

I suppose so, but would it address the questions I’m interested in??? more importantly, wouldn’t it presuppose/require/beg the bigger question of how the innate language device evolved…??? L

how are the agents modelled themselves???

it sounds as though they’re very simple, symbolic, with variable-slots for each word-meaning pair or something??? L

bear in mind that he’s using self-organisation in what I would consider to be a restricted/technical sense to mean a kind of statistical convergence (see pg 20)

is it localised, or does it depend on some global variables, e.g. communicative success???

I think it’s probably localised, i.e./e.g. I think the word-meaning pair changes only on the basis of the communicative success of a given/local/single dialog, though it might be based on each agent’s memory/history of communicative successes for a given word-meaning pair over all of its dialogs…

why does it bother me that he’s using this type of self-organisation rather than connectionism/evolution, say???

yes, I suppose it’s a tiny bit of an abstraction, but it’s not particularly biologically implausible, after all.

I guess it’s because the brain of each agent that has this rule of changing word-meaning pairs based on dialog communicative successes, and that rule (and its sigmoid probability function) is hard-wired. and so are the meaning sets.

I can’t put my finger on it, but both the rule to change word-meaning pairs according to communicative success, and the pre-specified meaning sets, both define something we might loosely want to call ‘purpose’. that is, the agents already have the hard-wired desire to communicate, and to communicate these particular spatial categories. so maybe I’m interested in the origin of communication…

 

Draft proposals

Preface

I’ve had real difficulty putting this together, because there are lots of things that I’d love to investigate, and because I’m good at assessing what will prove feasible to look at within the timeframe. I’ve tried to sketch a vague philosophical/methodological framework, and then tried to burrow out a feasible approach to a well-defined question.

I’m using the term ‘concept’ here in a loose, deliberately undefined fashion. Perhaps the term ‘representation’ would be better.

 

Approaches - discarded

To put this another way, I’ve assumed that language must be combinatorial, so the interesting question become ‘How do those combinatorial categories cut the world up? and ‘How do they relate to pre-linguistic concepts?’.

 

Misc

I want to play language games in blocksworld

 

Late night scribblings 021024

2 agents in a 1D or 2D world (spatially), and a 1D timeline

concentrate on near/far vs soon/longtime

the beauty of a connectionist representation is that it’s easier to coopt it to a different domain if the input template/representation matches

so we’d train them on near/far or above/below

those are magnitude relations (maybe with the ‘very’ qualifier)

then put them in the same environment with a time-based fitness function and see how well they can learn (with a tiny learning coefficient ‘k’)

then make them do both space and time comparisons alternately

syntax: object, near (spatially or temporally) – ???

nearING passED forward behind???

I knew you probably would want me to keep the course + voluntary projects separate, but time is limited and I can read more effectively – plus, I think that investigating spatial and spatial vs temporal vocabularies is an interesting but usefully restricted but scalable problem

I’m imagining 2 agents describing objects that flicker in and out of existence

meaning sets pre-specified

I don’t know whether they’d be making relative or absolute spatial judgements though

making the temporal domains analogous to the spatial is hard

the objects’ dis/appearance would have to be regular

trace rule??? Grossberg??? FSIT??? Jaynes on metaphor???

could we do this with symbolic representations???

the Regier spatial models are unnecessarily complicated for our purposes

you don’t think that the spatial model for (say) above might have a temporal analogue, do you???

for my project, we’d be using point landmarks (agents) and trajectors (other agents – though I suppose we could use extended objects…)

one problem with 2D worlds is that there’s nothing for left/right to map onto temporally

that might not be a problem – it would be a simple lower-dimensional projection, or alternately, just ignoring/stripping away one

counterfactuals???

maybe, use only 1 agent talking to a copy of itself, rather than 2 separate independent agents

would the speaker and hearer swap roles or be fixed???

possible dual S+T relations:

above, coming, next, near

maybe there needs to be movement in the spatial domain

e.g. the agents standing on a bridge watching things move towards them (cf Boroditsky, ego vs time moving)

maybe it’s equally essential that they can move of their own agency

e.g. dodging oncoming barrels

for the same reason that the blind kittens on trolleys never learned to see

the reason for adding all this complexity is that the Boroditsky phenomena are complex, and if we’re going to see a co-opted representation, we don’t know which bits make the sharing possible/beneficial

maybe the S/T representation sharing is a composite of co-opts

e.g. the ego/time moving is based on agential motor commands, and the space/time-line is based on spatial absolute vs relative references or something

and we don’t (I don’t think) know that it is definitely time based on space rather than a little bit the other way round, do we???

can we draw conclusions about representations based on one rather than the other if we do them both at once though??? i.e. if our agents are trained in a world that has both spatial and temporal aspects…

well, you can’t tell which is based on which, but you can look at the extent to which representations coincide

to what extent do we need language games at all to prove our point (initially)???

why not just have a single mute agent that builds up rich space AND time representations

and maybe then graft on the language games later

could a Kohonen net capture ‘near’ relations???

we should also try our agent in a spaceless world that masters periodic prediction and gets placed afterwards in a spatial prediction task (i.e. first time, then space)

would 3D space add anything???

how about using language as the internal protocol for controlling the agent’s own motor system???

then maybe make the agent control the body of some other mindless body using his own motor-language as a way of thinking of space in relative coordinates

thinking of time in relative vs absolut frameworks is hard

disembodied mind gets fed body coordinate (maybe space x space x time)

and object coordinate (maybe only space x space)

has to give +/-ve space commands

ah, big difference – you can choose to move through space but you cannot choose to move around time

it gives movement commands

maybe it has to move in advance to avoid the oncoming barrels in time

are we looking specifically for a Boroditsky effect???

or would we be happy if we could just show that a single NN module that wasn’t subdivided could do both tasks???

or that 2 modules that are isomorphic in internal structure each did 1 task???

how about babble-transforming into language as the means of constraining the two modules to be similar???

i.e./e.g. maybe the spatially-trained agent has to give babble-transformed to the temporally-trained agent

I’m imagining something like those little handheld games where you have to dodge or catch the falling objects

it’s less and less intuitively clear to me why on earth spatial and temporal representations are related L

 

Email to Prof. Roy

Discarded

Hang on - he talks about self-organising expansion of the vocabulary - this should be more interesting. Finish reading the paper.

Yes, the vocabulary expands, using the the generative alphabet, but the meaning-set is still fixed.

 

As I think I said, I’m really interested in the origins of language, particularly the origins of communication itself, the role of discourse models (- this might be the wrong term for it, but I’m thinking in terms of the differing roles played by statements vs questions vs imperatives) and the emergence of syntax. I’d welcome any guidance. You said that you were keen for people to do projects rather than papers, and that suits me, but I’m finding it difficult to restrict myself to an interesting problem that’s remotely manageable within the time-frame. I’m also worried that I might be straying outside the bounds of what you have in mind for the course.

 

I took on board what you said about considering search in terms of lifetime learning rather than GAs/evolution, so I’m going to dispense with any genetic component. However, I’d like to avoid explicitly defining a teacher/target stimulus for the agents to base their concept of ‘nearness’ and ‘farness’ on, so I’m considering reinforcement learning.

I’m considering how it might be possible to extend this. One option would be to have different types/sizes of objects, with different nearness thresholds, or for the nearness thresholds to decrease with age.

from either the front or back (or left/right if you prefer).

what can I show with this???

 

I’ve certain assumptions, some of which are intended to be uncontroversial, and others included simply to make things more amenable to simulation:

 

That is, human-like language is impossible without human-like experience and limitations, and human-like embodiment, and more loosely,

I’m worried though that this conflicts with my assumption that language can only exist within a community of more than one speaker. After all, if language has some adaptive value for the individual beyond the value of communicating with other speakers, then we might want to say that one could evolve to be the only speaker of a language in order to benefit from something like an internal concept-structuring or ‘inter-system neural communication protocol’.

 

Language reflects the syntax/structure of the environment

This idea is a little hazy in my mind.

the subject-object conceptual scheme arises inevitably for agents out of their very environment-syntax

Received email from Deb

“I just looked at your proposal. First comment is that I had asked for only one page -- that helps force you to focus more than you have here. I think of your two final ideas, the second is better. We can talk sometime next week about how to make it a practical project. Lera's suggestion is to look at 'ahead' and 'behind' as a good starting point for time-space conversion.”

New ideas

021105

so it looks as though the syntax is eventually hoped to employ 3 or so orthogonal words

physically close distance

fast

front

back

relativeToYou

relativeToMe

I don’t think I’d want to half this number of words and introduce a not()/opposite() quantifier

in fact, I might even want to further unpack some of the words

perhaps to include the relativeToMe and relativeToYou as part of the words???

if the agents are moving, then that would presumably also affect the speed too??? argh. perhaps the speed of the agents could be fixed. that could be a sort of clock/means of grounding fast (i.e. fasterThanUs)

I suspect that if I want to look at just front/back (as Deb + Lera suggest), at the expense (at least initially???) of near, then I would need to have the objects permanently coming from one side

in order to preserve a parallel between spatial here and temporal present

but why (in reinforcement learning terms) would they need to communicate about objects behind them???

I suppose that they could be required to comment on the current position of objects while they’re within 10m either in front or behind, with one object at a time, so a comment to the effect of ‘object is inFront/here/behind’ is the same as ‘object is coming/now/gone’

I suppose that could work. how could I interestingly extend this???

how could I get the time/space representation mixup effects to come about???