Cryptoeconomics on Gnosis - experiment discussion

Knowledge Base Archive
1

5 Cryptoeconomic Experiments

We would like to empirically prove that adding manipulators (or noisy traders) to a prediction market built on the Ethereum infrastructure does not reduce average price accuracy, and that prediction markets will therefore be among the most manipulation-resistant.

We’ll be running the five experiments proposed by Vitalik Buterin to test the viability of Futarchy:

1. Experiment: Information is publicly available - no risk

We will launch a prediction market on an output of a smart contract on September 30th, 2017. The contract will be simply and verifiably coded to output 5.

We will then launch a separate prediction market where a specific set of users is rewarded if the average market price exceeds 5. The more it exceeds 5, the higher will be their reward. Therefore, manipulators are encouraged to try to manipulate the market upwards.

From an honest trader’s perspective, you can make a trade to bring the price back to 5 with a guaranteed return if the forecast is above 5. Therefore, our hypothesis is that the market will manage to keep the average market price at 5.

2. Experiment: Information is publicly available - added risk

Same as 1), except that instead of the simple output of 5, the smart contract will return either 0 or 10 this time, depending on the value of a random bit. We assume that the expected value will still be 5, despite the added risk.

From an honest trader’s perspective, you can make a trade to bring the price back to 5 with a positive expected value if the forecast is above 5. However, there still is a chance to lose your investment.

3. Experiment: Information about relative prices is publicly available - but not about absolute prices

We’ll be launching two markets: one where the smart contract pays ethereum_blockchain_difficulty / 10^15 and the other pays just ethereum_blockchain_difficulty / 10^15 + 5. The manipulators who will manage to push the difference in prices above 5 will be rewarded.

Let’s take the example of a company deciding whether to hire a given CEO A or a CEO B, and making a prediction market on its revenues for the next 3 years in the cases of
a) hiring CEO A
b) hiring CEO B.

Assume that you know for a fact that CEO A will increase a company’s revenue by $5 million. However, you do not know what exactly the company’s revenue forecast will be, and therefore, how big of a part of the revenue can be attributed to CEO B.

The difference of the two markets (CEO A and CEO B) should be 5, but if the price of market A turns out to be 17 and the price of market B turns out to be 10, you can not be sure whether 17 is the accurate price (this would mean that the overall company’s revenue is forecasted to be at $12m since the CEO A attributes for $5m), or whether 10 is the accurate price since you do not have any information about the company’s revenue.

*From an honest trader’s perspective, you **know that a trade exists with a positive expected value **if the price difference of the two markets is above 5. However, it is not clear whether the price for market A is too low or whether the price for market B is too high. *

4. Experiment: A few experts are 100% sure about something but can’t prove their knowledge to the world (I know it but I can’t prove it)

Same as 3), except that the difference of 5 in revenue that CEO A would add to the company is this time replaced with X that is only revealed to a select team. We will use an interactive protocol to commit to the constant so that the team members are convinced that we will input a value into the smart contract that can only be X. However, the protocol will be set up in a way that they cannot prove the constant X to others.

From the insider’s perspective, you **know that a trade exists with a positive expected value **if the price difference of the two markets is above X. However, it is not clear whether the price for market A is too low or whether the price for market B is too high.

5. Experiment: “Unknown Bias”
Same as 3) or 4), but we will randomize and privately reveal the direction in which the manipulators are being incentivized to manipulate.

While in experiment 3) and 4), market participants are aware that manipulators had an incentive to manipulate the market upwards, i.e. to increase the price difference between the two markets, the direction for manipulation is not publicly known in this experiment. Here, only a select group will be informed about the direction in which the manipulators are being incentivized to manipulate.

-From the insider’s perspective, you **know that a trade exists with a positive expected value **if the price difference of the two markets is above or below X. However, it is not clear whether the price for market A is too low or whether the price for market B is too high. -

Through these experiments and general tools we plan to build at Gnosis, we aim to forge a solid platform for DAOs (and other types of organizations) to use Futarchy to inform and automate their decision-making.

We hope to conduct the first cryptoeconomic experiment at the end of September and are looking forward to see the manipulation strategies of our community. Will you be able to beat the market?

2

I copied the above from an upcoming blogpost of the experiments.
For experiments 1-4 it is clear to me what exactly we are testing.
1 and 2 are just “baseline” experiments. 3 test wether JUST relative price information is strong enough to make the markets immune to manipulation. And 4 tests wether it is sufficient when a small set of people (experts/insider) have those relative price information (and can not proof it to others)

My open questions are:
How should we pick the “experts/insiders” for experiment 4? How representative can this be?

Also - what are we testing in experiment 5?

Especially experiment 3 with unknown incentives does not make too much sense to me. Because IMO the differences between the markets should never be >5 unless someone has the incentive.

More interesting would be the following to me:
Experiment 6:
the manipulator gets payed out if the average forecast is too high…
(now people could just based on that information short the difficulty forecast)
Experiment 7:
the manipulator gets payed - but only they know wether it is for a too high or too low forecast.

3

Hi,

here are some of my assumptions and understandings of the cryptoeconomic experiements:

In experiment 4 manipulators are getting payed to increase the distance between the two prediction markets and make the difference bigger than X.
In real life this corresponds to the situation, where people know that CEO A will affect the revenue by X in contrast to CEO B and manipulators are payed by CEO A to increase this difference.

In experiment 5 we have the same situation, but we evaluate the situation, where we do not know whether CEO A pays more bribes to increase the difference or CEO B pays more bribes to decrease the distance X.
I think experiment 5 is the closest to reality, because you do not usually know, who pays which bribes.

I think Vitalik proposed also for experiment 3 to reward manipulators. He did not stated exactly how, but this is just up to the final experiment design.

Vitaliks experiment proposal are targeting futarchy experiments. Your proposed experiments are more about prediction markets in general. I think experiment 6 is quite similar to experiment 2, besides the fact that the target to prediction is moving in experiment 6 and stable in experiment 2. The advantage of having something stable is that you can see the manipulation forces more clearly.

4

So in 5 we run 3 again with the only difference that we do not know what manipulators are incentivised for.

My point is that everyone will immediately know when the difference gets bigger or smaller 5. Basically because no one else would have an incentive to do such a trade…

To get a step closer to actual futarchy we should do (7) run experiment 3 but revert the market with the lower average forecast.
This is how it would be actually done in futarchy.
So again: we predict the difficulty on block x and the difficulty on block x plus a constant y. All trades of the market that will on average predict a lower number will be reverted. This should make it significantly cheaper for a potential manipulator because they only pay if they succeed.

5

Yeah, it makes only sense if we run experiment 5 as experiment 4 with the additional randomization of incentive direction. Running experiment 5 as experiment 3 is indeed pointless.

If the manipulators are very confident that they can manipulate the markets, they can do this also for free. They are just manipulating the second market downwards and let the winning, higher market predict the right difficulty, then all their “bad” trades are reverted.(but the manipulators have to pay when they fail to manipulate).

These kind of manipulation are not so nice, but we can simulate them in our experiment. But the famous warrent buffet attack for example is very hard to simulate: http://www.gametheory.net/News/Items/013.html

6

Now that I thought about it a bit, it seems to me that Vitalik’s idea of the experiment is flawed. An experiment of this sort works well only if done with big data, ergo, with pre-existing data. Otherwise its results will be both flimsy and unverifiable. Something also tells me that the solutions could be mathematically formalized. In other words, you decide what you want to prove and then come out with the real-world constraints that would make such result achievable. So you know how to shape your prediction market to deploy those same constraints. It’s even better if you let machine learning mould and remould the initial constraints. But of course, none of this is viable if your aim is an app to predict the future without affecting it — a pre-Relativity aim, I fear…

7

hey @BalsamoGian - good to have you here.

An experiment of this sort works well only if done with big data, ergo, with pre-existing data. Otherwise its results will be both flimsy and unverifiable.

I heard this argument a few times. I am not sure about it. Sure, ultimately you want to have much more data but even demonstrating it once is IMO a step forward. Lets assume you want to demonstrate that you have build a functioning plane and you are flying cross the English Channel - even doing it once demonstrates a lot.

If we create a challenge where everyone who could brake the system could earn a significant price - and no one does - that demonstrates a lot as well in my opinion. The next step after those experiments would be to create a challenge where everyone could play against the futarchy mechanism e.g. a game of connect4 (for a small deposit). If they win they would get a significant price (e.g. $1million). If this challenge is open for a year and no one can beat the mechanism - that is of course still not a proof for the mechanism but a very strong indicator that it works as intended.

8

Hi, Martin. I just mentioned you and Gnosis here, you’ll like it:

1 Like
9

Vitalik 's orginial proposal of the experiments, does not include market roll backs. But in the original futarchy concepts, there always need to be a market rollback. Example: If the futarchy want to decide between CEO candidate A and CEO candidate B and elects A by EBIT prediction in 3 years, then the market for EBIT with condition CEO B needs to be rolled back, because B will not be the CEO.

I have read in several places, that people could do arbitrage between the two conditional markets. But if one market is rolled back, then there is simply no possibility of riskfree arbitraging, right? Hence a manipulation of the “probably not selected market” will be very cheap, right? It will only be expensive to manipulate the market, if you are manipulating the winning chain.

I think, we could modify vitaliks 3rd experiement in the following way, to get it closer to the original vision of a futarchy:
We launch two markets:
market A : where the smart contract pays ethereum_blockchain_difficulty / 1 trillion + 5
market B : pays just ethereum_blockchain_difficulty / 1 trillion. Reward the manipulators proportional to the price difference (p_B-p_A)*k>=0, for some factor k>0. Roll back the market with with a lower average price at the end of the trading.

In order to learn something meaningful, we would need to start these markets with a small k and increase it over time. It would be very interesting to see, at which k the manipulators break the futarchy.

What do you guys think about this experiment setup? @mkoeppelmann, this proposal is slightly different than the one we talked about…

1 Like
10

We are making progress with the experiment setup.

We decided that the value of exeriments 1 and 2 is quite low and that we prefer to directly start with 3 but therefor run it multiple times. Note that GUARANTEED arbitrage in experiment 3 is still possible. (If the difference between the two markets is bigger than 5 it is possible so buy short tokens from one market and long tokens from the other)

We will also change the manipulator incentives to make the difference SMALLER. This is relevant for the second version of 3 (3b).

11

Summary of the next experiment setup:

Experiment A)

(formerly know as 3)

A scalar market will be created predicting the difficulty in 2 weeks. To pick the upper and lower boundaries for this markets will will look at all historical 2 week period changes in % and select the boundaries in a way that in 98% of the historical adjustments the new difficulty would be within the boundaries.

A second market will be created with the same boundaries but this event will resolve to a different outcome: the difficulty in 2 weeks + 5% of (upper bound - lower bound) (formally known as just +5).

For both markets and LMSR market maker will be funded with 10 ETH.

Another scalar event will be created that will predict the average price difference of the two markets with boundaries from 0% to 5% (WITHIN THE FIRST WEEK of trading). Short tokens from this event will be used to incentives a manipulator. 100 short tokens (max payout of 100ETH) will be auctioned of in a dutch auction.

We plan to repeat this experiment at least 3 times.

difference between expected outcomes = 5%
funding LMSR = 10ETH
max manipulator reward = 100ETH
relevant trading time = 1st week of trading predicting something happening a week later

Experiment B)

(Futarchy variant of 3)
Exact same setup as in 3. However - in this case only the market with an higher average forecast will be executed. The other one will be rolled back. (like it is done in Futarchy)

12

IMG_3250.JPG4032×3024 1.75 MB

IMG_3249.JPG4032×3024 1.87 MB

13

Hey,

could someone explain which situation of the real life the experiment 3A is referring to?
I totally get the 3B set up and that it is referring to the futarchy.

I do not think that one can do risk free arbitrating in the experiment 3B. Just image I am buying a long in the market A and short in the market B. If market B is resolved and I bought the long in market A too high, I will just loose money. There is no risk free arbitrage.
Hence, I think that in the experiment 3B manipulators will push the market predicting the difficulty nearly as high as the market predicting the difficulty + the 5 % without any risk.

14

could someone explain which situation of the real life the experiment 3A is referring to?

3A is not a real life exeriment - it is more to get a base line and to show that the mechanism is working in general (e.g. the contracts are bug free/ there is no wired unexpected way to manipulate the prices)

Yes - right. However - I also would not go so far to say the manipulators will push the difference to almost 0 because the closer they get to 0 the higher there is a chance to actually enforce the “wrong” decision and therefor “pay the price” of the manipulated market.

However - in case you are right - the next experiment C would be: the manipulators will only be rewarded IF the “wrong” decision is actually made. (this is much closer to a real world example)
So we will auction of “decision 1 is made” tokens. (In the drawing I called them “diffic. 1 token”)

15

I am very much in favor of this variant C - as I wrote above.
As you said, this is the closed set up to the real world.
We should run this experiment C several times, starting with a small manipulation reward and increasing the manipulation reward with each new experiment.

I think, we will see that the market will exactly be manipulated in the iteration, when the manipulation loss is smaller than the manipulation reward.

16

I mean - this is the tough part. How big the manipulation loss is depends mainly on the liquidity and this should get higher if the price not correct. So IMO there is no upper bound for the “manipulation loss”.

17

Yes, I completely agree.

Maybe it would also be nice to see how this liquidity changes if we vary the difficulty input difference between the two markets, which is currently proposed to be about 5%.

18

I thought about this experiment setup once more:

The convergence behavior of these markets is far from perfect, basically as explained above, markets can pushed for free to quite similar price levels:

But with a different setup, we counter this challenge and other attacks.
Here a proposal for an exemplary setup for the CEO A or CEO B decision:

We set up one market - lets call it “total-market” - predicting the market share price of the company in 2 years, unconditionally. We let this market trade for a little while and then close it.

After the closure, we set up another market - lets call it “difference-market” - predicting a scalar difference between the two options between CEO A and CEO B for the share price of the company in two years. A high difference value >0 would predict a market favor of CEO A and a low difference value <0 would prediction a market in favor CEO B.

If the difference of the “difference -market” market is >0, then the CEO A is selected and the payoff for difference-market will be according to the difference between the real shareprice in 2 years and the (total-market score + difference-market score of each user)

If the difference of the “difference -market” market is <0, then the CEO B is selected and the payoff for difference-market will be according to the difference between the real shareprice in 2 years and the (total-market score - difference-market score of each user)

Note that for this set up, we can execute both markets, the total- market and the difference-market. This is the significant difference to the proposed set up in Experiment 3B, where only one market can be executed. This is archived by changing the reference point : Our new reference point is the total-market score, which will be in between the price conditionally that CEO A is elected and the price conditionally that CEO B is elected.

One challenge with the new proposed set up is that the total-market might assume that the better CEO will be elected for sure and sets the total-market score so high, that it will not be profitable to change the difference-market into one direction away from 0. If there is such a convergence of the difference market towards 0, we could migrate this by changing the payout function from (total-market score + difference-market score of each user) to: (total-market score*0.99 + difference-market score of each user)