Experimental Market Design and Methodology: The Alaska Gas Pipeline

8 The experiments

In this section, we report on the process and results of the stage one experiments. Since this is an investigative study, we proceeded sequentially using an exploratory approach. That is, we chose a set of rules (an institution) and tested those rules in the laboratory environment. After observing the results and subject behavior we adapt the rules and perform a new set experimental sessions. We continue this process, and ask whether the modification leads to an improved institution.

This approach differs from the more standard confirmatory presentation. In the confirmatory experimental approach, the experimenter chooses a fixed set of institutions to study. The experimenter then tests each institution in the experimental environment. Nothing is supposed to change between sessions, including instructions, protocols, software and seating arrangements.

The exploratory approach is most often the way in which an experimental investigation proceeds. Regrettably, even when the approach is obviously exploratory in hindsight the presentation is frequently confirmatory. This is unfortunate for the reader who is left without a complete understanding of the process or the validation of the institutions that are presented. At each stage the presentation does not make clear what the information state of the investigators was, what was learned from the experiment and why the next step was motivated by the results.

All results are reproducible, that is, all results are available through the “programmatic construction of plots and tables (in contrast with most of the current methods that are equivalent to cut-and-paste methodologies and have the associated problems.)” (Gentleman:2007) All data manipulations, results, tables, and figures are maintained in a script. Any data corrections, manipulations or calculations are documented, so that any reader can reproduce and validate the presentation in this paper. We are loosely following the concept of reproducible research as described by Buckheit:1995; Gentleman:2007. The scripts will be made available on a website at a later date.

The ordinary presentation of data is a form of advertisement, there is a leap of faith required by the reader; the reader must believe that all manipulations and calculations were done appropriately and without error. The ability for the reader to easily reproduce presented results also allows for pier-review error checking, and allows the reader to view details omitted from the published presentation.

To make comparisons between mechanisms (institutions) we use two measures of efficiency. Allocative efficiency \(e_1\) is defined as :

\begin{equation} e_1(x,q) = \frac {\sum _i \left ( V_i^{\ast }(q_i)x_i \right ) } {V^{\ast }} \label {eq:e1} \end{equation}

and surplus efficiency \(e_2\) is defined as :

\( \seteqnumber {2} \)

\begin{equation} e_2(x,q) = \frac {\sum _i \left ( (V_i(q_i)-C_i)x_i \right ) } {W^{\ast }}, \label {eq:e2} \end{equation}

where

\(x_i\)

is 1 if \(i\) is allocated capacity, zero otherwise,

\(q_i\)

is the amount of capacity allocated to \(i\),

\(\boldsymbol {x}\)

is the vector \(\lbrace x_1, x_2, \ldots \rbrace \),

\(\boldsymbol {q}\)

is the vector \(\lbrace q_1, q_2, \ldots \rbrace \),

\(V_i\)

is the value of the allocation to \(i\),

\(C_i\)

is the cost of allocation to \(i\),

\(V^{\ast }\)

is the summation of the optimal allocated values and

\(W^{\ast }\)

is the optimal summation of allocated profit.

Surplus efficiency \(e_2\) provides a more sensitive measure, since it eliminates the scaling problems due to the large cost baseline of e$1200.

8.1 Observations and what we learned

The purpose of the results section is to impart what we have learned rather than to make declarative statements.

After a few sessions of software testing, all sessions are reported below. We make no ex-post exclusion as is common, software and other failures are reported. All pilot testing of software and procedures, and the first set of experimental results used student subjects, undergraduate and graduate students at George Mason University. The experimental sessions are presented in the order in which they were run.

Random allocations

In order to provide a reference point for the efficiency numbers, we computed the efficiencies of random allocations for each environment. The random efficiency was computed assuming that a large pipeline was constructed and 40 units of capacity was randomly allocated among the six participants. The mean efficiencies are presented in Table 2 and a histogram of efficiencies for environment SP1 is in the Appendix Figure 35.

We first report our results by sessions grouped according to environment and institutional rules, presented in the order in which the sessions were run. In the individual session tables below, efficiencies for a session are averaged over all outcomes including zero or non-allocation outcomes. The number of periods with no allocations are provided in the third row of the table. A summary tables is at the end of the section.

Initial experimental sessions

Our first set of experimental sessions used environment SP1 and the proportional mechanism; subjects were allocated capacity proportional to their bids with no restrictions on the update rule (bidding protocol). There were four sessions, efficiencies are in Table 3. Except for session one which had two non-allocations efficiencies are higher than the random allocation efficiencies. Figure 10 compares the histogram of efficiencies from the random allocation to the efficiencies from session one. Environment SP1 was an easy allocation environment, in that, there was no excess demand for capacity for the large pipeline. If each subject had revealed their demand for capacity, they each would have obtained their maximum possible profit.

Adding complexity to the design

The next set of experiments where for design YP, proportional auction with no improvement restrictions, the results are displayed in Table 4. We wanted to see if the additional complexity had an effect on the observed outcomes. The efficiencies in these sessions where low compared to the random allocation (80 percent). Adding the additional complexity significantly reduced the observed efficiencies, the surplus inferior pipeline was often chosen, see Figure 11.

To understand why efficiencies and profits were lower than optimal in session sets SP1 and YP, we looked at the individual bidding behavior. We found that subjects tended to over-bid in many of the periods. That is, if their profitable capacity requirement was 10 units, they would request 13 units or higher. This often seemed to be initiated by a non-allocation in an early round, so a subject would request more to try to get a pipeline built. However, many of the subjects reacted in the same fashion, which meant that capacity was over requested so the individual request were proportionally cut back. Subjects would then request more in subsequent rounds, so that they would not be cut back to a lower level of capacity than they required, see Figures 12 and 13.

Figures 12 and 13 show the sum of all the capacity requests (\(q\) bids) for each bidding round for all the non-practice periods. If each subject had optimally revealed their capacity requirement, the sum of bids should have been 40 in each round and period. The graphs show that the sum of bids was frequently much greater than 40, often 100 or more.

Figure 14 shows how overbidding affects the allocation in the proportional mechanism. The top horizontal line are the bids of all six bidders a price of 35, the capacity requests sum to 85, which is over two times the capacity of the large pipeline. Each capacity bid is reduced by ratio \(\frac {40}{85}\), so each bidder receives just less than half of their request. If an individual’s allocation was less than he required, he has an incentive to increase his capacity bid, but this also applies to all the bidders so there is an upward creep in capacity bids.

Figure 15 compares the histogram of efficiencies for the random allocation and the observed efficiencies.

Figure 16 Compares the efficiencies for the random allocation of environment SP1 with the observed efficiencies of session set one.

Figure 17 compares the efficiencies for the random allocation of environment SP1 with the observed efficiencies of session set one and session set two.

Using the proportional mechanism adding the complexity of the YP design reduced efficiencies, it became harder for the subjects to coordinate on a pipeline path (in the SP1 design, since there is only a single pipeline there is no coordination problem). A successful allocation mechanism must allow the subjects to coordinate, perhaps by allowing contingent or simultaneous bidding for pipeline paths.

Imposing improvement restrictions

Session sets one and two did not employ an update rule. In an attempt to improve the outcomes we imposed improvement rules. Improvement rules are routinely used in auctions, the standard English auction is a prime example (mccabe1990; mccabe1991; mccabe1992).

We imposed the both a downward and upward update rule in subsequent sessions of the proportional mechanism. Update rules usually impose the restriction that a new bid must be greater than the last accepted bid. Since the proportional mechanism we are using accepts capacity bids and not price bids; it was not immediately clear whether the update restriction should be increasing or decreasing. We are not aware of any evidence (anecdotal, field, historical, or experimental) or theory to support either restriction. So we did what every good experimentalist should do and tested both rules (mccabe1991).

We observed that in environment YP the update rules imposed in session set 3 improved performance above the results in session set 2 (also YP environment), the results are displayed in Table 5. Efficiencies improved, but were not much better than the random efficiencies. In both sets significant overbidding was observed, see Figures 12 and 13. There was still a number of misallocations, building the AB pipeline instead of the surplus superior AC pipeline. Figure 18 displays the efficiency measure one for session eight, which imposed the increasing improvement rule, there were four misallocations to the AB pipeline.

Figure 20 compares the efficiencies for the random allocation of environment YP with the observed efficiencies of session set 2 and session set 3.

Imposing restrictions, reducing complexity

Since the increasing rule initially appeared to be slightly better than the decreasing rule (by on one and two percent), we imposed the increasing restriction in the SP1 environment. We wanted to see if it also helped in a simpler environment. The observed efficiencies were not much different from those in session set one. We only ran one session in the SP1 environment. It was felt that resources were best spent investigating the restriction rules in environment YP, where the bigger improvement was observed.

Figure 21 compares the histogram of efficiencies for the random allocation and the observed efficiencies.

Figure 22 Compares the efficiencies for the random allocation of environment YP with the observed efficiencies of session set 2 and session set 4.

Improving instructions in a complex environment

Session set five, repeated the YP environment with restriction rules. In addition, we made improvements and added more examples to the instructions. The results show some improvement, perhaps due to the improved instructions. The role of the improvement becomes more ambiguous. The improvement of instructions and examples did however improve the performance of the proportional auction in the more complex environment YP.

If the proportional mechanism were to become the preferred mechanism, more sessions (larger sample size) would be needed to distinguish between the increasing and decreasing rules. However, due to resource constraints, it was necessary to begin investigation of alternative mechanisms.

Figure 23 compares the histogram of efficiencies for the random allocation and the observed efficiencies session set 5.

Figure 24 compares the efficiencies for the random allocation of environment SP1 with the observed efficiencies for session set 4 and 5.

Trying a new mechanism

Session set 6 was the first implementation of a clock auction. We implemented it in environment YP. In this version a clock was used for each segment (A, B, and C) so there was a price for each segment and subjects where required to enter a capacity demand (request) for each segment separately. If no capacity demand was entered for a segment within a specified length of time (four minutes), the subject’s last capacity demand was used. The proportional rule was used if the clock stopped with excess demand.

We used the following stopping rule. If there is a change in total demand for a segment, do not increase the price for; if there is a change in total demand for segment, increase the price by one increment. The auction stops (all clocks stop) when there is no change in demand for any of the segments. Subjects were allowed to change their demand either up or down; there were no update restrictions placed on the subjects’ capacity request.

Performance was similar to the proportional rule with no update restrictions and capacity bidding on AB and AC segments. There could be many reasons for the poor performance, some of the most obvious are proportional allocation, bidding on separate segments (A, B, and C) instead of bidding on pipelines (AB, AC), lack of update restriction, bad stopping rule, poor instructions, or bad software interface. In the next sessions we make changes to the clock auction to try to improve the performance.

Figure 25 compares the histogram of efficiencies for the random allocation and the observed efficiencies.

Figure 26 compares the efficiencies for the random allocation of environment YP with the observed efficiencies for session set 5 and session set 6.

Improving the rules of the clock auction

In session set 7, we changed a number of aspects of the experimental sessions. Making a number of changes simultaneously, allows us to conserve resources, and while it does not allow us to pinpoint specific elements of the implementation, it does allow us to exclude and reduce the set of possible effective elements.

Using environment SP1, allowed us to focus on the rules in an environment that was less complex for the subjects, reducing the possibility of poor instructions, poor software interface, and subject confusion. I emphasize confusion since the term is often an euphemism for subject stupidity—if you want to believe that, or use it as an excuse. However, it is “ultimately the responsibility of the experimentalist to ensure that the subjects are not confused”. If subjects are confused, it is likely due to poor instructions, a poorly designed computer interface (if computers are being used), or poorly designed experimental protocols. This is one reason in an investigative protocol we work on improving the instructions, interface, and protocols between sessions. But to be useful it is important that these changes are documented and presented with the design changes.

We also imprinted a new stopping rule, the Opt Out rule that was discussed above.

The results of session set 7 showed a dramatic improvement compared to the results of session set 6 (YABC environment) for those periods where an allocation was made. There were several periods where no allocation was made and the efficiency 2 measure was negative (this mostly indicates that subjects paid more than their value, earning negative profits for that period).

Figure 27 compares the histogram of efficiencies for the random allocation and the observed efficiencies.

Figure 28 compares the efficiencies for the random allocation of environment YP with the observed efficiencies for session set 6 and session set 7.

A failed attempt at more clock improvements

The next set of sessions, set 8, shown in Table 10, used a more traditional clock mechanism for environment SP1 (mccabe1990). The clock was stopped when there was no excess demand, total demand was less than the capacity of the small pipeline, or all the subjects Opted out. We also improved the instructions. Changing the instructions was necessary due to the change in rules. Also, an improved display was implemented, that provided subjects with information on whether the allocation based on the current set of bids would cover the cost of a pipeline (and which pipeline small or large). Unfortunately, errors in the software prevented some periods from concluding or making an allocation. However, there was a significant improvement in efficiencies for those periods that did finish without error (97 and 94 percent). This encouraged us to correct our errors and run some more sessions with the same rule set. Figure 29 compares the histogram of efficiencies for the random allocation of environment SP1 with the observed efficiencies for session set 7 and session set 8.

Successful implementation of improved clock rules

The final design of this group of experiments was the improved clock, implemented in a more difficult environment, see Table 11. For session set 8, the rule set included the improved information and the excess demand Opt Out stopping rules.

The efficiency results are the best yet and there is an allocation in every period. Most impressively, in the three sessions there were 14 allocations with 100 percent efficiency, which no other rule set was able to achieve.

Figure 32 compares the efficiencies for the random allocation of environment SP2 with the observed efficiencies for session set 8 and session set 9.

At this point, we do not know if the results will be observed in the YP or SP1 environment. Nor, can we state with certainty that results are due solely to the change in environment. More sessions are needed to test these possibilities. We ran out of resources.

Summary of efficiencies

All the efficiency ans surplus measures are provided in Table 12. The efficiencies reported below include all periods, including periods with no allocation. Graphical display of efficiencies are presented in Figures 33 and 33.

8.2 Summary of observations

The proportional mechanism tended to encourage over-bidding for capacity, and often misallocated or did not supply a pipeline. An update rule improves performance, constraining new bids to be higher than the last bid slightly improves efficiency, see Figures 12 and 13. Increasing the complexity of the environment significantly reduced the performance of the proportional auction.

In the more complex environment Y1, creating an improved set of instructions and examples improved performance. This shows that care must be taken in presenting the experiment to the subjects, especially in more complex environments.

For the clock auction, the Excess demand rule with surplus maximization gives best results. The Hybrid clock rule does not perform as well as demand clock. Ability to opt out improves performance. It appears that bidding on paths (AB, AC, design Y1) may be better than bidding on segments (A, B, C; design YABC) auction, but there were too few sessions to make a conclusive determination. Use of the low accepted bid for price, improved results, but poses its own set of problems (mostly not being able to cover costs).