He taught my advisor John Rust in the 1980s, who passed on that influnce to me and he also had a direct influence of my research when I was reading his paper in the 2020s. How is this even possible?

His work is building on the work of classical dynamic programming (DP), which defines the optimal way to deal with sequential problem i.e. problems that involve thinking about the future state of the world.

Dynamic programming offers us a way to solve the problem if we have a perfect model of how the world evolves in a sequence. Prof Bertsekas’s work was about expanding the domain to cases where we do not have a perfect model of the environment and which must be learnt as we go.

This is called Model Predictive Control (MPC). We model the world, then make a prediction about where it will go if we take certain actions and then we do “Control” which is taking actions that give us the best possible world.

MPC is used widely in various industries from chemical plants to oil refineries, from data cooling centers to autonomous vehicles. It can be applied to advertising budget management, finance and other areas too.

Bertesekas also contributed to Reinforcement Learning (RL) which is mainly focused on learning without a model of the world and simply through trial and error. He was able to show that landmark progress in RL (e.g. AlphaZero) are deeply connected to MPC, and both are connected to DP.

Bertesekas was able to give us this powerful connection:

MPC = RL = DP

This is the closest we are to a grand and unified theory of solving dynamic problems from finance, economics to engineering. This essentially unifies planning and learning as fundamental solution methods to dynamic problems. He also taught a generation of students from 1980s right up to 2025, and that is 55 years of creating and motivating minds.

That is a hallmark of a true scholar and truely exceptional mind, and I want to take time to cherish this. I’m so glad that I educated myself enough to be able to see the value of this work.

Physics shows us that there is deep beauty in the universe, and now we see that there is a beauty to “solutions” to dynamic problems as well, they all follow the same “rhyme and rhythms”.

As we continue to work on problems that involve learning and planning, we cherish the work of Dimitri Bertsekas.

1. A one page ATS/AI friendly resume in standard format

2. Respond to openings quickly

3. full_name_resume please, and mention the role you are applying to.

4. In this transaction the following information was critical:

   1. The resume, linkedin and email

   2. Role information — what does this job actually entail and what could it lead to?

   3. Comp range — hourly wage, hours per week, total weeks

   4. Logistics: start date range, working hours, location/relocation, if remote and long distance then timezones and overlap hours

   5. Interview & followup availability, competing offers

   6. Visa situation

5. Anything less than “hell yes” is likely a “hell no”.

6. Show me verifiable evidence (public or private):

   1. Show don’t tell. Github, LinkedIn, working paper, website, blog, whatever. Add links wherever you can.

   2. Show me enthusiasm, promptness and decisiveness.

   3. Show me something that I can trust and I know is not hard to game.

As the recruiter here is what I paid most attention to:

• Your education, school and its rank [hard to game]

• Your work experiences (formal employments) [hard to game]

• Everything was game-able and not verifiable from my end.

Form the hiring side, there were things that I noticed I could do to make the candidates life a bit easier too:

• Be upfront and communicate / collect the critical information (see above). Candidates want that information too, so better to share it early.

• Humans are not machines, they take time to respond and often forget. This cuts both ways—attention is limited and things are often in a flux— so timely, gentle reminders are helpful.

• Don’t overpromise and manage expectations; usually there are many equally good candidates but just 1 role. In this economy, especially for students, in most roles that are not highly specialized, supply is just much lower than demand.

• Be quick about it: stringing someone along for a long time is really bad.

• Don’t use AI to write emails. Even if it takes a bit longer, and your email is shorter/not perfect; its better to write a personalized and heartfelt 2 liner than pages of flawless AI generated content. People want to see a human response.

• After weeding out the obvious misfits, having one short phone call or zoom interview is greatly helpful. Just a lot of context is brought up regarding the critical information.

• Collect information on what the hiring manager is actually looking for — often my own domain expertise does not match with theirs, so while I can prune out bad candidates using rules of thumb I actually do not have the ability to find that gem amongst candidates (all who on paper look equally good). So I think having a nuanced checklist of what verifiable signals (not mere keywords on a resume) the hiring manager is looking for.

On internationals with visa restrictions:

• Someone without visa restrictions is certainly much more preferable because there is no extra layer of hassle (this is very obvious!). International students face a lot more difficulty and will have to go the extra mile to find out the exact procedure that will allow them to work (the process). Unless its a big firm, a lot of firms don’t even know the procedure to hire internationals.

• However, paradoxically, someone with visa restrictions also looks like someone who would care enough to dedicate themselves to the job entirely. Not having those restrictions, actually might make someone seem more “distracted”.

On bias.

During this process I was very conscious about my own biases and presuppositions. All of us have biases and there is a lot of information that hiring decisions should not be made on: name, age, race, gender, caste, marital status, ethnicity, location, education, accent, rank of school, visa situation and so on. One is also worried that when AI systems will inherit this bias (since its trained on data with a lot of bias) and that could lead to algorithmic bias.

Bias exists for many reasons: statistical discrimination (using stereotypes as a way to save time), taste & preference (I just don’t like X), racism/hate, etc. and I’m not here to open that can of worms, but in general hiring should purge that information which could lead to bias. One useful I thing I did was to strip all such data and store the information in docs/.md files as candidate A, candidate B, etc. And I think that is better for both humans and AI systems, when evaluating candidates. I think as hiring & recruiting becomes more automated, we need a lot more research into both detecting and mitigating AI (and human) bias.

Some more thoughts on what excited candidates vs what the work actually was:

I think a lot of candidates think working in industry is sexy and they will get to work on cutting edge stuff because academia tells about so many clever things (in isolation). In reality, the actual industry work, if seen in isolation, is actually quite dry and dull. A lot of emails, data moving here and there, simple analysis, automation, etc. The process of building something real has many parts, and each of them might be boring if seen in isolation, but putting them together is actually quite exciting (because the final product is real and has force/materiality in the world)—so candidates should be excited about “the whole, final project” rather than narrower technical parts (a model, a tool or a pipeline).

On some other concerns I had about candidates:

• Did they seem to be having too much on their plate? If they were participating in 10 different projects then would they pay attention to this role.

• They all seemed technically sharp and diligent and would work when given perfect instructions from a senior expert, but often work is scrappy and there is not guidelines/manual and nobody to oversee you and so what I wanted to see with grad students was evidence that you could work without perfect instructions and without understanding the full context (can you keep chugging along?).

Hopefully these will be helpful to those on the other side.

1. EconIRL: This is suite of models and methods to analyze and understand sequential choices. These types of choices typically arise when there is something that “accumulates” over time. Consider the owner of durable asset e.g. a truck who must decide what is the best time to replace it (lest it break down on-route). Or consider, a taxi driver that is planning their route through a busy city, going from point A to point B; trading off traffic, time and quality of the road. Or consider, a user of a music app, deciding to skip or play the next song in a playlist. These models help us understand how exactly the user values now vs then, and what preferences drive his decision making at each step. If we can understand these we can then design routes, playlists or better pricing for durables.

2. PrefGraph: This is software that helps us detect “inconsistency” or “irrationality” in choices. For instance a common way that humans are inconsistent is when their choices change in the presence of “decoy” options which is an obviously bad option but presented to make other options look good. Another area where we might want to test for consistency is in AI agents or LLM responses. Ideally, we’d want our agents to be coherent and consistent, especially when they are to shop or reason for us. Note, that consistency is not the same as being right or wrong. Furthermore, often consistency can fail for other reasons such as changing preferences, a failure to a pay close attention to all options or simply making a ‘hand-trembling’ mistake. This software makes use of advanced graphical techniques developed in Operations Research.

3. DeepInference: This is a more statistical package that takes traditional statistical models such as the good old linear regression: Y = a + b T + error and makes it into Y = a(X) + b(X) T + error, so the slope and intercepts are now functions of X, which could be potentially very high dimensional and dense objects (e.g. image, text, sequences, graphs and so on). Well, in a world where such high dim. objects are first class citizens, we need to be able to work directly on them when studying empirical relationships in the data. And while it is easy to estimate such models, it is not so easy to run hypothesis tests such as H0: E[b(X)] = 0. We have to rely on statistical tools like influence functions to achieve that.

These packages make it clear that the main thrust of my PhD was to explore how computational machinery from machine learning could be used in service of economic and statistical modelling. And after 5 years, I have come to the conclusion that integration of machine and deep learning methods in economics is not only easy but is a natural direction if we want to model and work with dense objects (which are abundant in the social world).

Furthermore, I think a lot of interesting methodology remains buried in economics or econometrics papers or in arcane codebases, and it takes a significant effort and thinking to make these methods available to the public. However, I think there is value to this: the best validation of a method or tool is its widespread adoption and building open source software is the only way to do this.

The Bayesians learned this the hard way, but now with software like Stan we are able to run Bayesian workflows at ease. Similarily, if CS researchers had not come up with StableBaselines or the many benchmarks for robotics like MujoCo, it would be really hard to do research in the area of reinforcement learning where frameworks are very brittle and breakdown without careful hyper-parameter tuning.

Economics today does not emphasis clean, reproducible code and we do not really teach for this nor reward this in the publication process. But this is a vital aspect of making sure our work is not only reproduced and tested, but also that it is made available for others to build on, which in turn allows us to build on what they create using our work (a positive feedback loop).

The first is to work on problems at scale and spend less time on smaller, toy problems. The last 5 years were spent on small or medium scale, toy problems on which perfect solutions could be crafted and deep insights be gained. The real world is however much messier and does not permit such liberties. Everything takes much longer to run, requires going through more people, and dealing with all sorts of organizational and other resource constraints and barriers. However, solving problems at scale has impact that no toy solution can ever hope to have. In this way, my research activity can be more sustainable and my time spent on research can be more justified. This is the hardest strategy to follow as everyone, even me, loves to play with toys and one can really do a lot of interesting and clever things with toys. Working at scale can be sometimes boring, hard, and painful. But ultimately, it must be done.

The second is to keep learning more statistics (yes, pure stats!). For most people, their statistics journey ends at high school or undergrad. But unfortunately, this rabbit hole is very deep and never really stops paying rewards. From experimentation design, to uncertainty quantification to sensitivity analysis, learning more statistics has always helped sharpen thinking and enriched the discussion. Some people think statistics is “elementary” or “boring” or “oh, it’s just standard errors”. I think statistical concepts can be applied almost everywhere to improve analysis and rigor. As AI takes over more of the programming tasks, I think the one real differentiator between two scientists will be how much statistical depth one has. Statistics is also a great portal into algorithmic work and machine learning; each time we define a particular loss function to optimize for we are pushing in properties (convergence, uniqueness, instability, robustness) into the solution or decision rule or algorithm. And therefore understanding statistics really deeply is the best way to design better algorithms and systems.

Thirdly, it is imperative to work in teams as much as possible. Working alone is only helpful for geniuses or newbies. The context of the problem is often too spread across the organization for it to be possible for one person to “go it alone”. Working in teams forces you to pay attention to the final decision that will be taken from your analysis or algorithm, it keeps you focused on the relevant problems, not be distracted and more motivated. Working in teams keeps you updated with the latest know-how. Of course, the team ought to be smart, hardworking and inspiring. But working in teams solves half the problems that researchers face automatically, that it does not make sense any more to work on a problem alone. Working in teams gives you access to networks, people and advice that you can keep well beyond your time there.

And lastly, given the paucity of research time, one needs to work on one problem at a time and take it to completion. One of the problems I have faced is that nearly everything seems interesting and starting new things is easier than finishing old ones. Over my PhD, I have dabbled in too many topics to master any one. Real mastery is simply going to take more investment in one problem at a time. It takes time to absorb the problem properly, and properly appreciate both the problem structure and its proposed solutions. Monotony can be a real issue, so one can break the problem space up and work on different aspects of the problem, but it’s still better to work on one problem at a time. Some folks can manage 20 projects concurrently, but I’d spread myself too thin. Like most PhDs, I also find it easier to start projects than finish them (this is a very common issue in grad school), and the hardest part of the research project is in the final delivery, the “last-mile” so to speak. It is absolutely painful to finish, and the mind is screaming and kicking as one crosses the finish line.

With these 4 strategies, I hope I can still continue to do interesting work in a sustainable way. Of course, there is no way to compete with researchers who are paid to do only research. But competing with them was never the goal, the goal was always to solve problems and not accumulate publications. Why should the fish climb a mountain, when it has whole ocean to swim in? Industry careers are also long and give one unique advantages. First, is access to real, rich data and a means to experiment. Second, is compute and resources (Claude, GPUs, etc.). Third, is vital and deep context regarding real world problems. Fourth, is access to diverse domain experts from various backgrounds and a team who one can collaborate with. These advantages can be used to do impactful work, albeit at a slower pace, whose benefits could spill beyond the business setting (again, hopefully).

To recap, the mantra/heuristic I propose is:

• work on problems at scale

• study more statistics

• always work in teams

• take one project at a time to completion

These are neither particularly novel nor universally perfect, but they make sense for some folks. These guidelines maybe hard to follow, especially as this post has not elaborated on: a) how to differentiate toy vs problems at scale?, b) how to find a good team?, c) what statistics to focus on?, and d) how to commit to one problem only?. I do not know the answers to these, as these are things I will try to figure out in the coming months and years.

The long term vision is to help build and improve autonomous, intelligent economic systems and platforms that continually adapt, learn, experiment, and respect and facilitate the autonomy of humans and AI agents that populate it, especially regarding their incentives and motivations and cognitive limitations. Why does this matter? It matters because scarcity is a fact of life (time, money, attention, resources) and intelligent, non-coercive, economic systems help guide this allocation in a sustainable way. This knowledge/work is valuable because we do not know any society that cannot benefit from a better allocation of resources.

DoorDash, UberEats, GrubHub, Grab, Meituan, Zomato, Swiggy. These apps have been an important part of our lives. From late night binging, to last minute parties, to gifts send to loved ones, to groceries and more— on-demand food delivery has been a cornerstone of the conveniences we have come to expect as routine.

In this post, I’m going to peel back some layers on food delivery. We want to explore how it works. This article is just an introduction to this field of study—a touristy take of sorts—by no means am I qualified or experienced to talk authoritatively on these topics.

The Food Delivery Problem

Food delivery is complicated. Firstly, there are three distinct players in the market. Restaurants, who pay commission fees to platform, and want to deliver food in time. Couriers, who get paid a piecemeal wage for each order, and want to maximize earnings per hour. Eaters, who make orders at restaurants hoping to get it in time and want to get their food delivered quickly and cheaply.

Information about food orders arrives in real time, and is not known in advance and highly uncertain. Demand fluctuates. Orders must also be served within a very short time window, lest they get stale. There is no central depot (pickup point). Each restaurant is effectively a depot. Couriers are gig workers, so they choose their hours, may choose to reject order offers, and may also not follow the recommended route or order!

Unlike ride share, which is typically one-to-one matching between driver and rider, in food delivery we can bundle multiple orders at a single restaurant to a single courier, and drop them in multiple legs. We can also bundle multiple orders across nearby restaurants to a single courier aka “batched pickup”. One could also send a courier to a restaurant even before an order is ready i.e. an “anticipated” pickup.

Due to these factors food delivery is a hard logistics problem. The responsibility of the delivery platform is to coordinate the activity of all three (courier, customer, restaurant). First, they a) retrieve and serve search results for products, and then b) set prices (delivery fee, courier wage, commission fee) and lastly, c) dynamically match orders to couriers as they arrive. While not having any real control over the actions of the players, the platform seeks to be reliable, efficient, and profitable and with high throughput.

Demand-Supply Simulation

We create simple simulation where orders come in from customers for restaurants across a few hours of operation. We abstract away from the search problem and the pricing problem and focus on the matching problem.

Orders are allowed to arrive, accumulate— there is fundamentally more orders than couriers and this builds up. At fixed intervals of 5 minutes, a matching scan goes out. It picks up idle couriers and orders that are ready to be sent out, and then immediately matches them. Couriers are well behaved and follow the assignments and routes. They go to the restaurants, pick up their orders, and deliver them to their customers on the shortest path. There is a small amount of service time at pickup and drop-off.

The two distinctions are important: travel time and tour time are important. Travel time is simply the time taken from point A to point B. For example, it can tell us the time the courier will take to get to the restaurant. Tour time is the shortest time taken from origin, to sweep through a whole set of locations. For example, it can tell us the fastest way a courier at the restaurant can drop off many orders to all its customers.

Nearest Courier Matching

The simplest matching is to, in each batch, match the oldest order that is ready at a restaurant to the courier nearest to it. This “greedy” solution may seem silly, but may not be too far from the traditional approach to food delivery where each restaurant had its own couriers (who were usually nearby).

The graph above shows new assignments that happen every 5 minutes (not older ones). Orders are blue dots, restaurants are red squares and couriers are black triangles. The courier path is given by the dotted line and the GIF is a run through of 1 hour of simulation.

The problem with this method is that while pickups are quick, the drop offs might be absurdly long. This approach can exhaust couriers too early, leaving the platform vulnerable when demand picks up.

Bipartite Matching

A more “intelligent” matching is the bipartite or one-to-one matching, we essentially match exactly one order to one courier in a way so as to minimize total travel time. We can also call this, a “shortest delivery time” matching.

The objective to be first maximize total orders matched and then minimize the total time on all deliveries (courier —> pickup, pickup —> drop-off) and service time, and the constraints just reiterate that a) one courier cannot be matched to more than one order and b) one order cannot be matched to more than one courier. This is solved every 5 minutes.

The main advantage of this method is that unlike the greedy solution, it does not send couriers on very long drop offs. It is trying to reduce both pickup and drop off time together. The algorithm is very fast (essentially linear programming). Customers who are luckily to get their orders matched will see very short delivery times, since a dedicated courier is assigned to their order. However, the biggest limitation is that couriers simply matched to just one order, and this might be inefficient.

Single-Pickup Bundling

In this solution, for the same restaurant i.e a single pickup, if there are multiple orders, we can assign all of them (up to a certain cap) to a single courier. We simple bundle the orders and assign bundles to couriers via a set packing assignment. Couriers can pick up to 3 orders per restaurant and drop them off to their customers via a shortest path. Since orders can perish once ready, so we enforce checks to prevent food from becoming cold when bundled and taken on a multi-leg journey.

The objective function here is not to minimize delivery time but rather to maximize the total throughput or fulfillment (no. of orders matched). This allows a maximum of 3 times the throughput of the one-to-one matching. Delivery times are slower, since couriers are double or even triple booked, but we are able to serve much more demand. Sometimes couriers in the graph look “free” but they are working on prior assignments.

Batched Pickups / Network Bundling

An more interesting form of bundling will allow a courier to pickup from multiple restaurants. Here we allow couriers to pick up more than 1 order from across 2 restaurants. This is the “Double Dash” feature in DoorDash. The objective is to maximize orders matched, subject to the constraints that couriers can only carry up to 3 orders (that are ready to go) and orders can come from at most 2 nearby restaurants.

This allows us to do better than simple bundling which is tied to one restaurant.

Anticipated Pickups

So far we have waited for orders to be ready before matching them. However, typically we know about orders a little earlier than when they are prepared. Thus one way to improve the simpler bundling solution is to simply dispatch couriers to the restaurants even before the food gets ready and have them wait for 3 full orders to be ready.

In effect we are using a ‘forward-looking’ anticipatory window to maximize orders matches but consider orders that are ready now but, up to 3 orders, that will be ready in the future (e.g. next 10 minutes). This enables anticipated pickups.

To keep things simple, we do not allow pickups from across restaurants. But even so, its clear that the system is able to serve a lot of the demand. The waiting at restaurants allows much more bundling in general. Most couriers are working with 3 orders.

Performance Evaluation

The greedy strategy, of using the nearest courier, obviously does the worst. Despite being easy to execute and the couriers moving a lot, it delivers the worst avg. delivery time and worst fulfillment.

The bipartite matching is fast and tries to minimize total delivery time, and the effect is quite clear. Delivery times are lowest but fulfillment is not very high. Also, unlike the Greedy solution which also prioritizes “ready first” orders, the bipartite matching is more focused on delivery time so more orders perish without being matched. So a single minded focus on delivery time may not be wise.

Bundling multiple orders has a strong effect on fulfillment. While delivery time increases a little, the fulfillment rate jumps up sharply. Couriers are carrying about 3 orders each time. As a consequence, they are also moving more efficiently, showing lower courier distances. Bundling is done by set-packing algorithms since its many-to-one rather than one-to-one.

The batched pickup also is helpful. Even more fulfillment, even more bundling. However, the matching process is more computationally intensive (as can be seen by the run time), and avg. delivery time is quite high (as the courier has to move between restaurants). While this simulation was small enough to be “easy”, it may be possible that batched pickup might be an intractable problem at real world scale.

The anticipated bundling that knows when orders are ready is easily able to arrive at restaurant early and wait longer to pick up to 3 orders. This raises throughput, and reduces delivery times. Couriers are very busy; they are always going on very long journeys so end up covering longer distances. Seemingly the best of all worlds.

Its interesting to note that while simpler strategies like Greedy and One-to-One are more likely to have overlapping assignments, the most smarter methods likely do not seem to have similar assignments. Anticipated bundling and network bundling have about 5% overlap in terms of the same courier to same order.

All these results are specific to this simulation. But we can see a clear order where better algorithms which accommodate a wider search space do deliver better results. More sophisticated solutions will use rolling windows and forecasting to better match anticipated demand with anticipated supply.

Discussion

The first thing that I learned from this exercise is that the perfect solution to this problem is obviously intractable. The perfect solution is too complicated to even possibly fathom, and there may be many ways to think about it or frame it. The perfect solution would involve long term future planning, which means anticipating future demand and supply availability. It would take account some kind of network congestion. We might also want to use clever tricks like bundling, batched pickup and anticipation to improve performance. There may be other “hacks” that one could exploit. We would also want the system to be robust to the fact that most of the demand and supply is uncertain.

The second thing I learned is that each component of the system really interacts with other components. And so when you are making changes, you cannot simply rely on historical data on those changes. For example, if you want to understand the effect of a hike in the delivery fee, it may not be a wise strategy to simply look at historical data. One might even look at experimental data, but the reality is that food delivery works like a system. At the very least, one has to factor in the most important responses:

• The response of customers, which depends critically on delivery fees because that affects the final price that they see when they’re checking out their cart.

  • They may choose not to buy or buy something with lower prices.

  • They may reduce tips.

  • They may switch to other apps.

  • They may resort to “self pickup”.

  • They may choose to only buy from restaurants under UberOne or DashPass where delivery fees is zero.

• The response of couriers during peak hours to rises in their wages, which usually come with higher delivery fees.

  • Couriers may increase their availability, especially in peak hours.

  • Couriers may reject less.

• The response of restaurants to delivery fees is also crucial.

  • They may raise menu prices due to high delivery fees.

  • They may choose to downgrade their subscription on the app,

  • They may begin to use their own couriers.

Therefore it is likely that the hike in delivery fees may deteriorate the performance of current matching algorithm (whose performance has been benchmarked and calibrated to a lower delivery fee). For example, a reduction in courier rejection rate, due to higher wages, may present new opportunities to better utilize supply.

Overall, food delivery presents an interesting window into sophisticated pricing and matching methods; and understanding the dynamics of demand and supply.

References

1. Reyes, Damian, Alan Erera, Martin Savelsbergh, Sagar Sahasrabudhe, and Ryan O’Neil. “The meal delivery routing problem.” Optimization Online 6571, no. 2018 (2018): 2018.

2. Rios, Brenner Humberto Ojeda, Eduardo C. Xavier, Flávio K. Miyazawa, Pedro Amorim, Eduardo Curcio, and Maria João Santos. “Recent dynamic vehicle routing problems: A survey.” Computers & Industrial Engineering 160 (2021): 107604.

3. Yan, Chiwei, Helin Zhu, Nikita Korolko, and Dawn Woodard. “Dynamic pricing and matching in ride‐hailing platforms.” Naval Research Logistics (NRL) 67, no. 8 (2020): 705-724.

4. Zhao, L., H. Wang, Y. Liang, D. Li, J. Zhao, X. Ding, J. Hao, and R. He. “The first INFORMS TSL data-driven research challenge (TSL-Meituan 2024): Background and data description.” URL https://github. com/meituan/meituan_ informs_data, accessed: February 28 (2024): 2025.

5. Boysen, Nils, Stefan Fedtke, and Stefan Schwerdfeger. “Last-mile delivery concepts: a survey from an operational research perspective.” Or Spectrum 43, no. 1 (2021): 1-58.

6. Savelsbergh, Martin, and Marlin W. Ulmer. “Challenges and opportunities in crowdsourced delivery planning and operations—an update.” Annals of Operations Research 343, no. 2 (2024): 639-661.

7. What fees applies to my order, UberEats

Advertising is the backbone of the free internet. It’s what allows websites to offer content without paywalls, marketplaces to connect buyers and sellers, and businesses of all sizes to reach potential customers. For consumers, ads serve as a discovery mechanism, helping them find products they didn’t know existed or solutions they didn’t realize they needed.

However, despite having mountains of data about impressions, clicks, searches, and browsing behavior, measuring the actual impact of advertising remains notoriously difficult.

Consider what happens during holiday shopping seasons. People actively search for gifts, browse more frequently, and have high purchase intent. They also see many more ads simply because they’re online more often. Traditional attribution would credit all these holiday purchases to ads, showing massive returns on ad spend. But much of this purchasing would have happened anyway due to seasonal demand.

So correlation is not causation, and we have to experiment to test the impact of ads. However, even with randomized controlled trials, which are considered the gold standard in impact evaluation, getting clean estimates of advertising effectiveness is surprisingly elusive.

Randomization is not all you need

Imagine you want to test the impact of Nike ads on sales. The traditional solution would be to run an A/B test: divide your users randomly into treatment and control groups. The treatment group sees Nike ads, the control group doesn’t. Then you compare Nike sales between the two groups to estimate the advertising effect.

But the reality is that most users never search for Nike or even shoes. They won’t encounter a single Nike ad during their normal browsing experience for weeks or months. This means that in your treatment group, the vast majority of users won’t actually be exposed to a Nike ad, even though they’re technically eligible to see one. Similarly, in your control group, most users wouldn’t have seen a Nike ad anyway, even if they were allowed to.

So the difference in Nike sales between these massive treatment and control groups will be minuscule, and the true effect will be possibly undetectable.

The problem is that because you’ve blocked all ads for the control group, you can’t distinguish between two types of control users: those who would have seen a Nike ad if allowed, and those who wouldn’t have seen one regardless.

Adjust for Exposure?

Another approach is to estimate what fraction of users in the treatment group actually saw Nike ads, i.e. the exposure rate, then assume the same fraction would have seen ads in the control group. You can then mathematically adjust your estimate upwards by dividing by the fraction exposed, essentially scaling up your measurement to account for the dilution.

This approach has a flaw: it dramatically amplifies uncertainty. When you scale up a measurement by dividing by a small number (since exposure rates are typically low), the uncertainty of your estimate increases quadratically. On average you might be right, but your confidence intervals become so wide that the estimate becomes practically useless.

Adding to these challenges is the fact that advertising effects are typically small. Ads work through gentle nudges and repeated exposure, not dramatic immediate conversions like deep discounts or coupons might generate.

Placebos?

Another solution is the placebo ad approach. Instead of blocking ads entirely for the control group, you let the ad auction run normally, but at the last moment, when a Nike ad would have been shown to a control group user, you replace it with something else. Options can range from a public service announcement, a competitor’s ad, or a product feature highlight.

Now you know exactly which control users would have seen Nike ads because you tracked when Nike won the auction for them. You can create perfect comparison groups: treatment users who saw Nike ads versus control users who would have seen Nike ads but saw something else instead.

However, this approach brings contamination in the control group. Showing a different ad or content changes user behavior in unpredictable ways. A user who sees an Adidas ad instead of a Nike ad might behave differently than a user who sees no athletic shoe ad at all. A public service announcement might put users in a different mindset. The control group is no longer a clean counterfactual.

Simulation as Solution

Over the past decade, researchers at major tech companies have developed an elegant solution to these problems: ghost ads.

Under this method when a control group user comes online and triggers an ad opportunity (through a search, visiting a product page, or encountering an ad slot), the system runs two parallel auctions. The first is the real auction that determines what ad the user actually sees - and crucially, Nike is excluded from this auction. The second is a “ghost” auction that includes Nike’s ad.

The user only sees the result of the real auction, so Nike’s ad is never shown. But the ghost auction tells us something critical: would Nike have won this auction if it had been allowed to participate? If yes, we record a “ghost impression” - an impression that would have occurred in a world where this control user could see Nike ads.

This approach solves many problems. First, it eliminates dilution by identifying exactly which control users would have been exposed to Nike ads, allowing for precise comparisons with treated users who actually saw the ads. Second, it avoids contamination because control users see exactly what they would have seen in a world without Nike ads - the natural winner of the auction without Nike’s participation.

Implementation Issues

While conceptually elegant, implementing ghost ads requires careful attention to several technical details.

One challenge arises when a control user is in the control group for multiple advertisers simultaneously. Imagine a user is in the control group for both Nike and Adidas. If you run a single ghost auction including both, they might censor or crowd each other out in the ghost auction. The solution is to run separate ghost auctions for each advertiser. In Nike’s ghost auction, Adidas doesn’t exist, and vice versa. This ensures clean counterfactual impressions for each advertiser independently.

Another issue is that in marketplaces with thousands of advertisers, running ghost auctions for everyone becomes computationally expensive. Smart sampling strategies and caps on the number of simultaneous experiments become necessary for practical implementation. If one can store the exact configuration of the auction, and then one can run the ghost auctions later on in batch. This might be important given that there are strict latency constraints for real time ad serving.

Once these hiccups are resolved, calculating the average treatment effect on the treated is straightforward. It’s simply the average outcome for treatment users who saw the ad minus the average outcome for control users who received ghost impressions. This can be applied to any metric of interest: clicks, conversions, time on site, add-to-cart rates, or total spend.

Beyond Measurement

The ghost ads approach is particularly powerful when compared to traditional digital marketing attribution methods like last-click or first-click attribution. These older, simpler methods fully credit a conversion to the last (or first) ad clicked before purchase, ignoring the complex customer journey and multiple touchpoints that influence buying decisions.

However, the value of incrementality measurement can extend beyond reporting. These measurements become crucial inputs for improving the advertising ecosystem itself. Advertisers can use incrementality data to refine their bidding strategies, focusing budget on audiences and contexts where ads drive real behavioral change. Platforms can improve their ranking algorithms to surface more effective ads.

The ghost ads story illustrates an important principle. Having lots of data or even randomization isn’t enough. What is needed is thoughtful experimental design that goes beyond simple A/B tests.

Limitations

Of course, this approach requires significant engineering effort. And this approach is only possible from the point of view of the platform. So if you’re not the platform but merely the advertiser, then implementing something like this becomes much harder (though not impossible, a version called Ghost Bids does exist). Ghost ads also does not solve the problem of interference: users in treatment group can simply talk to their friends, who may be in the control group, distorting their behavior.

One alternative can be geo-experimentation, where you shut off ads for an entire region or geo; and then compare outcomes across other geos. However, that too requires the availability of different geographies as well as a willingness to experiment for extended periods in them.

References

1. Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867-884.

2. Johnson, G. A., Lewis, R. A., & Reiley, D. H. (2017). When less is more: Data and power in advertising experiments. Marketing Science, 36(1), 43-53.

3. Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193-225.

Planning helps.

Lets start by acknowledging that it would be nice if you yourself knew what you want. So take the time to pencil out, daydream through, whiteboard a plan even before you ask the LLM to do something. Having a very clear target is helpful for LLMs, as they are executers and do’ers and struggle with ambiguity and creativity.

It also helps to understand how LLMs work. Each time you chat to them, your whole chat history is picked up and fed back in. This includes any text they themselves produce. Think of a monster swallowing your entire chat every single time. Then they spit out words in a sequence. Everything in, sequence out.

So you can dump stuff into them in any order, but you cannot ask them to output stuff in simply any order. Generally, the LLM will understand this and only output stuff in a particular manner (e.g. not answering a math question in reverse). But you can help it considerably by asking it to do stuff in a particular order. By asking it to break the task down into logical chunks and steps that lead to the next. So break the task into smaller sequences.

Memory and attention.

Next, find out what the context window length is. Depending on which model you are working with, the context history can be very small or very large. For instance, the Gemini models have a huge context window while some GPT models have a very small one. A large context window means you can choke it with long text and it will remember everything mostly well. A small context window means that just after a few interactions—the critical instructions you began with have been forgotten and you will have to remind.

Beyond context window, which is a kind of long run memory, LLMs also have a short-run attention window. This means that you only have a limited budget of attention in each response. So if you bundle task 1, 2 and 3; then you may get poor performance on all three. But if you let it work each task in different interactions, it will pay more attention to the details of each task. Hence, it is even more important to break up the core tasks and plan before working with LLMs.

Context, specificity and prompt libraries.

It is easier to ask them to do something, rather than not do something. If I say, “Find me all the insulting words in this document”, then the LLM will deliver. But if I say, “Find me all the not so insulting words in this document”, then that could mean anything. In a lot of human conversation, for each assertion the negation is usually understood as per the context of the discussion. GPTs have no such context.

So context matters. Dump as much context as you can. Papers, books, docs, photos, email threads, your chat history, screenshots of your file structure, audio messages. I find helpful to speak out what I want and convert that to text — often that is easier than typing. Human’s slow speed in typing is a huge limitation to how well we interact with LLMs. Overcome that by spamming ctrl C + ctrl V.

Beyond context, specificity matters. Be vague, and get vague results. One way to help yourself is after providing context, ask the LLM to ask you highly specific and probing questions about the plan. This pushes the onus of thinking back to you. This will be painful as you will have to spend a lot of time clarifying and thinking about design choices.

Offer examples and prompts, that have worked for you in the past. Initially you will have nothing, but over time you can create a personal library of prompts and examples. Working with LLMs is a long term affair. You make it do something well. Then later you use that as an example. You find a good prompt and you save that for later. So we have to replace libraries of code / docs for libraries of prompts. There is no free lunch after all.

Diagnosis, Design and Execution

One simple way to work with GPTs is to take it slower. Try to separate tasks into diagnosis, design and execution.

Diagnosis is about understanding the crux of the problem or the root cause of the error we are trying to fix. Prompts like: “what is the issue in this code?” or “what is the problem here?” will help focus the GPT on pinpointing the root cause.

Once we have diagnosed the problem, we can design a solution. Here we can ask the GPT to offer options. You can try, “Offer some options?” or “Plan some possible solutions”. Then you can weigh in and choose the option you prefer. These design decisions contain important domain knowledge which GPTs do not have, and should be provided by you.

Execution is about how you do it (e.g. code writing, charting, table creation, etc.). My personal view is that GPTs are good for execution once you already know broadly what to do.

Varying strengths and common weaknesses.

Different models have different strengths and vary along different axis: general intelligence, coding abilities, writing prowess, memory and context windows, framing and communications, internal-tools, cost, etc. For instance, Claude writes well, Gemini remembers well and spits out text fast, Cursor is great for making many smaller edits at once and repo management, NotebookLM is great for crawling large corpus of documents to get you precise answers, Perplexity searches through the largest number of documents possible, GPTs has great internal-tools (Python command line, visualizations, etc) and has the most user-friendly interface. If you are using APIs, the cost structure of the APIs will also vary quite a bit (even within the GPT models). So try to pick the model as per the task (if you can afford it).

You should also know the common problems that LLM suffer from:

1. Sycophancy— the art of always being agreeable and agreeing.

2. Hallucination—straight up passing fiction as truth.

3. Limited memory — they may have simply forgotten what you initially asked for.

4. Calculation—despite recent progress, LLMs make calculation mistakes.

Try not to lose your mind.

So while LLMs have taken up the grunt work, for anything beyond simple tasks they force us to do the thinking. This is particularly unpleasant—as any one who has been forced to do thinking will attest. And reading—a lot of reading. Try not to lose your mind, but you probably will.

Recently, I read this old article, titled Mindless Statistics, which criticized the “ritual” of hypothesis testing as it was practiced at the time. It was interesting because it clarified a few distinct approaches. I thought I’d write a story around it.

Scene 1

Once upon a very bright and sunny morning I came across a strange coin on the road— it was glossy and had strange 19th century markings. Curious, I pocketed it. Google said it was a vintage gambling coin. Excited, I called up my local gambling den. They said that they would buy for a huge sum it if it were functioning perfectly— they could use it for gala nights. So now I really needed to determine if it was fair or biased. Not knowing much about statistics I decided to approach the experts.

Scene 2

It was high noon when I reached Fisher’s home. He was chatting with Gelman.

Fisher said “Assume the coin is fair. Then flip it a few times. Tally the number of heads and tails. Is this far from what you expect? If the coin was fair, how unlikely or surprising is a tally this extreme or more so?”.

So I flipped the coin n=30 times and saw k=19 heads. If p is the probability of the coin landing heads the chance of this happening is:

But this is not what Fisher was asking for. He was asking for the chance of observing 19 or more heads, assuming the coin is fair. That would be:

This is the p-value. At 0.10 it implies that if we repeated this n=30 flip experiment a million times, then 1 in every 10 would be k=19 heads or more.

So Fisher says, “Look, seeing 19 heads out of 30 has a one-sided p-value of 0.10 under fairness, so it’s not really a surprise. We cannot reject the notion that the coin is fair.”

Me (excited): “So is it fair, then?”

Fisher (sombre): “No, we just know that it could be. We only have evidence that we cannot reject that notion.”

Me: “Oh no! I wish had gotten just 1 more heads. Then I would have a p-value below 0.05!”

At this Gelman, who had been silent so far, quipped, “Not really. The difference between statistical insignificance (19) and significance (20), is itself not statistically significant.”

Frustrated with Fisher’s lack of definitiveness and Gelman’s hard-to-follow language, I did what is typically done in these situations. I took a second opinion.

Scene 3

I met Neyman and Pearson at their office in the afternoon.

After hearing me out, Neyman said: “Well, we can’t only rely on a “surprise” to take a decision. We must first specify the costs and benefits and what decision this test is tied to?”

Me: “I want to know if this coin is fair because I want to sell it for a lot of money. However, if the coin is actually fair but my test wrongly calls it biased, I’d simply lose the sale. I can live with that risk, so maybe we can tolerate up to 20% chance of discarding a fair coin.”

Pearson: “So your Type I error (alpha)—rejecting fairness when it is in fact fair—can be as high as 20%.”

Me: “On the other hand, if the coin is truly biased, I absolutely do not want to end up calling it fair—I’d be discovered as a fraud. Let’s keep that risk really low, say 5%.”

Neyman: “So your Type II error (beta)—failing to reject fairness when it’s actually biased—must be below 5%.”

Pearson: “Now we need to compute a sample size for this experiment. Before we do that we need to figure out how minute a difference can your gambling den detect.”

Me: “Well they are experts so they would immediately catch minute differences. I guess if the coin truly has p>0.51, they would be sure to detect it.”

Neyman: “So your minimum detectable effect is 0.01, your null hypothesis is H0:p0=0.5 and your alternative hypothesis is H1:p1=0.51.”

The decision rule will be a cut-off: choose H1 if k>=c and choose H0 if k<c.

Now under fairness, given a value of cutoff c, the probability of observing k and declaring the coin as biased when it is not (false positive) must be kept below alpha.

Now under unfairness, and given c, the probability of observing k and declaring the coin as fair (false negative) must be kept less than beta.

I scratch my head, how on earth does one find a n and c that satisfy thisPearson blitzes out some code and says: “I ran some computer simulations and found that n=15,482 and c=7,794.”

Me (shocked): “What! So many coin flips for something so trivial. Fisher just had me flip 30.”

Neyman: “That’s because you want to detect a tiny difference. We need much more data to test a difference of 0.01.”

Me: “Then what can I get with just 30 flips?”

Pearson: “If we allow only 30 flips, the best cutoff that maintains alpha at 0.2 is c=18, but yields a beta=0.79. So the test is underpowered as power = 1-beta. And if we use the commonly used alpha = 0.05, then the cutoff becomes c=20 but now beta soars to 0.94—power only 0.06! Your test will never detect the bias, even if there was!”

Now I did not have time to flip the coin so many times, so tongue in cheek, I thanked them and left feeling more frustrated than before.

Scene 4

Abraham Wald was smoking a pipe as I explained to him I simply did not have time to flip the coin 15,482 times.

Wald: “I see you spoke to the other guys. They want you to flip a fixed number of coins. That’s so last century. Why not collect data sequentially and stop as soon as you’ve learned enough?”

Me: “How do we do that?”

Wald: “Well we define some metric that measures the weight of evidence, and define thresholds. Then we flip coins and track this metric over time. If the metric breaches the boundaries we have sufficient evidence to call it, otherwise we continue flipping.”

For n flips and k observed heads, the likelihood ratio is given as,

If this is greater than 1 we have evidence for bias (p1) and if it is less than 1 we have evidence for fairness (p0). Fortunately, we do not need to compute this at each flip. If we flip another coin and find heads all we need to do is multiply this ratio by p1/p0, and if tails multiply by (1-p1)/(1-p0).

Wald scribbles some illegible math to show that thresholds can be derived as,

Intuitively, being cautious (low alpha) increases A. In contrast, being ambitious (low beta) reduces A. The reverse for B. So all we need to do is wait for this metric to either rise higher than A or lower than B.

Wald: “And so we just need to watch this ratio, which moves like a stock price.”

Being an expert stock price watcher and eager to finish this in time, I began flipping coins and tallying heads. I was very hopeful that I’d have an answer by sundown.

However, that was not so.

The metric never breached the threshold at all.

After 3000 flips this is what it looked like:

It just meandered on and on. And soon it was midnight.

Me: “Wald, I think I need to go home now. What if it never crosses the threshold? Can we conclude anything from what we have now?”

Wald: “No we cannot. If a conclusion was not found then you cannot stop flipping.”

Me: “But will this end at some point?”

Wald: “Theoretically, this could go on forever. It’s less likely though.”

My head was spinning. I had wasted all evening—and was no closer to finding out if this bloody coin was fair or not.

Scene 5

It was midnight, I was just getting home when a figure emerged from the shadows, wearing what looked like an old academic robe. “I couldn't help but overhear your conversations with my... more conventional colleagues,” he said with a knowing smile. “Perhaps you're asking the wrong question entirely.”

Me (startled): “Bayes! You scared me. Yes, I just need to know if this coin is fair”

Bayes (whispering): “Let me guess—you’re tired of yes/no decisions, thresholds, and indefinite tests? What if I told you that what you need is an interval where p should most likely lie.”

“Wow…Yes, that is indeed what I’m looking for.”

Bayes (draws closer): “But first, tell me what you deeply believe. In your heart of hearts, what are your apriori beliefs about p?”

“Eh? I don’t think the coin is either fully biased or fully fair, and since we have struggled to find anything so far, p does not seem to be too far from 0.5.”

“Let’s take a prior belief of p~Beta(10, 10), it is symmetric and narrowly distributed around 0.5, and update it with the k=19 heads you saw from n=30 trials.”

Bayes explained that if p is believed to be distributed as:

Then we can exploit the Beta-Binomial relationship to easily derive the conjugate posterior which represents our updated beliefs about p. Using the fact that we we saw k=19 heads out of n=30, p should be:

Bayes (excited): “We can find a 95% credible interval for p to lie in: [0.442, 0.712].”

Me: “Wow! But this… too wide for me to decide. Could I instead get to directly compare hypothesis p = 0.51 vs p = 0.50?”

Bayes noted that as I observed 19 out of 30, the likelihood of the null hypothesis is,

and the likelihood of the alternative hypothesis is,

Which is just slightly more. We construct the Bayes factor or posterior odds ratio:

Bayes: “And so there is a 0.53 probability of H1:p=0.51 and 0.47 probability of H0:p=0.50.”

Me (dismayed): “Great! I …wait…that’s not very helpful. That’s like flipping a coin!”

Bayes: “Well? do you have one lying around?"

Needless to say, that day I learned a lot about statistics but not if the coin was fair.

Epilogue

Frustrated with statisticians the next day I went to meet economists. But they laughed and said if the coin was valuable someone would have picked it up already.

Code.

TL’DR A toy demand-supply model shows that improvements to Generative AI will lower prices, increase sales, and generally increase AI engineers' wages and employment. Its effects on non-AI engineers depend on their substitutability with AI: complementary labor benefits, while substitutable labor suffers.

Goods Market

Industry demand is downward sloping:

Supply depends on the labor inputs of AI engineers, who create Gen AI content, and non AI engineers, who do other stuff. The level of Gen AI technology, represented by “A”, magnifies the output from the AI engineers.

The parameter “rho” measures the substitution between labor inputs. Examining input values that yield the same outputs (5, 10, 15), it becomes clear that when rho ≈ 1, inputs are fully substitutable on a one-to-one basis. As rho approaches -∞, the inputs become perfectly complementary, meaning either input can become a bottleneck.

Labor Market

Firms hire labor until marginal cost equals marginal benefit, given fixed prices and wages. This means that the demand for labor is falling in its wage.

The labor supply is an upward sloping function of the wage for both AI engineers (indexed by 1) and non AI engineers (indexed by 2). A higher wage causes more workers to pivot to that role.

Equilibrium

Prices and wages adjust so that demand and supply are in balance.

Numerical Solution

This gives rise to a system of equations with 6 unknowns and 6 equations, the solution representing the equilibrium. I select specific values for the parameters (a, b, etc.), utilize a numerical solver to find the equilibrium, and analyze how the equilibrium shifts when Gen AI technologies are introduced, specifically when the parameter “A” increases.

Example 1: High Substitutability (Rho = 0.5)

In roles like marketing content creation, entry level programming, report writing, and customer service, the role of non AI engineers is highly substitutable to what AI can create. The introduction of Gen AI here increases the productivity of AI engineers and firms will hire more of them. The need for non AI engineers falls, as they are substitutable, and they are displaced. Consumers are spoilt for choice and prices fall and sales increase.

Example 2: Somewhat Complementary (Rho near 0)

In some industries, the role of non AI engineers is somewhat complementary but also somewhat substitutable to AI. Example: analytics, software, etc. When the two labor types are more complementary, the productivity boost for AI engineers does not displace non AI engineers. Non AI engineers can remain in demand, and their wages and employment may rise.

Example 3: Very strongly complementary (Rho = -0.5)

There is also a situation where the complementarity of non AI engineers is so high that AI engineers get displaced! For example: legal or medical research, R&D, managerial work. The firm can do just as well, if not better, by reducing AI engineers, because the technical experts or managers are the real bottleneck.

Conclusion

An Econ 101 analysis contests the view that Gen AI is necessarily bad for labor. The more complementary work you do with respect to Gen AI, the more you stand to benefit. Consumers, on the other hand, generally stand to gain.

Disclaimer: It should be noted that all this is just an elaborate thought experiment, as I have left out many important real world aspects and parameter values have been cherry picked to make a point.

Code

1. Vernon Smith's classic classroom experiments on the double auction (bid-ask) market showing that even with little information and few traders, the market finds the price very quickly.

2. The Gode-Sunder result that market mechanisms are so powerful that even if idiots / dummies were trading in it, they would be led to efficient outcomes. Markets do not need rational traders at all.

3. Rust, Palmer, Miller's famous computerized double auction tournament which highlighted how information is revealed through bidding in the market and why passive sniping (wait, watch.. jump in to steal the deal) will always be powerful as long as there are a few active traders revealing information.

Obviously this does not imply that markets will always be efficient at a macro-level. This only shows the power of markets at a very micro level, with restricted scope.

I really like these papers because:

• They do not use an equilibrium model, but just use simple experiments and computer simulations to make very important inference.

• They also try to open the messy trading process instead of treating markets as black boxes i.e. automatically clearing and frictionless.

• Very easy to read and understand.

Auctions are complex because we do not know how much opponents value the same item (sometimes, we do not even know how much we should value it). This uncertainty makes finding the mutual best response strategies even harder. Further, every piece of information can be something others could respond to. Adding rounds and multiple items makes this even worse. As a result—even today, we do not know the strategic equilibria of many multi-item and multi-round auctions ordinarily found worldwide.

Reinforcement Learning

Reinforcement learning (RL) has shown remarkable success in finding powerful strategies in games of imperfect information. This raises an interesting question: Can RL help us approximate equilibria in auctions where analytical solutions are often intractable?

Reinforcement learning allows agents to learn optimal strategies through trial and error, receiving feedback from the environment. In games like poker—where players have imperfect information about the game state—RL algorithms have achieved superhuman performance.

Auctions as a Learning Problem

Auctions are typically modeled as Bayesian games, where each bidder has private information (their own valuation and other private observations), and public information and strategies depend on this information as well as beliefs about other bidders' valuations.

For example, in a first-price sealed-bid auction, the auction is a one-shot game where each player “i” draws a private valuation about the item:

This is their “state,” and they can bid using a strategy or policy, which is, in general, a probability distribution over bids that depends on the private valuation (mixed strategy). It could be deterministic as well (pure strategies).

And the reward for bidder “i” will depend on whether they won the auction.

Under first price rules, the winner pays the complete bid. The Bayes-Nash equilibria of this auction, given the total number of players N, is well known to be,

It is unique, symmetric, and pure (no randomness). It also implies that each player will shade their bids (be a little dishonest in their bidding). As the number of players rises, this bidding strategy nears truthtelling.

Our central question is whether reinforcement learning algorithms can converge to the equilibrium bidding strategy of any arbitrary auction simply through self-play.

Policy Gradient Algorithms

With a slight change of notation:

• States (s): Each bidder's private valuation and any public information.

• Actions (a): The bids they choose to submit.

• Rewards (r): The payoff received depends on the auction rules.

• Round (t): This can take value higher than 1 if the auction has multiple rounds

We can parameterize policies by making them Normally distributed, where the mean and deviation depend on the valuation. Some clipping keeps it within reasonable ranges (e.g., 0 and 1). But importantly, the mean and deviation can be deep neural networks, allowing us to express strategies vividly.

The agent will keep track of value functions representing states' value under that policy. We also have action-value and advantage functions that will tell us how good an action is compared to the rest.

The agent estimates these as they play the auction repeatedly. One way of expressing the objective is a weighted expected log-likelihood function. We’d like to maximize this by tweaking the parameters.

One way to make sense of this is to realize that if the estimate for advantage was 1, then this is a standard maximization of the expected-log likelihood of observing those actions at those states. This would mean we are re-estimating the same parameters that generated the data. By weighing the instances by the “advantage” of actions, we are pulling the policy towards parameters that generate more rewards. So, we compute the policy gradient by playing a few auctions and take a hill-climbing step:

Note that while the advantage function will depend on the policy, we treat it as exogenous (obtained from elsewhere) when taking the derivative.

Proximal Policy Optimization

Proximal Policy Optimization (PPO) is a state-of-the-art RL algorithm known for its stability and efficiency. Instead of log probabilities, it uses the ratio between old policies (which it currently follows) and the subsequent policy it tries to attain. So, the objective is to improve the probability of playing actions with a higher advantage relative to the current probability of observing them. This is a change in emphasis over seeking relative improvement rather than absolute.

Further, to prevent significant changes in the policy, we explicitly clip the objective if this fraction becomes too large or too small.

The result is an incredibly stable and fast algorithm that does not need to use second-order methods to explicitly create “trust regions” where to improve the policy.

Convergence Issues and Multiple Equilibria

Generally, best response dynamics (like the one above) do not have any guarantee of convergence, except in particular games like zero-sum or potential games (e.g. traffic congestion). They may oscillate forever and never find an equilibrium. For instance, in a simple game like rock-paper-scissors, best response dynamics will oscillate forever (even as the time average converges to the only equilibria: mixing 1/3, 1/3, 1/3). But if they do converge, they will converge to a Nash equilibrium.

This is the same guarantee that fictitious play has. But fictitious play approximately solved Poker. Interestingly, other algorithms, like counterfactual regret minimization, which are guaranteed to converge to coarse correlated equilibrium, could not solve Poker (initially).

Most of the auctions we study below have unique equilibria, so we do not need to worry about finding the wrong one. But generally, auctions can have more than one equilibrium, and if they converge, best-response dynamics will converge to only one of them, and that too if they are “stable”. But which equilibria would you be interested in? Unstable, weird ones? I’d prefer stable, ones that algorithms can find quickly—that suggests that humans can find them too.

Visualizing Learning

This is what learning by two PPO agents in a first-price auction looks like. The left graph has the mean bidding as a valuation function, and the right graph has the actual bids to valuations.

These are two PPO agents in the infamous all-pay auction where losers must pay their total bid. Not surprisingly, the equilibrium strategy is conservative when valuations are low and is b = v^2/2.

And this is for the second-price auction, which leads us to truth-telling b = v.

More Experiments

Asymmetric Valuations: The first price auction is when one bidder is a “strong” bidder and draws valuations from UNIF(0,1.33) while the weak bidder draws from UNIF(0,0.8). Equilibrium bidding strategies come from a complex differential equation.

Korean Auction is a multi-round restricted auction where you only know if you hold the highest bid. The only equilibrium for a two-player, two-round Korean auction is the sniping equilibrium, where agents choose to completely suppress any information in the first round by bidding 0, and then a standard first-price equilibrium arises in the last round. This was proved in a paper by my advisor Prof, John Rust. We see that RL finds this equilibrium as well.

Reserve prices are price floors that auctioneers can set to eliminate any bids below them. This is typically done to thwart cartels but slightly changes the optimal bidding strategy in first-price auctions. We see that RL can converge to this as well (reserve at 0.25).

Common values. In some cases, like oil rigs, often nobody knows the true value but receives a noisy signal about the value. Players also know that if we collected all the signals (the research others’ did), we could find a better estimate. In these auctions, bidders tend to be more aggressive and face the potential threat of the winners curse (if you win, you kinda paid too much). Here too, RL finds the equilibria where bidding is done as a function of a noisy signal.

Conclusion

While these are relatively simple auctions, where we know the answer already—it is helpful to know that RL can find the equilibria. Whether this method can scale up to more complex situations is an open question. But if we can transplant the success of RL in Poker, Chess, and Go into solving markets, that would do wonders for market design.

References

• Cai, Y., & Papadimitriou, C. H. (2014). Simultaneous Auctions and Computational Complexity. In Proceedings of the 15th ACM Conference on Economics and Computation.

• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

  Chicago

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.

• Rust, J. (2024). Explaining Early Bidding in Informationally-Restricted, Ascending-Bid Auctions.

• Krishna, V. (2009). Auction theory. Academic press.

  Chicago

There are two main reasons why additional information beyond treatment and outcome is useful in experiments:

• To obtain more precise estimates of the treatment effect

• To understand how the treatment effect varies across context

In this post, I will focus on the former and compare methods using Monte Carlo study similar to Negi & Wooldridge (2021).

Definitions

• i: The index of the individual in the experiment, N total.

• Y: The outcome that we are interested in. e.g. purchases, retention, clicks, etc.

• D: The intervention or lever that we want to use to achieve good outcomes. Takes values 1 or 0. This was randomly assigned to ‘p’ fraction of total individuals.

• X: The side information or context that was available. Also called covariates.

• Base machine learning models:

  • g: An LightGBM model for Y from X and D

  • l: An LightGBM model for Y from X

Synthetic Data

I will test two synthetic data generating processes: one simple and one complex. The potential outcomes is what would have been, depending on what treatment was assigned. The difference between potential outcomes are the individual treatment effects. For the K = 2 covariates, I can plot the surfaces.

DGP Simple

DGP Complex

Models

• B-OLS: The Bivariate OLS computes the simple difference in means of Y in the treated and untreated. It can be obtained as a regression of Y on D. Here we ignore X completely.

• M-OLS: The Multivariate OLS controls for X. This method was harshly criticized by Friedman (2008), for being biased and yet has always been popular (Athey and Imbens, 2017).

• I-OLS: This Interacted OLS was proposed by Lin (2013) to resolve the Friedman critique by allowing the treatment effect to vary with X.

• S-Learner: The Single Learner (Hill 2011) directly models potential outcomes from the data using a single machine learning model.

• T-Learner: The Two Learner approach (Foster, 2013) fits two mappings between Y and X, one in the treated sample and one in the untreated. Both mappings are used to ‘score’ the entire the dataset to obtain ‘what-if’ for each individual and then aggregated to find the ATE.

• X-Learner: The Cross learner (Kunzal 2019) tries to resolve the issues faced by the T-learner in imbalanced samples. We use the individual treatment effects from the X-Learner to fit two CATE models. The final ATE estimate is a rebalanced average of the individual CATE predictions.

• R-Learner (aka Double Machine Learning): The Robinson or Residual Learner (Nie and Wager 2020, Chernozhukov et al., 2016) residualizes Y and D using X, and then regresses the residuals to obtain the ATE.

• I-Learner: The Interactive ML or MLRATE estimator of Guo et al (2019) estimates the conditional expectation function and uses that with an Interacted OLS.

• DR-Learner: The Doubly Robust Learner (Kennedy 2023, Foster 2023) takes the S-Learner’s individual treatment effects but then correct it using a rebalanced estimate of the error.

Time Taken

The X learner takes the most time to compute (>1sec) since it has two cross-fitting procedures. The R, DR and T learners are next in line since they use two models. The S and I learners are relatively fast as they only need a single model. The OLS models are obviously the fastest, and can be made recursive, and would shine if latency was important (e.g. in High Frequency Trading).

Monte Carlo Results

I sample repeatedly from the DGPs and estimate ATE and obtain its sampling distribution. The experimental parameters are:

• p: The fraction of individuals that were given the treatment.

• N: The number of individuals in the experiment

• K: The number of covariates.

• q: Std of measurement error in the outcome.

• M: Number of Monte Carlo iterations

Baseline (N=10,000, p=0.3, q=0, K=2, M=50)

The OLS estimators generally have little bias but carry larger variance. The S learner is biased towards 0 but has the largest variance reduction. The DR, T, X and I learners are able to shave a lot of variance for minimal bias.

Imbalanced Sample (p=0.03)

The bias tends to rise in imbalanced samples. The M-OLS can actually increase variance. The overall best performance was from the DR and X Learner, and the I-Learner on the simple DGP.

Small Sample (N=500)

In smaller samples, bias rises. Here the I-Learner performs very well, followed by the DR and X-Learner. The R-Learner actually shaves off a lot of variance, but is much more biased. Notably, ML methods do not perform very poorly in smaller samples.

Measurement Error (q=0.1)

A small amount of noise in outcome measurement can bias the estimates towards 0. Here the DR learner did very well on both DGPs and I-Learner did well on the Simpler DGP.

High Dimensional Covariates (K=20)

Here the DR learner outperformed all others in both DGPs. The I-Learner also did very well on the simpler DGP. The M-OLS and I-OLS also did surprisingly well, despite not being designed for high dimensional covariates.

Cost-Benefit Analysis

As promised some of these estimators are able to reduce variance by 70%-90% at minimal increase in bias. What is best for the problem depends on the relative valuation of absolute bias (as % of True ATE) vs the variance reduction (as % of variance of B-OLS). Note, more bias is bad while more variance reduced is good. If I take ‘r’ to be a simple linear conversion factor:

And if we treat all the five scenarios above as equally likely to occur, then we can score and add up the quality of each estimator under a value of r. Then by changing r from 0 to 1000 we can discover the best and second best estimators.

• If variance reduction matters most, then the S-Learner or X-Learner is your choice. But this situation is practically less likely.

• But in general, the top two choices, in order, are:

  • Doubly Robust Learner which is robust to imbalances, high dimensional covariates and complex DGPs.

  • Interactive ML Learner which is more conservative and cautious about bias and more suited for simpler DGPs.

References

1. Athey, S., & Imbens, G. W. (2017). The econometrics of randomized experiments. In *Handbook of economic field experiments* (Vol. 1, pp. 73-140). North-Holland. Chicago.

2. Bai, Y., Shaikh, A. M., & Tabord-Meehan, M. (2024). A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances. *arXiv preprint arXiv:2405.03910*. Chicago.

3. Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters.

4. Foster, D. J., & Syrgkanis, V. (2023). Orthogonal statistical learning. The Annals of Statistics, 51(3), 879-908.

5. Freedman, D. A. (2008). On regression adjustments to experimental data. Advances in Applied Mathematics, 40*(2), 180-193.

6. Guo, Y., Coey, D., Konutgan, M., Li, W., Schoener, C., & Goldman, M. (2021). Machine learning for variance reduction in online experiments. *Advances in Neural Information Processing Systems, 34*, 8637-8648.

7. Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2), 3008-3049.

   Chicago

8. Künzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. *Proceedings of the national academy of sciences, 116*(10), 4156-4165.

9. Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique.

10. List, J. A., Muir, I., & Sun, G. (2024). Using machine learning for efficient flexible regression adjustment in economic experiments. *Econometric Reviews*, 1-39.

11. Negi, A., & Wooldridge, J. M. (2021). Revisiting regression adjustment in experiments with heterogeneous treatment effects. *Econometric Reviews, 40*(5), 504-534.

12. Nie, X., & Wager, S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. *Biometrika, 108*(2), 299-319.

Implementation Notes

• I have not tested methods that also use pre-experimental outcomes (e.g. CUPED, DiD, etc.).

• Since treatment is randomized, the propensity scores are identical and equal to p. If we are able to control for all confounders, we can use propensity scores in place of p.

• All methods above used out of the box LightGBM regressor and cross-splitting to prevent overfitting.