
Nov 21, 2026 · 1h 15m
AI cannot independently audit the strategies it helps build
SI427: Don't Let AI Mark Its Own Investment Homework ft. Rob Carver
As AI enters quantitative research, weak validation can turn plausible backtests into false confidence and crowded trades.
- 1AI can accelerate research, but data leakage, stale training data, and self-validation undermine strategy testing.
- 2Trend following remains a diversified source of resilience, while leveraged ETFs introduce reset effects, costs, and implementation limits.
- 3Rob Carver favors independently written, add-only backtests that preserve rejected ideas and keep humans accountable.
Don't miss
Rob Carver explains why an AI should not validate a trading strategy or code that it helped create, then proposes independent add-only testing.
The brief
Rob Carver and Niels Kaastrup-Larsen examine AI’s growing role in systematic investing, from research and coding to backtesting and portfolio decisions.
The discussion places that debate in a broader market context: trend following has remained resilient through weakness in energy, equities, metals, and other markets.
Carver warns that AI-generated strategies can inherit data leakage, stale training data, hallucinated references, and complexity that looks like sophistication but behaves like overfitting.
He also weighs practical choices, including leveraged ETFs versus futures, the risks of frequent algorithm changes, and whether asset-specific response curves justify added complexity.
The standout proposal is an independently checked, add-only research process: AI may help implement ideas, but it must not validate the work it helped create.
What was said on this episode
36 statements · 16 positive · 18 negative · 1 mixed · 1 neutral
AI validation of its own trading strategy can amount to marking its own homework
“effectively, it's sort of, I call it marking your own homework”
Listen at 0:08
Tesla cybercabs become unusable when owners stop paying subscriptions
“If you stop paying the subscription, then the car gets bricked, so you can't drive your own car.”
Listen at 3:35
Tesla cybercabs can operate as taxis only within a geofenced area
“Taxis can only be used in a geofence area”
Listen at 3:43
Tesla cybercabs cannot legally be used by owners to earn taxi income
“you can't actually use it as a taxi to earn money from because that's not allowed in the terms and conditions”
Listen at 4:32
Rob Carver reports his portfolio is up approximately 13.6% year to date
“I'm up about 13.6% for the year”
Listen at 11:47
Rob Carver evaluates his current portfolio as well diversified
“it's actually quite a nice diversified set of bets, which is what you really want, of course. We love diversification.”
Listen at 14:19
Similar AI outputs can cause investors to adopt crowded, similar trades
“if people are using AI models and those AI models are giving us the same kind of responses, then yes, that will lead to crowding because people will end up with the same trades on.”
Listen at 19:06
Crowding particularly harms relative-value trading performance
“some trades do very badly when they get crowded, like particularly relative value trades.”
Listen at 19:28
AI can create crowding when it gives many people identical advice
“the short answer to the question is it could lead to crowding if it does tend to give the same advice to lots of people.”
Listen at 22:01
AI may worsen investment position crowding
“AI may make it worse. Who knows?”
Listen at 22:43
Futures are probably preferable to leveraged ETFs for sufficiently large accounts
“if you can trade futures if you've got a reasonably large account then futures are probably a better bet”
Listen at 25:35
Leveraged ETFs provide a crude way to obtain equity and bond exposure
“leveraged etfs are a crude way of at least getting the sort of equity bond exposure in in your portfolio in a in a sort of reasonable way”
Listen at 25:42
Medium-term trading algorithms are probably better left stable
“I would say the question is, am I better letting it off run stable? Well, yes, probably.”
Listen at 27:46
High-frequency traders should continue periodically refitting their systems
“if Chris is in that space, then yes, carry on refitting.”
Listen at 28:27
Trying many fitting methods can amount to methodology shopping and overfitting
“you can end up doing what I call methodology shopping”
Listen at 28:40
AI cannot generate genuinely original thoughts
“the obvious answer is no it can't”
Listen at 31:08
AI can sometimes prompt humans to develop original thoughts
“what it can do though sometimes is help prompt you come up with an original thought”
Listen at 31:19
AI-generated research summaries can contain hallucinations
“there were some hallucinations in there so it wasn't perfect”
Listen at 31:46
AI will not produce genuinely original trading ideas
“it's not going to be coming up with genuinely original ideas. That's just not what it can do.”
Listen at 32:53
Performance gains from market-specific response curves may not justify complexity
“the gain compared to the complexity involved is a bit much”
Listen at 39:41
Rob Carver probably would not implement market-specific response curves
“not an approach I would probably implement myself”
Listen at 40:37
Investors hedging concentrated exposure should buy less matching trend exposure
“you're actually better off buying less of that in your trend-following fund”
Listen at 46:32
Trend-following crisis alpha is historically observed but not guaranteed
“the crisis alpha bit is not guaranteed but empirically that's what we've generally seen in the past”
Listen at 47:58
Diversified trend-following portfolios tend to perform better during market selloffs
“more diversified portfolios tend to do better, particularly in market sale events”
Listen at 49:11
Investors should generally choose diversified trend following across asset classes
“you're probably better off buying just a diversified trend-following fund that allocates across all asset classes”
Listen at 49:44
Trading systems should always be tested using pure out-of-sample data
“in my opinion is always to do a pure out of sample test”
Listen at 59:11
Sequential out-of-sample testing prevents information leakage into backtests
“There's no so-called data leakage or information leakage backwards into the backtest.”
Listen at 59:43
LLM-based trading systems are generally not refit annually in backtests
“I don't think I've ever seen anyone yet properly do a thing where you'd actually refit your LLM weights every single year”
Listen at 1:00:24
Effective financial use of LLMs requires substantial expertise
“You really need to know what you're doing. And this is the same with LLMs, in my opinion.”
Listen at 1:04:33
Rob Carver is highly concerned about AI-written code
“I'm very nervous about AI written code, generally.”
Listen at 1:04:44
AI can be useful for summarizing research
“i think it can be quite good also at summarizing research”
Listen at 1:05:41
AI-generated research summaries require independent verification
“You do need to read the underlying papers yourself. You do need to check that they exist”
Listen at 1:06:02
AI can effectively write small, independently verifiable code segments
“AI is quite good for writing small chunks of code”
Listen at 1:06:49
AI should not write a trading system’s core execution algorithm
“I wouldn't get AI to write my kind of core execution algorithm”
Listen at 1:07:00
AI-generated strategy code should undergo independently written robust backtesting
“that needs to then go into a robust backtesting framework that you have written yourself that you trust”
Listen at 1:09:00
Backtests should use an add-only process that never deletes previously tested ideas
“the backtest has to be what I'd call an add-only backtest. You can only add new ideas to it. You can't delete ones.”
Listen at 1:10:12
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Weekly email
State of the Pod, every Monday
The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.