Mark Erikson on Replay QA: from time-travel debugging to…

Mark Erikson, Redux maintainer and senior time-travel engineer here at Replay, joined Jeff Auriemma for part two of their conversation. This episode covers Replay QA: what it is, why we built it, and the technology underneath.

Episode summary

From debugger to agent. Replay started as browser DevTools with time travel built in. You record an app, then step backward and forward through it, inspect the DOM and React tree at any point, and add log points after the fact. The technology works, but most developers don't use graphical debuggers, so few people adopted it. Agents are different. When Replay exposed its time travel data through an MCP server, agents could investigate as persistently as needed. Mark tested this on public React bugs from Dan Abramov and Nadia Makarevich, and the agent found them.

Three layers. Mark describes Replay as a stack:

Time travel recording. Record a browser session, replay it deterministically, and query any point in time.
Replay MCP. Exposes that data to agents: stack frames, variables, React renders, Redux dispatches, network activity, and performance insights.
Replay QA. Runs the investigation workflow for you. Point it at your site and repo and it generates user journeys, runs and records them, and looks for bugs over time.

Why QA, and why now. AI makes writing code cheap. Review is now the bottleneck, and AI code review is still static analysis. Nobody has time to test what ships. Replay QA runs the app, then searches the recordings for broken flows, accessibility issues, vulnerabilities, slow React renders, and network waterfalls. It ships as a GitHub app and plugs into the PR process, like CodeRabbit or Qodo do for code review.

A real example. A customer's app wasn't loading inside Replay QA journeys. Mark pointed his agent at the recording, and it found the cause in about 10 minutes: the app passed an HTTPS URL to the WebSocket constructor, which Replay's Chrome 108 runtime doesn't support. Replay was already capturing WebSocket data, so Mark built a new MCP tool for it over a weekend.

React analysis without source maps. Replay's React insights used to need source maps, and only 5–7% of recordings had them. Mateusz Burzyński (Andarist) rebuilt the instrumentation to work directly on the AST. It detects bundler output from webpack, Vite, Rolldown, Parcel, and others, and instruments React inside minified production bundles. Every recording that contains React can now get render and performance analysis.

So Replay QA as a product is autonomous QA. We point agents at your website. We figure out what kind of website it is. Is it an e-commerce site? Is it Gmail? Is it a customer relationship manager? We generate a bunch of user journeys that would represent a person browsing through the website and doing various things, possibly kind of similar to what your Playwright tests would have been, if you even had any Playwright tests.

Full Interview Transcript

Lightly edited for readability: timestamps and filler words removed, obvious transcription errors fixed.

Jeff: Mark, welcome back to part two. Thank you again so much for joining the channel. Why don't you give a quick self intro, let folks know who you are, and we'll pick up where we left off.

Mark: Yep. I'm Mark Erikson. Most folks know me as the primary Redux maintainer, or that guy with the Simpsons avatar. I've been maintaining the Redux libraries for over 10 years now, which is still absurd every time I think about it. And then my day job for the last few years has been at Replay, where we've built a time travel debugger and a whole bunch of other really awesome technology on top of that.

Jeff: Awesome. Well, I would love to talk about Replay now. One of the new hotness that y'all are shipping is built as autonomous QA. What does that mean? And why can't I just tell my agent to do QA from scratch? Tell me more about Replay's QA product.

Mark: So, as usual, I'm going to go backwards and give a history lesson first and then circle around to where things stand today. Our two founders started as Firefox DevTools engineers. They had this idea that they could make recordings of an app being used and then deterministically replay that and use that to improve debugging capabilities. So the technology started in Firefox and then eventually spun out as a startup.

We've been around for a few years. Our original product idea was essentially browser DevTools with time travel baked in. In fact, the debugging UI started as a literal fork of the Firefox browser DevTools, which are, by the way, a really old React Redux application. When I joined Replay, my first year was spent modernizing our fork of that codebase.

So the idea is you make a DVR-style recording of your app, try to reproduce a bug. You have to use our special modified browser that has all the time travel recording functionality baked in. It makes a recording, it uploads it. The original workflow then was that you as an individual developer would open up our time travel browser DevTools on the website. We'd spin up the session in the cloud. There's a real browser process exactly the same as it was on your machine. You think you're clicking back and forth in the timeline to debug. It's actually making lots of forks of the process in the cloud and running them ahead. I can talk for hours about how the tech works.

So conceptually it's browser DevTools with time travel, and that allows you to do things like inspecting the DOM tree or the React component tree at any point in time, seeing exactly how many times any line of code executed, being able to step forward and step backwards in the debugger, or dynamically add a log point to a given line of code, which will then evaluate and print the expression at every time the line of code ran. We also built integrations with things like React DevTools, Redux DevTools, and various other awesome time travel powered features. Still legitimately awesome. There is nothing else like it in the web dev space. There are other non-web time travel debuggers out there. I spent a lot of time getting to build things with the time travel features.

Unfortunately, what we found is like 90% of developers don't actually even use graphical debuggers in the first place. And so trying to tell them that they should pay us money for a more advanced graphical debugger that also requires some additional setup process turned out to not go so well in the product department. Everyone who looked at it said, "Yeah, this is great, but I would rarely use this, and only for really hard bugs." And we also found that most people never spend any time thinking about how to debug. It's something most of us learn by osmosis over the course of our careers rather than taking a class on how to debug in college.

So: incredible product, still exists, legitimately incredible, very few people wanted to use it. We tried building an end-to-end test suite integration where you could configure your Playwright or Cypress suite once to use our recording browser to run the tests. All the tests make recordings and upload, and anytime there's a failure, you can open it up, view the test steps, and still do the investigation yourself, but at least you don't have to rerun the Playwright tests on your own machine again. We built it. It's awesome. People didn't really want to use it.

So we've ended up bouncing back and forth on what else we can build on top of that technology. And of course, AI is the hotness these days. So early this year, we built an MCP. And this actually had a couple of key advantages. Debugging is hard work. It takes persistence. It takes digging through all the nooks and crannies in your codebase, knowing how the pieces fit together. I've given conference talks on how to debug using a scientific method process. People don't like doing it for some reason. I don't know why, it's so much fun.

Jeff: Like, actually, legitimately fun. I think so. Yeah. The building is great, but also the figuring something out. I'm with you.

Mark: Yeah. So it turns out that agents are really good at this sort of thing. The point of having the time travel recording is that it allows you to dive as deep as you want or need to into the behavior. And we can treat the execution as data, because we built the whole time travel API to control the backend. You can tell it: give me a paused browser at this point in time. Tell me what the stack frames were. Tell me what variables were in scope. It's how we built all the different time travel functionality, like the React DevTools integration.

So what happens when you expose all this data as an MCP? Well, essentially you've rebuilt the Replay DevTools UI, but we've given all that power to an agent to do the investigation. And it turns out that when you can just point an agent at a recording and say, "You go investigate why this broke," they're very persistent and they will happily do all that. And when you give them data that no other tool is capable of providing, they are very, very good at finding bugs.

I actually did a couple of blog posts back in the spring, shortly after we shipped the MCP and I'd added some tools. Two case studies. Dan Abramov had filed a real React bug: the useDeferredValue hook occasionally got stuck. He was eventually able to file the fix with his agent, but it took him rebuilding the library with a bunch of console logs added so it could trace the behavior in prod. I took the example, made some recordings, pointed my agent at the recordings and said, "See if you can find the bug," and it did. Then I tried varying the prompts and instructions, and eventually it found the same bug faster.

Similarly, Nadia Makarevich, who's written a bunch of React tutorials, put up an example of "can LLMs actually find real-world bugs?" She had an example Next project that had increasingly complex issues. Her conclusion was, eh, not so much. I tried making recordings of her example app, again pointed my agent at it, and varied the instructions and the prompts and the skills. With those plus the recordings and the MCP, it actually found all the issues. So it was an intentional product plug, but also I'm legitimately trying to experiment and show that with access to the MCP and the data, agents can produce better results.

So the MCP is there, but it's still kind of low level. You've got to make the recording, you have to look at it. So what we've done is we've built a higher level product on top of that, and that's Replay QA.

Developers hate testing. We don't like writing tests. It takes time. Tests get out of date. Maybe someone on your team has a Playwright suite, and eventually the tests get out of date, and they're always running, they take a long time, and you don't know how many of those are still accurate. Or maybe you're a small agency and you've got a couple of devs maintaining a bunch of different websites, but you don't have an actual QA person to do the work.

And that was before AI. Now with AI, we all have a magic button that when we hit it allows us to generate essentially infinite amounts of code. We used to have to spend all this time planning and sprint tracking and story pointing and figuring out what is the right work to do, because writing the code was expensive. And then you do your pull reviews, and even that requires getting someone else's attention. And then they look at your thousand-line PR and say, "Looks good to me." We weren't really all that great at doing testing before AI, and now we all crank out multiple 5,000-line PRs a day, and the code review side is now a bottleneck. And then anything after that is a bottleneck. And ain't nobody got time to actually test all these changes properly.

So what we're seeing is: okay, if we're using AI to write all this code, let's use AI to review all this code. But that's still generally static analysis, which is valuable, but it's also not the code running.

So Replay QA as a product is autonomous QA. We point agents at your website. We figure out what kind of website it is. Is it an e-commerce site? Is it Gmail? Is it a customer relationship manager? We generate a bunch of user journeys that would represent a person browsing through the website and doing various things, possibly kind of similar to what your Playwright tests would have been, if you even had any Playwright tests.

We run those journeys. We also time travel record those journeys. And so then we can look for bugs. Obviously if a button breaks during the journey, then that's worth looking at, but we can also go back and look at the time travel data and look for other problems that could have occurred, whether it's accessibility issues, vulnerabilities, slow React renders, network waterfalls. We can deterministically look for a lot of those patterns by pulling them out of the time travel recording as data and surfacing them, and then we can do additional analysis to see if those are actual bugs.

So ultimately we are the bug investigation experts, and we've turned our expertise into a product that can do the automatic QA work for people that hook it up. And there's the sales pitch.

Jeff: That's very compelling, because a lot of folks are talking about agents in the code writing process, and now we're figuring out review and everything like that. But the way that you're making it sound is like there's this third bucket of what happens when you deploy it, and you can have an agentic strategy for that too, for regression testing, for everything like that. So is it really just pointing it at your production application and it kind of takes care of the rest? I'm really curious how the feedback has guided you all into how people use it.

Mark: Yeah, the very first prototype was just "here's a URL, go browse the website." We've fleshed all that out and productized it since then. It's now a GitHub app that you would install in your repo, just like you would with a CodeRabbit or a Qodo for code review. It's integrated with the PR process. It has a centralized bug database that we know across all the runs. It's able to look at those, see what's different in the PR, try to be intelligent about what it's actually surfacing, and detect new things as well.

Jeff: And am I correct in saying this is building upon the foundation of the time travel? It's using all the building blocks that were there in that v1 of the Replay suite and essentially putting that in the hands of an agent, not necessarily of a developer who, as you pointed out, might not be thinking about debugging all day.

Mark: Right. So let's call it three layers. There's the foundational time travel technology: the ability to make a recording and then replay it later, pause, ask for data, describe the call stack, interact with it, and build programmatic time travel features. There's the MCP layer that exposes all that data to an agent, as the equivalent of the DevTools UI for a person to look at. And now we've built the product layer on top of that, which says: rather than you connecting your agent to the MCP and saying "my agent, please go investigate," we've built the whole agentic investigation workflow already, and you just say, "Here's my website, here's my repo, go monitor this for bugs over time."

Jeff: So it builds on that special sauce of hooking into the internals of React or Redux, all the DevTools and everything like that, and it puts those in the hands of a machine that can make extremely rapid use of all that.

Mark: Yeah. Let me give a recent practical example. We had a new customer, just on Friday last week, come in and say: our app heavily uses WebSockets for transporting data to the front end. I tried to set up Replay QA and run some journeys, and for some reason in the journeys it looks like my app isn't even loading properly. The actual website's fine, but when Replay QA tries to look at it, it's not.

Our baseline recording browser is actually stuck on an old version of Chrome at the moment. We're on Chrome 108. We're very aware of this. We have been wanting to modernize to latest Chrome for a long time and have not had the time or priority to do so. We've got some internal experimental rebases that are very agentically driven, and we're trying to get the moving pieces in place, but there are thousands of lines of complex C++ Chrome internals involved. So we're hoping to get that done soon, but at the moment we do see cases where our customers are using modern browser features that we just don't support.

So the customer's journey made a time travel recording. I pointed my agent at it Friday night or Saturday morning, and it used the MCP and said, "Oh, well, the problem is the user is passing an HTTPS URL into the WebSocket constructor, which isn't supported until Chrome 125." And so the constructor itself was throwing an error. So, number one, the power of Replay MCP: my agent figured it out in about 10 minutes once it had the recording. Okay, well, what can I do about this? In Replay QA we can polyfill the constructor and at least make it cooperate.

But then, I had built several of our MCP tools in the spring, so I had some practice with this. I had exposed our deep React analysis and integration layer that we built: React renders, Redux dispatches, performance insights, a whole bunch of other tools like that. So I wondered, do we even expose WebSocket data? We've got a network requests MCP tool. Do we expose WebSocket data anywhere? And it turns out we were already capturing it in our Chrome runtime. We just weren't exposing that data anywhere.

So over the weekend I whipped up a WebSocket tool. I took the data that we essentially already had sitting around in memory unused, and I said, "Hey agent, build me a new tool that follows exactly the same patterns as the other MCP tools I've built." What WebSocket connections are there? How many messages? When do the sockets open and close? And if the contents of a given message are JSON, that could be large amounts of JSON, so let's not even send that back as a string. I'd already built a programmatic state tree inspector command for use with the Redux state, so let's just treat the JSON message as an inspectable object and allow the agent to drill down into the data structure if it wants to. I whipped that up in the course of about 48 hours and shipped it, and it's live.

Jeff: Super cool. Oh, that's awesome. My mind's racing on all the use cases.

Mark: I know, right? This is why I've been so excited about all this stuff for the last few years and trying to get people to understand how awesome this technology is. But that is part of the problem. As a geek, as a nerd, as a programmer, I get excited by the tech as well as how it can help programmers and save them time. Most people don't want to spend the time and effort to do debugging and QA and investigation themselves. That's why we struggled with product-market fit for a long time.

And that's why I think Replay QA is actually the right answer product-wise, because it encodes our expertise. It uses our genuinely awesome time travel abilities to do things no one else can. And it fits the AI market, because everyone's generating more code than they can look at, and that they don't actually understand in the first place. So why not have a tool that can do all that checking for you and surface bugs?

We're even actually looking at partnerships with a couple of companies that I cannot name at the moment. But let's say: looking at hooking into the software factory life cycle. If you've got a software factory that's cranking out thousands of apps, someone has to actually test those and make sure they work. But you can set up a feedback loop where we do the bug finding and then the factory does the bug fixing.

Jeff: Yeah, that's cool. I'm just thinking about our tendency, once something's out there and we've been working on it forever... It's like once I've given a conference talk that I've practiced a million times, I'm not watching the video afterwards, right? And so maybe that's the same thing with engineers. I've shipped it.

Mark: We need that parachute.

Jeff: We need the parachute. We talk to our users, obviously. We take their word for it and everything, but there's nothing like having that automated security blanket of knowing that there's something out there checking your work.

I'll change gears a little bit. I ask everybody on the show to do this: if you could show us something on your desk or within arm's reach and tell us about it.

Mark: So this is a golf ball. It looks like many other golf balls. I've lost many golf balls exactly like it. But this is the golf ball from my one and only hole in one in my life.

Jeff: Wow.

Mark: It was three or four years ago. I'm near Dayton, Ohio. This was at a course called Locust Hills. Locust Hills has two different courses on the property. Locust course, number eight, and I think they even had me playing the back nine first, so this was probably the last hole of the day. Uphill par three, about 160-some yards, elevated green. You can't really see the surface. I hit an eight iron. I flushed it. I saw the ball bounce once, and then I got up there and I couldn't find it. I walked around back and forth for five minutes trying to find the ball, and I finally walked past the hole and happened to glance down. It was in the hole, and I lost my mind.

Jeff: What a way to find out you hit a hole in one.

Mark: Anticlimactic. That one was worth keeping. That holds a place of honor up on the desk.

Jeff: Congrats. What a cool story, too. I didn't realize I hit a hole in one until five minutes later. There's probably a song about that somewhere. That's amazing. Thanks for sharing, that's special.

Mark, is there anything else that you want folks to know about or try or do before we say goodbye for now?

Mark: Please check out Replay QA. Generally it's a valuable product, and I think a lot of people would benefit from it if they looked at it. Check out the MCP. I've put a bunch of time in there, and I think even for the one-off agentic investigation case, there's a lot of good stuff there.

Okay, one other thing I need to babble about. I have been wanting to write a blog post for the last two years about the awesome React analysis technology we've built. Let me see if I can give the two-minute version.

Jeff: Yeah.

Mark: I originally built our time travel React DevTools integration for the DevTools UI product. It worked by injecting the React DevTools bundle into the paused recording every time React finished a render and extracting the components. I did a conference talk on it three years ago.

Jeff: Yep. Yeah, that was so cool.

Mark: So I did a conference talk about it at React Advanced 2023, and it worked. It was incredible. We built a much bigger, better, generalized version of this over the last couple of years. So work with me here.

The problem with that, and with doing deeper analysis, is we really need to know hundreds of thousands or millions of points of information for when certain functions inside of React itself got hit. Originally we didn't have a way to do that performantly. So what we ended up building is this instrumentation layer.

Here's how it works. We do one run-through of the recording to extract basic data: sources, breakpoints, network requests, all the stuff that is a known fact about this recording that could never change. Once we have that data, we can slice it and dice it and build cool features.

Originally, if your app had source maps, we would find the React bundle in the source-mapped list and look for 40 different function locations, like renderWithHooks. We would rewrite those functions to insert callbacks, like before and after your component runs. And then we would rerun the time travel recording with the modified minified bundle. We would extract the hundreds of thousands of hits and some extra data, and then we turn it into a data graph. Now we can say: do we care about this point? That point is right between this component's start and finish, so it must be while the TreeListItem component is running, instance number seven. We turn it into a data graph and query it, which gives us the list of all the renders: when every component rendered, why it rendered, what triggered the render pass, and how long it took. We've been able to build all the different MCP analysis tools on top of this data graph.

The instrumentation process originally required source maps, and then it would also do a bunch of Babel transforms to go from the readable source-mapped file to where that is in the minified bundle to insert the callbacks. Insanely complicated. It worked. I put in a ton of effort to make it work with both React 18 and 19, because the guts of React have kept evolving over time. You're familiar with some of these things.

Jeff: A little bit. Yeah. The 18 to 19 was a big leap in the internals. I could not imagine having to abstract over those.

Mark: I had multiple copies of React open and was trying to figure out: well, they used to have this logic here, and then it moved to a completely different function over here. Also, React never shipped source maps for production artifacts. I filed a PR to add that to their pipeline three years ago during the 19 dev process. It got merged, and then Andrew reverted it before they shipped 19. But at least they switched to having unminified production artifacts that still had function names and readable variables. Garbage names, but readable. So as of 19, you at least get a more valid-looking stack trace, and if you have source maps, you can see the readable artifact in there.

But a lot of apps don't ship source maps at all. And if the app doesn't ship source maps, then you still can't see the React artifact in there. It's just a minified bundle. One line, three million characters. That's it.

So one of my teammates is Mateusz Burzyński, handle Andarist. He's a legend. He works on the TypeScript compiler for fun. He maintains Emotion and a bunch of other projects. In the last couple of months, he built a brand new instrumentation system that is purely AST-based. It identifies all the bundle module wrappers in the output minified artifact. It knows how to find webpack and Vite and Rolldown and Parcel and all the other common bundlers. It looks for the patterns of all the different functions we care about within the minified code, and it does all the instrumentation with no source maps needed.

Which was great, because we were only seeing about 7% of our recordings with app source maps at all, which means we could only run the React analysis on 5 to 7% of our recordings. This works for any recording. If React is in the bundle, we will find it and we can instrument it. Which also means we can get these insights against production builds of an app with no source maps at all: the renders, the extra performance insights tool I've built that tells you what's expensive and what's slowing things down. This is actually really, really cool at a technical level, and I'm geeking out about that plus what it unlocks functionality-wise.

Jeff: Your enthusiasm is infectious. I want to try this. That's rad. Thank you for sharing that. Anything else that you'd like people to know about?

Mark: That's probably about it. I want to write a blog post about that stuff. I just need to find the time.

Jeff: It's so hard. We all got the same 24 hours, right? Well, it's amazing. Sounds like y'all are really hitting the hole in one out there, so keep it up. Mark, thank you so much for joining the channel and for sharing your time and insights with us. It's been great.

Mark: Yeah, thank you very much for having me on here. I appreciate it.

Want to see what Replay QA finds on your site? Start a project →