S5E2 Innovative Tool To Keep Docs in Sync with Your Product with Manny Silva

Zohra Mutabanna: 0:04

Hello folks, welcome to season five of Inside Techcom with Zahra Mudabana. This season we are focusing on tools, tips and strategies to elevate your craft. Let's dive right in. Hello folks, welcome to another episode of Inside Techcom with Zahra Mudabana. Today I have Manny Silva. He is the head of docs and UXW at Skyflow. We're going to find out what UXW is. Skyflow is a data privacy vault company. His claim to fame is he's a technical writer by day and an engineer by night. He's also the codifier of docs as tests, a tool agnostic strategy for keeping docs and their products in sync. By using doc content as product tests, we are going to demystify all that I just explained. Hey, manny, welcome to my show.

Manny Silva: 1:01

Happy to be here. Zora, Thanks for having me.

Zohra Mutabanna: 1:03

Manny, welcome to my show. Happy to be here. Zora, thanks for having me. Absolutely, it is my pleasure and I am excited to be talking to you about something that we think about but don't know how to address, or more in an automated way, or I think that's what it is sounding like to me. But, manny, please tell us a little about yourself. Sure Like you already said, I am head of docs and UXW, which is UX writing at Skyflow.

Manny Silva: 1:30

That was so close. I like to think that my title is really head of words. My team name internally is Verba words in Latin, and so most things that are written that aren't owned by marketing fall under my purview. Now, outside of Skyflow, I've. So Skyflow is a privacy and security company. We offer data privacy vaults, but I've also been at big companies like Google. I work for a Apple subsidiary called Claris when I was first getting started. I've been doing tech writing for a decent while now, but outside of tech writing. I am a father of three. All of them are five or under, so my life is chaos, but I'm also a tinkerer. I have a server rack in my garage. I have multiple 3D printers, but I also like gardening. I like tabletop gaming like Dungeons and Dragons. I have a background in martial arts as well as musical theater, so I'm kind of all over the place.

Zohra Mutabanna: 2:23

Wow, how do you manage to do all of this with three kids that are five and under? It's a beautiful age, but hats off to you. I don't think I could manage that. I have to, and there was no way I could have. I just needed sleep when they were that little, honestly, so hats off to you.

Manny Silva: 2:42

Well, it's funny you mentioned sleep, because it's actually because of a lack of sleep and my little ones that doc says tests is even a thing oh, so they are the inspiration behind it awesome no, but the way things ended up working out was doc says tests is actually a byproduct of a project that I work on that I created called Doc Detective, and the way Doc Detective came about was I was doing sleep training with one of my little ones and I was up all hours of the night and there was a lot of yelling and screaming and I certainly wasn't getting any sleep, and I was having trouble doing anything that required synchronous thought, like reading or listening to podcasts or an audiobook or anything like that, and I thought to myself okay, I may as well try to use this time productively. So what can I do in this time between sessions of going in and trying to put my little one down? And so I came back to an old problem that I had talked to people about at Google, I had talked to engineering friends about, and that is that documentation inevitably goes out of date. Something breaks, something changes in the product, so that a piece of documentation that was previously accurate no longer is. And even if you have very good channels of communication between yourself, your engineering teams, your UX teams, your product managers, inevitably something slips through the cracks, and every time I brought this up to somebody, especially the engineers.

Manny Silva: 4:24

They're like you're right, manny, that is a problem, but that's why we have you here. It's like no, I'm not here to fix bugs, I'm here to educate users on the product that you are creating. I'm here to help you. I'm not here to get myself into a mess where I have to fix my own docs every week. All the same, they didn't try to create anything. It wasn't their problem. They weren't invested enough. So when I was up late at night, I was like okay, I have a little bit of experience with programming I'm self-taught but I decided I wanted to try to take a crack at it. By the time the five weeks of that session of paternity leave was over. At that point I was able to write a test and run it against an interface in a web browser. Things have come a long way since then, but that's where it all started.

Zohra Mutabanna: 5:19

Awesome Now. So Doc Detective is a tool.

Manny Silva: 5:23

Yeah, so let me separate the concepts for a moment. So, though it came first, DocDetective is the tool. Docdetective is a tool that lets you write tests that can then run against interfaces like user interfaces right now in browsers, but eventually it'll also run against native apps, mobile apps, that sort of thing. It has the backend tooling for it. I just haven't fully built that portion all out yet, but in addition to UIs, it can also ping APIs. It can also run native shell commands so you can test CLIs and SDKs. It can run arbitrary scripts, Regardless what kind of interface your product has aside from native interfaces. For the moment, Doc Detective can help you validate that they are accurate, that your documentation for the product matches the product UX itself, thereby making sure that the UX is in sync.

Zohra Mutabanna: 6:21

Awesome. Now let's talk a little bit about DocSys tests and then revisit how one can use DocDetective. So I want to talk a little bit more about DocSys tests.

Manny Silva: 6:32

So DocSys tests was a result of me trying to figure out what category DocDetective fit into as far as tooling was concerned.

Zohra Mutabanna: 6:40

Right.

Manny Silva: 6:41

Because there really wasn't a category that it fell into and I was trying to explain to people what Doc Detective did and they kept looking at me with blank stares and so I took some time. I thought about it. I was like, okay, you know this, we have Docs as code. We have there's the idea of Docs as ecosystem, but this is something else. This is docs as tests. This is a way of using documentation as tests for the product. And that's great that doc detective fits in that bucket. But it's a bucket and it can does and I want to see it can and does fit more than just doc detective and I want to see more than just doc detective in that bucket.

Manny Silva: 7:23

And Docs as Tests is a tools agnostic strategy for using docs as tests. Very literally, there are a few core principles that make it up. The first is that docs are the tests, Whether or not they are programmatically executed. A procedure that you write in documentation is an assertion that your product works a certain way, that you can follow those steps successfully. For docs as tests, the whole idea is to make these assertions programmatically testable, so you don't have to have somebody sit there and step through every single step of every single procedure that you write, you can have a computer do it for you, and the strategy is well, how do you go about doing that? What the tooling do you use to help you do that? And even if you don't test your docs programmatically, your users are going to do it for you and historically this is what's happened.

Manny Silva: 8:21

Your docs are always tests for the product. It's just, instead of having our users be our testers, we're having our computers be our testers, and that helps move the issue detection earlier up in the process, which has a lot of benefits that we'll get into a little later. But another principle is that the tests are run against the product itself. It's really important that your docs, when you're running them as tests, actually test the thing that they're documenting, Not a mock, Not running in a different environment that differs in some way from what your users experience. It needs to run against what your users experience to make sure that your docs are technically and factually accurate.

Zohra Mutabanna: 9:07

In an ideal world, I can imagine this working beautifully before you release the product and the documentation. Yep, and thank you for explaining what docs as tests means. I love the concept. Now, is this concept something that you invented, or has this bucket quote unquote existed?

Manny Silva: 9:25

So the term I believe I invented. But there have been people who have been doing what I consider to be docs as test strategies long before I started talking about it. I'm just able to. Example is there are some writers who have code snippets in their docs that unit test those code snippets and then they use tools like Bluehawk to extract those snippets from the unit test and embed them in the documentation. This is actually very similar to what the Raspberry Pi Foundation does. They have all of their scripts, every single script that you can run in any of their official tutorials are on the back end. Well, actual scripts that run unit tests. They run them end to end to make sure that they can successfully complete and then they extract those snippets and embed them into their tutorials. That's fascinating.

Manny Silva: 10:21

Also, if you think about notebooks like Jupyter notebooks, those can also be used natively as doc test tests, because they can be used for documentation, but they also contain code that you can run, and as long as you can complete it top to bottom. Well, congratulations, you just tested that all of your code works properly, so that a Jupyter notebook while I have opinions on whether or not it's the best source file for documentation itself is excellent from a DOCSIS test perspective.

Zohra Mutabanna: 10:53

The one thing that comes to my mind from what as I was listening to you speak is this to me sounds quite technical. Am I thinking right?

Manny Silva: 11:04

You are For the moment it is quite technical. Am I thinking right? You are For the moment it is quite technical, and that's part of why it's hard. For the time being, there aren't many tools out there that really assist tech writers in testing their content. Now I want to take a step back. The way I see Docs as tests is that in the perspective of other movements in the documentation space. We've got Docs as code that lets writers, content creators, use engineering tools and things like continuous integration, CI, to benefit their documentation. Then we've got things like Docs' ecosystem that borrows from the open source community ethos to improve documentation and how documentation can improve the community and the entire ecosystem of a product. We've got things like Docs observability, which is another new term that I've come across recently that leverages engineering, telemetry, tooling to provide insights into documentation and documentation usage in the broader product context. Docs as Tests does a very similar thing. It takes the not necessarily the same tools, but it could be the same tools that engineers use for testing and applies them to documentation.

Zohra Mutabanna: 12:25

Okay. So, although it is technical, the reason I ask that is because I have not been in an environment where I have had the opportunity to be technical or maybe put another way, maybe I'm fearful of being technical, so I don't push my limits. So there are two ways of looking at it. But that's besides the point. The first time where I really got to test Docs as Code is in my current role at Blackboard. So now I understand what Docs as Code means, where I am doing continuous integration with engineering, production or cadence, using their tools, their environment. So I understand that concept. But to me, to understand this DOCSIS test, I think will help me process it as we continue this discussion. But also one interesting thing that struck me was, again, using engineering tools, you could leverage the DOCSIS test framework. Is that right?

Manny Silva: 13:26

You can. So let me give a breakdown of a few different kinds of tools and buckets of how I think about current tooling for DOCSIS tests. On the complex side, we've got the engineering tools. We've got the things that are intended for engineers, like Puppeteer, like Cypress, like Playwright and don't get me wrong, there are tech writers who are using those tools today and that's great.

Manny Silva: 13:52

But those are heavily engineering focused tools and you have to be comfortable with whatever syntax they require and be able to write codes and like engineering level test assertions to be able to make use of them. So they are outside the scope of the vast majority of tech writers. So they're very powerful. If you're comfortable with them and they're associated to really great, go for it. They're wonderful, they're as powerful as you're going to get. In the middle we have something like doc detective and right now I consider it the middle, I don't consider it the simple, but my target audience for Doc Detective long term is the least technical tech writer. I'm trying to take the engineering tools and do the heavy lifting myself to simplify the interface, to simplify the way in which you interact with the tool, so that any tech writer, whether you're in a Docs as Code environment or whether you are writing in Word files, can use Doc Detective to create and run tests against your product. I'm not there today, but that is the goal.

Zohra Mutabanna: 15:03

I see, I see your point. So when you said the docs as tests is tools agnostic, what you're trying to say is that using doc detective doesn't matter what tool you're using to author. You can run tests against that documentation to verify.

Manny Silva: 15:22

So docs as tests is a strategy that is tool agnostic. You can take any of the principles of the strategy and apply them using whatever tooling you want.

Zohra Mutabanna: 15:32

The question that I would like to follow up with is are there tools as you create doc as detective? Are there other tools that are in the market that are close to doc detective? Are there other tools that are in the market that are close to?

Manny Silva: 15:44

Doc Detective Not that I'm aware of, no. Okay, that may rapidly change. With the state of AI and startups coming out of stealth left and right doing all sorts of stuff, including test automation, that might change. One of the things that I feel sets Doc Detective apart is that it is 100% open source. I have not spun this up as a company. This is a hobby that I'm running on the side, so, and you know, as with most open source projects, we are open to contributors, including just tech writers, helping our docs. Unfortunately, though I am a tech writer, I am also the primary engineer on DocDetective, so our docs have fallen a bit behind, I'm sorry to say, but right now, as far as that middle ground, as far as specifically testing documentation, there are no other tools that I'm aware of.

Zohra Mutabanna: 16:36

Okay, so it is open source. That is fantastic. That was going to be my question to you. How does one access this tool? So have there been any adopters so far?

Manny Silva: 16:46

Yes, actually there are a few.

Manny Silva: 16:48

I have not gotten express permission to mention some of them, but they've posted on the GitHub, so forgive me if I can't or if you prefer I don't, but Red Panda out in the UK uses Doc Detective Also.

Manny Silva: 17:03

I use Doc Detectiveective internally at Skyflow and I use it very extensively, and Skyflow has been very supportive of me doing DocDetective because it's improved the quality of our documentation dramatically. It's given us more confidence in our documentation and its accuracy. Also, again, skyfall is a privacy and security company and so user trust in the company is paramount. We're asking people to trust us with their most sensitive data, like credit card numbers and social security numbers, every single time that even a little thing is wrong with the documentation, with anything as far as the product is concerned, then it negatively impacts a user's perception of the product and the company, and so by making sure that my tutorials are accurate, down to the capitalization and spelling of a button, that helps make sure that our users have as consistent and accurate an experience with our products as possible and instills trust in our product and our company.

Zohra Mutabanna: 18:10

Yeah, I mean, it is definitely very important for an open source product to be trustworthy, and thanks for again giving us that context. Before we talk about and have the demo for Dog Detective, which I'm super excited about, can you give me a use case or a case study where this was extremely beneficial to you in your current role?

Manny Silva: 18:32

So again, I've been using it extensively in Skyflow and with all this sensitive data. It's so to be clear, I'm not testing with the sensitive data, but it's really important that our procedures are accurate. It's really important that our API calls are accurate. We're an API first company and so when I first implemented DocDetective in the Skyflow doc stack, then I discovered, oh hey, this tutorial that nobody has updated in the last six months. It has three broken API calls because it was written in beta status. And then the product got updated when it went to GA. But we thought we caught the doc update but we didn't.

Zohra Mutabanna: 19:36

Okay, so that's an excellent use case that I think sheds light on where Doc Detective can be helpful.

Manny Silva: 19:43

Now many companies may beverse to open source is supply chain attacks, which, for those unfamiliar with the term, are when you decide to use a particular piece of tooling and then somewhere along the lines it gets maliciously updated and you're not aware of it and that means that it gets access to whatever systems you feed into it and it becomes a problem. Doc Detective can run in isolation. It doesn't have to touch any of your other systems. It runs just fine in Docker it runs. Doc Detective can be tied into your other tooling if you want it to be. There are ways to install it that way. But it also runs completely independently. So if you want it to only run on your machine and not touch anything else, that's fine, it can do that. If you want it to run only in your CI process, so if you're running Docs as code and you check in a change and then a whole bunch of tests and tools run, Doc Detective can be one of those. So it's not even run on your machine or any of your company's machines. It runs exclusively in GitHub Actions or wherever your CI pipeline runs. So it can't touch any of your stuff.

Manny Silva: 21:07

Also, the content that's typically fed into Doc Detective is significantly less sensitive than most of what a lot of open source tooling eventually touches. Like we're not talking about databases. We're not talking about caches. We're not talking about databases. We're not talking about caches. We're not talking about logging systems that are directly integrating with the engineering stack. This shouldn't touch the engineering stack. It'll touch the documentation stack or run separately from everything else, but it's not touching the sensitive data. It's touching the documentation and it's running against the product, but it should only be running against the product in the same way that a user is, so there should be limited area for concern for vulnerability.

Zohra Mutabanna: 21:51

What are some of the challenges that one could run into, or what have you run into in your environment, if somebody were to, after listening to this episode, they want to try it out, and it would be great for them to know what to keep in mind before they start testing your open source.

Manny Silva: 22:06

Sure, the biggest challenge today is that you have to write the tests, meaning it. Doc detective is great at taking tests and running tests is not yet great at creating tests. So what you have to do is, unless you want to dive deep into very esoteric configuration, which I don't recommend then you have to go and you have to look at your procedure and you have to write tests for Doc Detective that mirror the procedure, and you either have to do that by hand in JSON or I have a prototype action builder that helps you build. Oh hey, I want to do a go-to action, and so you select that and then it shows you all of the fields that you need to fill out and then it gives you the JSON for that. But right now, the biggest barrier of entry is writing the tests, which I am working to solve.

Manny Silva: 23:03

I haven't talked about this publicly yet, but I am working on a set of tooling that will dynamically parse source documentation and interactively build tests along with the user, so it'll scan and say, oh hey, this looks like a procedure that you might want to test. Cool, this step looks like it would map to this action. Let's verify all of this and then let's actively try to run it and do it one step at a time, so that you can validate that. Oh hey, yeah, the test is doing what I want it to do. It matches my procedure and by the time you're done with it, you've not only validated that your procedure is accurate, but you have a test that you can have a computer run to validate the procedure over and over again as often as you like.

Zohra Mutabanna: 23:45

Okay. So I'm not able to hold on to the excitement anymore and I think let's take a look at the demo, because I think that will answer a lot of my questions as well, probably because I'm a visual person. And for listeners. We are going to make this demo available to you, so Manning over to you.

Manny Silva: 24:04

Sharing my screen right now. What we're seeing here is my VS Code window, and over here we have a markdown file that has a whole bunch of markup for Doc Detective tests. Here on the right side is it rendered as markdown? So there isn't a whole lot here, it's just talking about the Doc Detective documentation and the bits that you might want to pay attention to. There are some hyperlinks down here we have a little bit of on-screen text in bold and we have an image, and you can see all of that over here.

Manny Silva: 24:40

But all of these comments are Doc Detective steps. So here we declare that, oh hey, test is starting and I want to give it this particular ID. And then here we can see that we're doing things like oh hey, I want to check the link and here's the URL. I want to check another link and there's another URL, and it goes on and on. And then, oh hey, you know what? This one, I actually want to go to this link and here's the URL, and so that'll open it up in a web browser. And then, oh hey, I want to find an element matching this CSS selector and I want to make sure that the text has matches description and so on and so forth, and then here at the end, I also want to save a screenshot, and let me show you how that works.

Zohra Mutabanna: 25:26

Awesome.

Manny Silva: 25:27

So npx doc detective run tests, and this is doc content inline tests. And I'm going to run it. So what you're going to see is it's going to parse the file and then, because I'm demoing this, I'm having it open up head full web browser, navigate it to the page, check the content and hey, look, it's done. That was fast. And then here in the console, it outputs a report that it also creates a JSON file for and we can see cool, the test specification passed, the test within that specification passed. Cool, the test specification passed, the test within that specification passed, the context that it was run, it passed, and the 11 steps that made up the test passed. And you can go in here and check the nitty gritty of every single one and see what the output was, whether it passed or failed. And we can see that, yeah, hey, look, it matched the text, it found that selector and it saved the screenshot. Wait, it saved the screenshot, but it's not over here. Oh right, because VS Code takes a moment to update. There it is, and so there we go. That's a screenshot that was just programmatically captured. That is now embedded in the documentation. One of the reasons that I created Doc Detective the way that I did was because I hate screenshots. I hate them with a fiery, burning passion, because they always go out of date and they are the hardest thing in the world to update. And so there's a lot more that I'm going to do in the future in regards to improving the screenshot procedure, but right now we can already do a really fun thing called pixel diffing, which means that if a screenshot already exists and it captures a different one, it'll actually compare them and it'll say oh hey, these are different, like these have different percentages of pixels than the approved threshold, so I'm going to fail the step. So and I'll show you that in just a moment Now, this one is the markdown with all of the tests, all the steps for the test declared in line.

Manny Silva: 27:27

Now, I mentioned earlier that you can also, if you're willing to dig into the config, make it so that Doc Detective can automatically generate tests and run them just from your source content. That's what we're about to see here. I have an advanced config that I put together for Markdown that checks oh hey, this is Markdown syntax for a hyperlink, so it's going to run a check link action against it. This is Markdown syntax for on-screen text, so it's going to make sure that something, that an element on that page, matches that text dynamically. Oh hey, this is Markdown syntax for a link, but it is preceded by the word open. So instead of just checking the link, it's actually going to run a go-to action to navigate there. And that is all powered by regular expressions, which is not a favorite of many people, but it is powerful. So that's what we have today.

Manny Silva: 28:26

Again, I'm working on improving tooling for creating tests, but down here I'm going to run so doc contentmd, which is the file we're looking at. And here we go. It generated the tests, it ran the tests, they all passed, everything passed the whole test spec that it generated. There was nothing other than that file that you were just looking at. And the nifty thing is down here the screenshots are within maximum accepted variation. There was a 0.00% difference between the screenshots that it captured, the first test run and this one, and the maximum accepted variation is 5%. So if there was more than 5% difference in pixels between those two screenshots, instead of the result being passed, the result would have been failed, but it also would have captured that new screenshot, so you could update your documentation with the new screenshot without having to run the tests again documentation with the new screenshot, without having to run the tests again.

Zohra Mutabanna: 29:29

I have worked in Markdown and I can see the potential for this. As you said, you can really scale this, but thank you for sharing this demo with us. Now I understand what the tool does and how it does it.

Manny Silva: 29:38

Well, if you're willing to let me, it's just a very, very simple procedure on doing a search in DuckDuckGo for American short hair kittens, because I like cats. The nifty thing about this one let me just get it running and I can talk while it's running is, in addition to being able to take screenshots, doc Detective can also record videos of tests as they're being executed. So ignore the little pop up for the screen recording thing right there. That's just Mac OS not being happy with me, but it'll record just fine. And there we go. We just ran a test. It did American short hair kittens. It's just waiting for a couple of moments. It just saved the video, it is processing the video and it just converted it to a GIF. So there we have it. There is a recording of the test that we just ran, that you watched be recorded, and it is in GIF format that is now directly embedded in the rendered markdown that I can then ship with my documentation. Doc Detective supports GIF format, webm format and MP4, so you know, whatever animated medium you want, go for it. But this allows us to have procedures be even more visual. So, in addition to just having static screenshots, if there are snippets of procedures that you want to be animated so that people can actually see what it looks like end to end. You can do it Additionally.

Manny Silva: 31:09

Now this is the format of the tests written in raw JSON. So everything that you've seen up to this point, all of the steps, are directly embedded in the source content, which is great for when you really want to make sure that whenever you update the content, you update the tests accordingly. But there are some times where you might want to create just a raw JSON file and have that be where you define your tests, because you can do things like okay, run shell scripts, I want to run a Docker container before I spin up a test, or I want to run a Docker container and check the output. Or I want to make an arbitrary HTTP request to an API endpoint and this is really powerful to or all of you API writers out there, because you can chain these together. You can say hey, look, I want to send this payload as the request and I want to validate that the response payload has these values, and you can even check to make sure that it returns the appropriate status codes.

Manny Silva: 32:07

You can also go in and specify a JQ query for the response payload to say, hey, I want to extract a name and then store that in an environment variable that you can then use in a subsequent API call. So if you need to do something like, hey, I need to create a resource in a REST API and that returns an ID that the system created for you, then you can store that to a variable and then say make a get call in the next API call to make sure that it responds as you expect. So Doc Detective can do a lot and it's going to do more as time goes on. But today, whether it is a UI, web-based UI, whether it's an API, a CLI or an SDK because RunShell can run scripts to test SDKs then DocDetective's got you covered.

Zohra Mutabanna: 33:01

Fantastic. I think I got almost all of it except the API part, because I can pretend to be technical, but I'm going to admit that that was a little over me, but I got the gist of it. How about that? Because I'm an aspiring API writer and this was educational for me to kind of catch on to the lingo and pretend okay, this sounds great. But thank you so much for this demo. Is there anything to install on your end for DocDetective the?

Manny Silva: 33:31

only prerequisite for DocDetective is Nodejs on your machine. Okay, so you have to have that. And then the command here npx docdetective. If you haven't run docdetective before, that will automatically download and run docdetective for you, so you don't have to install anything separately. As long as you have Node, you can run docdetective.

Zohra Mutabanna: 33:54

Just the latest version of Node.

Manny Silva: 33:56

Yeah, I tested on Node 18 and 20. And if you want to cache it, then you can do a global install of docdetective and npx will pull from that first, instead of redownloading it every time. But also, if you want to use it in ci, then it's just as simple as npx dash y to say yes, download it please, and then doc detective and you're off to the races. You don't have to install anything else okay, fantastic.

Zohra Mutabanna: 34:20

This concludes our demo, I suppose. Or is there anything more that we don't want to miss.

Manny Silva: 34:24

That's everything I've got for now.

Zohra Mutabanna: 34:26

So thanks for the demo, manny. This surfaces to me what power we can have as writers if we can just think a little bit out of the box.

Manny Silva: 34:35

Yeah, there's a lot that we can do, and I mean it goes back to what I said before. Docs are tests whether or not we have computers, test them for us. And technical writers, by trade, are people who work in all sorts of different areas, who interact with all sorts of different teams, who have diverse skill sets, and I think that this is just one more, and this is the lens that I'm bringing to technical communication. Broadly, it's like okay, I don't want things to break, I don't want my things to break, and if I can help other people, I don't want their things to break either. So, as long as we are willing to take a step back and learn about this new domain, which is new to us but is not new to our engineering friends, then there is so much we can do within our work groups and with the people that we already interact with daily, known for being a source of reliability rather than a downstream liability. Then it changes the perspective of the writer. You start getting pulled into meetings early with design, early with engineering, so and you become an asset to those teams. If you have a dedicated quality assurance team, then you might become their new best friend, though I do want to call out that there are two things to watch out for here.

Manny Silva: 36:09

First is that docs as tests is not a replacement for existing engineering best practices. It is not a replacement for unit tests. It is not a replacement for end-to-end tests, because you can only with Docs as Tests. You can only test what you document, and there's so much more to a product than the happy paths that we typically document, and they do all of the edge case testing. We can't handle that. There just aren't enough of us and users don't want to read all of the infinite variations that we would have to present to test everything with Docs as tests.

Manny Silva: 36:53

But the other thing to watch out for is that there are going to be people that don't immediately see the value, and there are going to be people that question why are you doing this? Why are we bringing in yet another tool? We already have the engineering team using Cypress. The only way to really show them is to show them. In my experience, you can tell them oh, the benefit to me is that okay. Yeah, the engineering team is using Cypress, but they're not using Cypress to test my documentation content. You want to dedicate their resources to helping me test my documentation content, no, no, okay. Well then, I'm going to do this because this is going to improve the user experience. It's going to improve the perception of our company. It's going to make sure that our docs don't break when a prospect is coming in and looking at our stuff and we lose a deal because of it.

Zohra Mutabanna: 37:43

Very important.

Manny Silva: 37:45

Very important.

Zohra Mutabanna: 37:45

I think you made some fantastic points. One that I want to call out is advocacy of our profession, which is my intention for doing this podcast, and what you're bringing to light is how technical writers can be an asset by presenting these options, and I'm so thrilled to see the change that can happen when one just explores and experiments and, thanks to those sleepless nights, we have something great on the market, hopefully.

Zohra Mutabanna: 38:19

But then reliability, not a liability Another cool phrase that I'm going to steal and then being imploded in meetings upstream, improving user experience. I think everything that we do that is our eventual goal is to improve user experience and to make sure that the docs that we bring to the customers are not broken and they're working. Since you've been the creator and the user of this or you also mentioned that there are others that are using it have they found a significant benefit in terms of time, cost saving that you've learned and that you'd like to share?

Manny Silva: 38:56

So I can't speak to other users but for myself. What it comes down to isn't necessarily that there has been time savings, but it's enabled me to do so much more that I wouldn't have been able to do otherwise, because I run a very small documentation team it's me and one other writer and however many other people across the company I can convince to contribute. What that means is, when it comes to documentation, freshness testing doesn't really happen until we're forced to go through some, or it hadn't really been happening until we're forced to go through something because oh hey, this is a new feature or there's a prospect. Who's interested in this? Let's make sure that everything's good that sort of thing.

Manny Silva: 39:40

But with docs as tests, instead of having to have a freshness schedule that promptly gets ignored because there are other more pressing matters, I have the computer check everything for me and I can have confidence in my docs on a daily basis.

Manny Silva: 39:53

But also it helps make sure, like you can run docs as tests in all of your engineering environments, because you can run it in production against directly what the users are using right now and help monitor for product downtime.

Manny Silva: 40:09

Now, obviously, again, this is not a replacement for best practice engineering product downtime or uptime checking, but it helps. It doesn't hurt to have another check. But also, when you're going from development to staging, before you get to production, run all of the tests against staging. And that's helped me catch some issues because like, oh hey, this has worked or not. When running tests and staging, it validates that all of the content that I currently have live works against the next release of the product. And if it doesn't work against the next release of the product, there's a problem, and oftentimes the problem will be with the documentation. I'm not saying that everything that comes up through Docs as tests is a problem with the product Far from it but all the same it helps you get everything Docs and product in tip-top shape before it goes out to production and users start seeing it.

Zohra Mutabanna: 41:06

I mean, there's clear value that I see in it. But what I would ask is probably this is not the right time for this question, but have you had an opportunity to provide any metrics or any sort of reporting on this where your company has felt, wow, this has really made a difference? I know you said not in terms of time savings, but higher ups love to see reports, love to see dashboards. So just curious about that.

Manny Silva: 41:36

I'll be honest, I'm still working on that bit. I do not have an aggregated board that surfaces issues found by Doc Detective yet, but the intent by having the report results be output in JSON is that it could very easily be adapted to whatever anyone needs. It's machine readable, it's as close to human readable as a machine readable thing gets and you can easily manipulate it, ingest it into whatever you want. I know myself, when I initially implemented it, I found a decent number of issues in my docs that the docs had gone stale Since then. We get something here and there, but it's pretty rare.

Zohra Mutabanna: 42:17

Yeah, you talked about running this quote unquote every day. But for somebody who is new to this, who wants to experiment with it, what would you recommend? What should they start with a small use case just to experiment, and how frequently should one run Docs Detective to really benefit from it? Just some thoughts on that.

Manny Silva: 42:36

As often as you feel is necessary. I have a lot of thoughts on, depending on your environments and the frequency of how often your product publishes, when should you do it? My general feeling is run it once per release at least, because your product shouldn't be changing between releases. So as long as everything's good with the release, you don't necessarily have to run it again after that. But if you're just getting started, it's kind of a cupcake to wedding cake scenario. Start with a cupcake and figure out okay, what is my most important procedure in the entire product? Cool, can I test that? Yes, perfect, let's do it.

Manny Silva: 43:19

Is it too complicated or long to test and find something smaller or find a subsection of it, just to get your feet wet, just to get used to writing tests, and then, a little bit at a time, write a little bit more of the test, write a little bit more of the test, write a little bit more of the test. Or, if you finish that one, find another high priority or small test that you can just keep adding on, and adding on, and adding on, and then eventually you will have that wedding cake of tests. It doesn't have to happen overnight, frankly. It shouldn't happen overnight because then you would be getting as little sleep as I've been getting, but long term you can build up to having a notable portion of your doc stack, of your documentation source content testable.

Zohra Mutabanna: 44:03

This conversation has been an enlightening one for me. Is there anything that you think I may have missed and you'd like to add?

Manny Silva: 44:10

We covered it pretty well. Is there anything that you think I may have missed and you'd like to add? We covered it pretty well. I just want to reiterate that Doc Detective is an open source project. It's up on GitHub. You can look at all of the changes I'm making to it if you're so inclined. But you know we have the issue tracker there if you run into issues. We also have a Discord that I am on daily and we regularly get questions there and you know feature requests. But so if you're having any troubles, you know there isn't an SLA here, but I'll do what I can to help if you run into issues implementing or building tests.

Manny Silva: 44:44

Additionally, I am going on tour with Doc Detective a little bit. I'm going to be at well, based on recording. I will have been at Write the Docs Portland writing day to help people write tests there. I'm going to be presenting docs as tests at the Linux Open Source Summit, opening their Tech Docs sub-conference, and I'm also going to be presenting docs as tests at the STC Summit in May. So if you don't want to go on Discord, there are some opportunities for people to find me in person.

Zohra Mutabanna: 45:16

That's awesome and, manny, we are going to provide all this information in the show notes so that they can have it, so that the listeners can know where to find you if they cannot get to one of these events, and that way, even I know where to get you right. Yeah, thank you for your time and thank you for sharing your knowledge with us. This has been a fantastic conversation and I know this is going to benefit so many of us. Thank you for bringing this to us, manny. Appreciate it.

Manny Silva: 45:43

It's been my pleasure, zora.

Zohra Mutabanna: 45:45

Thank you, it's been my pleasure too. Thank you. Subscribe to the podcast on your favorite app such as Apple, spotify or YouTube.

Manny Silva: 46:16

Thank you.