The survey is now closed and as promised I've written up the results of the responses here. You can read the write-up to the previous survey from the middle of 2016 over here which can be interesting to compare and contrast with. It's worth noting that I ran the previous survey when I began working on RenderDoc full-time.
I also really want to thank everyone for taking part. There were almost double the number of responses as last year at 635 total. Whether that's down to more people using RenderDoc or more people filling out the survey it's great news. Hopefully you'll find the results interesting.
One thing we can do now is compare the numbers from the survey against the number of users reported in RenderDoc's public analytics. Due to the way the reporting works once per month at most, November and October are the most recent up to date at the time of writing (January 2019). They indicate about 1900 unique analytics reports per month, though that does not necessarily equate to the number of RenderDoc users. Analytics can over-estimate by counting multiple machines for a single user, and under-estimate by not counting anyone who has the analytics disabled. Exactly how many users this represents is unclear, but this indicates a relatively high percentage of RenderDoc users filled out the survey when bearing in mind that e.g. a 40% response rate for an anonymous optional survey would be a good achievement.
I'm going to jump around the survey out of order a bit to make it easier to follow and organise the results.
Please remember that none of the questions in the survey were required, so I've listed the number of data points for each chart - this gives you an idea of how many people answered. The survey was advertised on twitter, in the release notes and upgrade text. Without putting a pop-up in the program (which would be rather obnoxious) this reached as many people as is reasonable. However it's quite possible that there are other users who either didn't find the time or didn't want to fill out the survey, or never noticed it. This is especially true of users who have never tried RenderDoc before.
First up we have some information about how many people have used RenderDoc.
How many respondants have used RenderDoc before.
I asked everyone to fill out the survey whether they had used the tool or not. The survey still selects for users who are one degree of separation away from a RenderDoc user or else they'd never hear about it, so it's not a fair sample of all people who might want to use the tool. However without some extra marketing/outreach push this is going to be the most representative sample of users I can get.
While we can't conclude too much from this due to the likely high chance that someone who hasn't used RenderDoc also wouldn't hear about the survey, we can see that this has improved since a couple of years ago. In 2016 18% of respondants hadn't used RenderDoc, so while we can't tell how many people have tried it once and don't use it regularly at least it indicates that probably most people who have heard of the tool have at least tried it.
For those who selected that they hadn't used RenderDoc, they only had to answer two other questions - which debugger they preferred (and why), and then how they heard of the tool. We'll see the latter question in a little bit, but for now let's look at what other tool they're using and why.
Users who have not used RenderDoc - preferred debugger.
For users who have not used RenderDoc - why they prefer another debugger.
The number of responses in this section is too small to put too much weight in, but we can see a few changes from the last survey - in particular that out of people who don't use RenderDoc, Xcode is a much more popular alternative than it was previously. We'll discuss other platforms later in the analysis but this is something to bear in mind.
Looking at the reasons for preferring other debuggers, lack of support for platforms or APIs covers most responses. Again we can't conclude anything from a small sample like this but it suggests that most users who can use RenderDoc (because it supports the platform they're working on) have tried it.
Next while we're talking about this, let's look at the same question for users who have used RenderDoc. At the end of the survey I asked those users whether they preferred and used RenderDoc most often, or whether they used another tool more often (and why!). Almost three-quarters of users said that they prefer RenderDoc, higher than last time! Here are the breakdowns of the remaining quarter:
Users who have used RenderDoc but don't prefer it - preferred debugger.
For users who have used RenderDoc - why they prefer another debugger.
By comparison, within people who have used RenderDoc but don't prefer it their preferred tool is most likely to be on a console platform. On desktop the most popular tools are NVIDIA Nsight, the new D3D12 Microsoft PIX, and Xcode's GPU Tools. Looking at the reasons people choose these tools the desire for a combined debugger and profiler ranks very highly.
As in the previous survey, a lot of people use whichever tool is available for their platform. This makes sense with a large proportion of these users working on consoles where often there is only one tool available and it's highly specialised for work on that plaform.
The last question in this section was for curiosity, asking where people had heard about RenderDoc.
Where people heard about RenderDoc.
I changed some of the options this time which makes a direct comparison to the last survey a bit imprecise. Since it's the most open-ended question in the survey there are overlaps between the different categories that can be confusing, but the overall picture is pretty clear: Most people hear about RenderDoc from either someone online via twitter or another chat program, or else at work.
Although the options changed from last year (to try and clarify which is which), the results seem fairly similar. I'm glad that people continue to feel the tool is worth recommending to others, and I hope to keep it that way!
I asked about whether people wanted further API/platform support, or more improvements to the existing platforms.
In the last survey RenderDoc was quite different - there was no native UI on linux, D3D12 was unsupported, and there was no support for any API on Android. This question was trying to take stock and see whether there is further support needed or if the current set of platforms encompasses enough use-cases.
First I just asked a simple Yes/No question about whether users wanted further platform support.
Do you want new OS/API support?
Almost 80% of users are happy with the current OS/API support, at least that it's more important to improve it than expand platform support. This is a pretty good proportion of users although it does leave a fifth strongly wanting more functionality. It's always a bit of a subjective choice as to where to draw the line, and although it's interesting that some users requested D3D10, D3D7, or even gamecube support, that's part of a long tail that is not worth the implementation cost. To see how much of this 20% is a long-tail or how much is centralised on a few key platforms we need to dig in deeper.
The next question deliberately asked all users - even those who said 'no' above - to list what platform support they would use if it were added. I listed it this way so that even those people who would rather see impromvents to existing platforms could list what they would like.
Which OS/APIs would you like to see supported?
At the top of the list (barely!) is WebGL which I wasn't expecting at all. It seems that developers on WebGL don't have a lot of good tools available, so desire for native WebGL support in RenderDoc is high.
Next is Metal for macOS with almost as many responses, followed by Vulkan. Vulkan support I wasn't too surprised to see asked for, since to my knowledge there isn't yet a native vulkan debugger on macOS - though users can of course debug the underlying Metal. I was surprised to see Metal itself up so high, as I had the impression that most users were happy with Apple's tools. My guess is that this is partly due to the value of having a single tool working across multiple platforms in a cross-platform project, to translate familiarity with a tool to different platforms, and ease comparisons between different rendering backends.
Rounding out the list were iPhone, D3D9, Legacy (pre-3.0 fixed function) GL and macOS GL support. Apart from iPhone support being quite highly prized these were more what I was expecting.
There is another view of this data though - what if we selected only the responses which said 'yes' above when asked if new platform support was high priority for them - do the answers shift?
Among users who prioritised it, which OS/APIs would you like to see supported?
We can see that filtering in this way does change the priorities. No longer is WebGL near the top, instead it has dropped down to the second tier. By comparison, iPhone is more important to users, being equivalent to macOS vulkan support and WebGL.
D3D9 and Legacy GL have dropped in priority even further, sitting at the bottom of the pack.
As a side-note, this question did also have quite a number of write-in answers. For the users requesting D3D12 support I have good news - it's been available for quite some time now! Not all features in the API are supported as yet e.g. ray-tracing and DXIL, but D3D12 programs not using experimental/bleeding-edge features will capture and replay. There were also a few requests for console support - in particular PS4. Unfortunately that's not something I can do whenever I like, but if you're listening Sony then call me!
In general any API which only had a handful of responses is unlikely to be worth the implementation and support burden, as unfortunately being realistic the priority is never going to be high enough. This means that soon RenderDoc is likely to hit a steady state of platform support and focus more on improvements to the existing coverage.
RenderDoc has two user-facing APIs - an in-application API to control capturing, and a more extensive python API during replay to script the replay. I asked about how many people were using them.
How many users have integrated RenderDoc's API.
Comparing the results of this response to last year shows that not too much has changed. A higher percentage of people are aware of the API - 83% up from 72% - though the proportion of people using the API has stayed roughly steady. The results are roughly expected, since generally RenderDoc aims to 'work out of the box' and the most common use for the API is when manual capturing is needed for headless programs or offline baking steps. Generally speaking about 40% of the users use it one way or another, though the 11% using it through Unity/UE4 integration are most likely only using it indirectly for tighter integration with their workflow.
Since the last survey UE4 is now shipping the RenderDoc plugin by default rather than it being a separate download, which does make using it easier.
Ultimately this API is a utility for when you really need it, so aiming for 100% usage is not a goal by any means. I think the API is pretty stable at this point and it seems like it fills most uses that it's needed for, so this is a good sign that it's both stable and satisfying that need. I had in the past considered very tight integration between application and RenderDoc but in practice the benefit doesn't outweigh the cost of such an implementation.
How many users have used the Python API.
The responses to this question were a bit disappointing but not too surprising in the end. The large majority of people were not aware of RenderDoc's python API, and only a small number had used it even a little.
I do think there is a great deal of power available within the python API, and with v1.2 I've started to use it to expose extensions in the qrenderdoc tool itself for customising the UI even further. I hope that those who hadn't heard of the API before give it a look and see if they'd find it useful, as it exposes all of the functionality that the UI uses and more. However this does suggest that pouring a lot of time into developing the python API is not the best bang-for-buck as right now the userbase largely isn't going to benefit from that.
These questions were here to find out how people viewed RenderDoc's stability, and to compare against the results from last year.
Often with RenderDoc I only hear from people when it breaks or they want a new feature. This is expected (though it's always nice to hear positive stories as well!), but it means that I don't get a good intuition on how stable it is overall by essentially only hearing about failures. Getting a rating from people gives a better idea of how RenderDoc is doing, even if it is a bit subjective.
As in the previous survey I've combined three questions: Rating from 1 to 10 of how often blocking bugs like crashes or corrupted garbage appears; 1 to 10 of how often non-blocking bugs happen like inconveniences, incorrect output or broken individual features; and then finally a general impression of how RenderDoc development is going, whether bugs are fixed swiftly and user feedback responded to.
How users rated RenderDoc's stability and development efforts.
Compared to last year the results are actually even better, and I hope that's due to a legitimate improvement in the quality of the tool rather than a change in rating assignment. The previous results were already great, and even maintaining that level of perceived quality would have been an achievement, so having a shift towards higher ratings is great news. It's possible that this is due to a different sense of rating, as the previous results showed a dip at '9' perhaps indicating a marking bias away from a full '10' marks. That said the questions were identical this time around so it's quite possible there is a more positive view overall.
This is by no means an excuse to take things for granted, but I'm glad that ther's generally a positive opinion of the tool and I hope I will be able to live up to and continue to fulfil those expectations.
Lastly I asked a few questions about the community in general
What RenderDoc is being used for.
The proportion of users reporting for each category remains almost unchanged since last time, apart from the addition of a new option for academic research which has a small but not insignificant user base. Real-time 3D graphics remains a field heavily dominated by game development, but other graphics fields are still common. Likewise most people using the tool are using it at work, but not by a massive margin.
Setting up a new RenderDoc chatroom.
RenderDoc has had an IRC channel on freenode for a number of years now, but even with webchat programs like kiwi-irc many people prefer other chat services that are more mobile friendly or provide offline history. This question was to ask whether or not setting up a new chatroom would be desired and if so then which one.
The results are fairly close but it seems clear that a good number of people would like a service like Discord/Slack. Since Discord is slightly more popular and also provides better features for free servers (Slack message history quickly disappears), I've set up a RenderDoc Discord server which is free to join.
What people have used the github repository for.
The last question just gives some insight into what people are using github for. Compared to the previous survey, more people have compiled or browsed the source. It's also more common to make local changes. I think this is a great shift, even if it is relatively minor, as I think the open-source nature of the tool is a boon for any user.
I would still hope to reduce the number of users who haven't had any interaction with github. As much as I'd like to believe it, I don't expect that they have had a perfectly bug-free experience with RenderDoc. I still find that there is a hesitation to report bugs or request features, and if that applies to you then please let me know what would make you more likely to give feedback. If you didn't rate RenderDoc as 10/10 then I hope you can let me know what would improve it - even if it's a long-shot request or feels unlikely to happen, it doesn't cost anything to let me know about it.
There were a few write-in responses that referred to bugs encountered. In general it's best not to assume that I'm aware of a particular bug, because while it might be a 100% reliable issue for you there might be a specific set of circumstances you're encountering which do not affect most other people. I prioritise bug reports when working on RenderDoc but I can only fix the bugs I know about!
I'd like to thank everyone who filled out the survey once again. I haven't mentioned it above but I also appreciate everyone who left kind words in the comments field at the end of the survey, as I read them and it's very much appreciated!
I touched on it in passing above, but there were a few responses that mentioned specific bugs or features. For things like this please do Get in touch with me or file an issue on Github so it can be tracked, as I have no way of responding to anyone through this survey as it was completely anonymous.
In terms of future action based on these results, I'll take some time to process them further but it seems like expanding RenderDoc support onto Apple platforms and WebGL would likely mean the widest reach that is feasible on limited development time. I can't promise any particular timeline or feature set currently but this will be something I look into soon.
Similarly I've now set up the Discord chat for those who want to use that, as hopefully that will be an extra way to get in contact or discuss RenderDoc issues.
Hopefully these results are of interest, and keep an eye out for future development over at Github or my twitter, and if you've been inspired to get in touch and want to email me then please do!