You find a bug in macOS or iOS. Something in the software isn’t working the way it should work. You decide to let Apple know, and head over to http://bugreport.apple.com and login with your Apple ID. You fill out the required form, list the reproduction steps, hit the special key chord to generate a fat sysdiagnose, attach some screen shots, and hope they fix it.
What happens to your bug?
All bugs are tracked with an internal application called Radar; internal Apple employees simply log bugs directly into this application. External people use the website “Bug Reporter”, which only shows specific developer interactions with the bug, and hide all other details of the bug. Those details are engineer comments, progress on the bug, when they “expect” to fix it or which build it is fixed in, and priority. Priority is important for bugs; they are considered “unscreened” if they don’t have a priority set. Bugs with a high priority (1) are highly likely to get fixed, whereas bugs with a lower priority (3 or 4) are not likely to ever get fixed.
Each bug is placed into a particular top level component, which can have sub components. Savvy internal developers will log bugs directly into the component that the bug corresponds to, such as “AppKit / NSTableView”. This will get it directly to the right person who needs to look at it. Top level components have a large dropping areas that generic bugs can fall into, such as “iTunes / All”. These components get saturated with bug reports.
Once your bug hits Radar someone will look at it. Developer Support will often take a first look at bugs that are related to SDK/programming issues. They will then pass it on to a more specific top level component, or directly to a more specific component (ie: NSWindow).
This is where things get screwy depending on the component your bug lands in, since bug management is group dependent. Many groups will have only one or two QA people to do the initial screening of those large drop areas for bugs. QA engineers are sometimes instructed to screen bugs with a priority and “fix period” before passing them off to the engineer responsible for the code. This is terrible because many engineers will not look at bugs with a low priority. It is much better for the engineer who “owns the code” to look at a bug and determine the priority. The QA engineers will frequently get a huge back log of bugs to screen, and it can take weeks, or even months, for some bugs to get screened. Sometimes this leads to a mass screening of bugs, marking them all with a low priority. Bug originators have to notice this, and complain about it for the priority to get increased. Worse yet, some groups mass close bugs older than a year or so, and ask the originator to re-open the bug if the issue still exists. A lot of people don’t pay attention to bugs that need verification, and they simply become lost.
From an internal developer’s perspective, it was quite frustrating to take the time to log a detailed bug, only to see it sit “unscreened” for weeks and weeks. It is demoralizing to see this happen to your bugs. But worse yet, sometimes your bug would get forward-duped. Someone else at a later time logs the same bug, and your older bug report gets duplicated to it. More often than not, this is simply because of the delayed screening of bugs.
From an external developer’s perspective, it is even worse. There is very little feedback on bugs when they are in progress. You don’t know what priority the bug has, or if it has even been looked at and screened. This is Apple intentionally being opaque. More transparency could easily leak features or products, so this is expected.
Hopefully a bug ends up with an individual engineer who works on the code. A lot of these bugs will still be unscreened, and Apple engineers seem to loathe the process of screening bugs. So they let them sit around for a while, sometimes for weeks or months. People would rather focus on coding something new up, or working on already screened bugs that they have to fix, as opposed to taking the time to read a bug report and figure out what needs to be done with it. They don’t have to read it and fix it right then, but just do a screening: determine priority, when to fix it (this is sometimes nebulous), and possibly ask the originator for more information (or mark it as a duplicate.)
Engineers also dislike screening bugs because sometimes they have to add them to their queue for the current release. This increases their required workload for that release, which is something people don’t like doing. So, instead, many bugs stay unscreened.
Unscreened bugs indicate potential software problems that need to be fixed. Obviously this is bad, and management doesn’t like to have high unscreened bug counts. Managers will frequently instruct engineers to screen bugs, or ask QA engineers to do it (which is bad).
Sometimes QA screen bugs with a low priority and holds onto them. They never get moved to the appropriate code engineers, and effectively become lost in the system. Sadly, I had seen this happen way too often.
Bugs screened with a low priority and a nebulous-future fix date are also lost. They rarely will be revisited and fixed. Sometimes this is okay. Some bugs are just too obscure or rare to spend time fixing.
Some bug owners don’t want to take the time to go back and forth with the originator to find out some piece of information they may need. They sit on bugs for a long time, and eventually try to reproduce a bug on a later version of the un-released OS. They can’t reproduce it, so they just assume it is fixed, and send it back to the originator. This is particularly bad for external developers, and I’ll describe why below.
When a bug is sent back as fixed, the internal developer who originated the bug is supposed to verify the problem is resolved. They can send it back if the problem isn’t resolved. However, internal developers don’t really have an incentive to verify bugs. Management doesn’t keep track of bugs that need verification or really require developers to verify them. Most engineers do verify bugs; they like to make sure problems are resolved. But external developers are left in a more sad state. The bug becomes closed for them, and is dead.
External developers do get notified when bugs are fixed, but only after the fix is available in a public release of the OS. This means it could take about a year for some bugs to get back to the external developer, since bugs start to get fixed *before* an OS is released to the public. Generally, the bug will ask the external developer to verify the bug by installing the latest OS. For some people, including me, this just isn’t possible; they need to run and support an older OS, and don’t have the ability to install a newer OS on a particular machine. Or, they just won’t want to install the latest OS because it doesn’t have anything they want or need (that’s me, I run 10.13). Even if they do install the OS, they may find the bug isn’t fixed. At this point, it is simply too late; engineering has likely closed the bug, and it is lost in the ether.
What can Apple do?
It is simple: Internal engineers need to take more responsibility in promptly screening bugs. Management needs to allow engineers to have more time to do this, which is at the expense of working on features or fixing already screened bugs. Engineers should always be expected to have a very low unscreened bug count.
Internal engineers need to promptly verify bugs when they are sent back to them. They need to install beta versions of the OS and do some testing. If it isn’t fixed, send it back.
If a bug report refers to some problem, then the developer should look at the code that interacts with the area described and try to think about what would cause such a problem to happen. For instance, if a button is incorrectly disabled in iTunes, then the developer should try to figure out why that could happen instead of just asking for “how to reproduce this?”
When I worked at Apple I practiced what I am now preaching. I would screen all my bugs within a day or so. I would verify bugs within a week or so. I would poke around at code and attempt to make theoretical fixes for bugs I couldn’t reproduce. I felt like it was part of my job, and I just did these things as a daily task.
What can you do?
Obviously, you have to log bug reports. It’s okay to log bugs that are missing information; developers can ask for specifics of what they need. In general, for AppKit related bugs, I would find that a “sysdiagnose” rarely would provide any useful information. I would hate it when people (usually QA, and not the engineers who worked on the code) would ask me for a sysdiagnose, when I know it wouldn’t help them figure anything out.
Log duplicates. If you know someone else logged a bug, but you are encountering the same issue, then log the same bug. It will get marked as a duplicate, but this will help increase the priority of a bug. It is sad, but Apple does look at this, and bugs with “high dupe counts” are looked at.
[…] The Sad State of Logging Bugs for Apple 10 by fireflies_ | 1 comments on Hacker News. […]
[…] Source: http://www.corbinstreehouse.com […]
My “favourite” macOS bug is 14 years old and I have reported it countless times, including in email exchanges with Federighi. The OS doesn’t respect an app’s input language and changes it when switching apps. Default input, English, gets overwritten on app switch. There is an option, it doesn’t really work. This is my only serious problem with the macOS I love and it’s sad we get new bugs galore while old ones are ignored.
That is a pretty sucky problem. One thing you can do is get other people to log the same bug; advocate to others to report the issue. This will help with the priority.
radar id?
Yeah that’s a funny one – imho I think it’s a feature but I see where you’re coming from
If I go to messages and switch to German, I like the fact that it goes back to English in Xcode. But maybe that’s just me?
I have keyboard shortcuts to switch between languages btw which I use all the time
I would add one very annoying sad thing about logging bugs at Apple and which is a consequence of different points you described:
Reported bugs are rarely fixed in minor OS updates. Waiting a year to get a simple bug fix is not reasonable and can make people stop using one’s application.
It sounds like you’d like a more progressive approach; lots of constant OS updates that improve it frequently, as opposed to a yearly release.
While there is very little to disagree with here, there is a rather large neon green elephant in the room that is not being addressed: It costs time to not only work around these defects in Apple products but it takes time to diagnose, document and report them. Time == Money.
Small companies can’t afford to have their engineers for free Apple for free as Apple’s QA Dept. We include an invoice with each bug report we file now to make the point.
Hi William – that is very true! It is hard for any software product to be 100% bug free, especially with something as complex as an OS, and as diverse as an SDK; people use SDKs in ways that a developer may not have thought about or intended. Sometimes it is good to let the SDK developer know when you hit a snag; I realize this takes time, but I’m sure you would also like your users to report issues in your products. Do you pay your beta testers or users when they find issues?
Another relevant part of logging bugs is the third-party site Open Radar (openradar dot me). When you submit a bug to Apple, if it doesn’t contain sensitive information, you can also post it to Open Radar. This can help other developers to understand relevant bugs, know their status, and intentionally file duplicates that reference a Radar number.
That is an excellent point! When I worked t Apple, I always liked seeing people utilize Open Radar. I need to start being on the other end and logging bugs into it myself.
This is one of the top 3 reasons we move out of MacOS system on our company. The global sansation is an unsafe OS ecosystem.
What OS are you using? Linux is probably the only transparent OS. Windows has its problems too.
[…] les bugs sont suivis avec une application interne appelée Radar », explique-t-il. « Les employés d’Apple se connectent simplement à cette application […]
Thanks for posting this. Explains what I’ve been experiencing on the outside. Curious if you saw any trends over the time you were there. Did things get worse over time? Or stay pretty much the same?
One thing that might help would be a way for devs to mark their bugs – or portions of their bugs (description and steps but not attachments, say) – as public. Would reduce total count to review & waste less dev time filing duplicates (could vote up or add to existing maybe).
cheers.
I did see a trend of people caring less about bugs. As the company grew, so the the teams and the influx of bugs, which probably contributed to this problem. But another problem was with new people being added on; they didn’t seem to have the same care that older people did with regard to handling bug reports. I saw this as particularly bad in iOS. Another problem I saw is with how older bugs were handled when a new release cycle was started; I can’t go into details with that.
I saw this too — when I started, each bug was special (not really, but it was pretty close, and dup chains were well-tended for active components). Then the numbers started ramping up really really fast (remember when the number of digits changed, and Radar’s key chord stopped working for a while? During Lion or right at the end I think?). Then passive-aggressive auto-close-if-not-fixed-in-30-seconds policies got adopted, and spread, and it made everyone despondent.
Yup! I remember all those things. I’m starting to sound like an old man “back in the good ‘ol days of Radar…”
[…] Corbin Dunn (tweet, Hacker News): […]
The need for secrecy can be frustrating for both sides. I requested a feature in a framework, and the report was closed as “Working as intended.” I grudgingly worked around the missing functionality. Next WWDC, a sister framework was announced that was capable of doing what I needed. So from my perspective, engineers agreed with me that the functionality was needed but weren’t able to tell me that they were working on it in a new framework. “Working as intended” was both accurate and misleading!
I think that was the wrong way to handle such a bug. They should have duplicated it to the bug that tracked the new feature in the new framework, and sent the bug back to you as a duplicate. It clearly wasn’t “working as intended”. Granted, you wouldn’t have known anymore more, but at least you would know it was a known request.
They don’t really have a crackerjack team at Apple – do they :( Maybe they need to fix that first?
I think there are a lot of great and talented people at Apple. I worked with some amazing people when I was there. I don’t think the team is the problem; I think management needs to change its style and priorities for quality.
It can be a problem when a individual behaviour triggered by bad guidelines by management (e.g. not creating enough hard room for bux fixing) becomes a culture that everybody handles similarly and floor-talks about as an fact of daily routine. At that point only a official change in culture and the needed environment and thinking has to be established by management again since an individual would have to fight (and probably loose against) too many who just accept the developed routine.
You are very right!
Since there’s no way of officially reporting Apple Music bugs (pity, because it’s arguably their most bug-ridden service), I’ve tried to do it with their Twitter support. They are utterly unwilling to admit the existence of a bug and they will do everything possible to slightly change the topic enough times until you give up. I’ve gotten links to help articles for other features and been told a bug is ‘as designed’. This is the frustration that made me stop using Microsoft products 8 years ago.
I’m pretty sure you can use Bug Reporter to do it. They will track all bugs in the same system. Have you tired logging it there, say in the iTunes component?
[…] via corbinstreehouse […]
[…] @nibroc shared a post about the ugly details for how bugs are dealt with at Apple (spoiler alert: not well, or consistently), I enjoyed seeing my feelings echoed by someone much […]
I have filled bugs for 10 years. After realizing none of my bugs got ever fixed, I stopped filling them. Apple hates developers. They give us subpar tools as Xcode and most of their docs, that are a piece of crap. They don’t give a crap to developers. At one time they were giving hardware discount to musicians and not to developers. They have almost live support to book authors and musicians. Developers for them, are crap.
[…] Apple. The “What can you do?” section of Corbin Dunn’s well-publicized blog post “The Sad State of Logging Bugs for Apple” is far too thin. The post is mostly inside baseball on how bugs are screened at Apple. Neither you […]
Good ideas! I am hoping to write something similar to what you have elaborated on.
[…] The Sad State of Logging Bugs for Apple […]
[…] The Sad State of Logging Bugs for Apple??? ?????????? ? ??????, ????? ?? ????????? ?? ? Apple? ? ???????????? ???? ??????????? ?????, ??? ??? 14 ??? ?? ?????? ??? ???. […]
[…] dan mengakibatkan banyak bug terseret selamanya atau menghilang ke dalam lubang hitam. Itu kata pengembang perangkat lunak Corbin Dunnyang dirinya bekerja di Apple selama 13 […]