It’s no secret that I’m an Apple fan. It’s not an exclusive relationship; I’ve also got a couple of Lenovo systems running Linux at home. But over the last decade, I’ve purchased multiple iPods, an iMac, a Mac Mini, a MacBook and my current 13” MacBook Pro, iPhone 4 and new iPad. I’ve spent more of my personal income on Apple hardware over the years than with any other single computer vendor. For the most part, Apple solutions work well for me in a wide variety of roles, both personal and professional. Clearly Apple stands to gain from long-term customer relationships like mine. Unfortunately, despite many positive experiences with Apple, not everything coming out of Cupertino is perfect.
Today’s long story began in mid-June, 2011, when I finally climbed aboard the iPhone bandwagon with the shiny new, white iPhone 4 on AT&T. The phone shipped with iOS 4.x; there was no reason to note the exact version number at the time, though it was probably 4.2.1. I set up my various work and personal e-mail addresses, synced my music, and was off and running into the world of iOS apps. Admittedly I was relatively late to the party. But I wasn’t the last. My employer has since all but abandoned BlackBerrys in favor of iPhones, and the BYOD phenomenon is slowly bringing in more iPhones and iPads here as time goes on. And with them came some strange problems, affecting me perhaps most of all.
During my first year of iPhone use, I twice ran into the following scenario. For a while everything would be fine. My iPhone was set to use Wi-Fi both at home and at the office, meaning that I only used 3G cellular data while in transit back and forth, or while out and about, traveling, and the like. Typically I’d use about 200 MB of 3G data per month, though my plan allowed for 2 GB. Suddenly I’d notice that my iPhone battery life – previously good from the start of the day to the end – would instead run down before the noon hour regardless of use or lack thereof. If I handled the phone, it would be warm to the touch. Finally, when I’d get the cell phone bill, the 3G data use had surged from 200 MB per month up to 2 GB or more, an increase of at least 900% from my norm. What the heck!? I talked AT&T into forgiving an overage charge the first time. I’ve since become a prolific user of the myAT&T app, keeping a close tab on data consumption patterns.
You could say that a significant part of my day job includes troubleshooting technical problems. Of course this is balanced with a desire to find a scenario that works and move on, rather than exhaustively analyze something with relatively minor monetary or business value. On the first occurrence of the battery drain + warm to the touch + 3G data explosion, I began closing apps, rebooting the iPhone, etc. No change. I did my Google homework, and while many seemed to be in a similar boat, I couldn’t find a solution. I may have called Apple, but didn’t keep a record of it. Finally, I wiped the device, set it up as if it were a new iPhone, and began syncing my apps back on, a few per day. I hoped to discover that a rogue app was the culprit. Eventually I had all the same apps back on, and the phone performed great for months. I hoped it was a one-time thing.
Some time after my initial encounter of battery drain + warm + 3G data surge, an IT colleague across the hall from me at work began experiencing the very same thing on her iPhone 4S on Verizon. Her phone had a connection configured to our corporate Microsoft Exchange e-mail server, but she’d literally made no other customizations. No Apple ID. No apps. Nothing. In hindsight, this should have been a major clue. I helped her wipe and reconfigure the device as if it were new, and she was off and running again. For months, but not forever.
When the scenario returned a second time on my iPhone, I contacted Apple Support on case number 311217544. (I’ll mention case and follow-up numbers as I go, on the chance that anyone inside Apple ever reviews this post.) I believe that this contact was on May 1st, 2012. I didn’t really get anywhere. As expected, the Support Agent was sure that it was a user problem, and gave me a list of settings to check and suggestions about how to reduce battery drain by hobbling certain functionality. Having been on the other side of the support desk, I know how easy it is to assume that the customer is using the technology wrong. How could there be something inherently wrong with the technology? Apple is perfect. By the way, I’d long since upgraded to iOS 5.x by this time.
Feeling more certain that there was an inherent but sporadic flaw somewhere in iOS, I wasn’t yet content to just wipe and reload, and go on with a few more months of normal use before the problem reared it’s head again. I tried reaching out to a couple of Apple employees who listed an e-mail address publicly on their LinkedIn profile. No response. I pasted what may have been a long-winded letter into an Apple.com web form, even offering to FedEx them my malfunctioning iPhone for analysis. I got an e-mail reply follow-up 205549136 indicating that my concern had been documented in case number 311234175. Eventually I wiped and reloaded my iPhone again, and went on with life. It’s not like I didn’t have better things to do.
Following Thanksgiving, 2012, I noted that both my iPhone 4 and a recently-acquired new iPad – both now running iOS 6.0.1 – were doing the battery drain + warm to the touch + 3G data slurp again. And here I’d been hoping that Apple finally got it right with iOS 6. I finally contacted Apple Support again on case number 383766567. This time I spent an hour and six minutes on the phone, and was advanced to a person with the title Senior Advisor iOS Tier 2. Before being escalated, I was advised to turn off cellular data, cut back on location services, and otherwise hobble the iPhone. After escalation to Tier 2 and sharing my history with this issue, I was advised to wipe and reload each device, and then put my apps back on slowly. Just as I had the very first time around. She couldn’t explain why my iPad was using cellular data even when it hadn’t left my apartment and available Wi-Fi in the several days leading up to our conversation. Based on the advice to put apps back on slowly, she was probably thinking rogue app. I wiped my devices, set them up as if they were new, synced back all the same apps, set up all the same e-mail accounts and settings, etc. And they’re once again fine.
As I alluded to several paragraphs ago, my employer uses Microsoft Exchange 2003 for e-mail, calendaring and the like. While Exchange 2003 may be a little long in the tooth at this point, it does the job, and we’ve been investing our efforts in many line-of-business application enhancements instead. When we have a new iOS device, we use the Exchange e-mail client native to iOS to connect to our corporate Exchange ActiveSync server. We then let the end-user manage everything else. For most of us, a connection to our Exchange environment may be the one thing that we have in common between our iOS devices.
This week our front-end Exchange ActiveSync server became unresponsive. First it had filled the disk space clear full with large IIS log files in C:\WINDOWS\system32\LogFiles\W3SVC1\, and I quickly cleared all but December’s logs to regain room. (Normally one should redirect their log files to a drive other than C:\.) Even after re-gaining quite a few GBs of disk space and rebooting, the server was still intermittently sluggish. I found myself restarting it two more times, and finally adding additional virtual processors and RAM. At the same time, I began examining these log files, which were between 40 MB and 100 MB per day. The logs cover every connection to Exchange Outlook Web Access and ActiveSync. It’s an understatement to say that there were a lot of connections. And that’s where I finally caught a break.
Out of thirty-five iOS devices that connected to our corporate Exchange environment that day, five of them (three iPhones and two iPads), were connecting to Exchange every two to three seconds, continuously, twenty-four hours a day, as long as they were powered on. That’s well past 10,000 individual connections per day for each of those five devices. To put it another way, this over-connection to Exchange was affecting 14% of our Apple inventory. If you factor in all the devices that have ever had this problem, it represents over 22% of our Apple iOS inventory. It bears mentioning that this behavior is not normal, even with Push e-mail enabled, as evidenced by the remaining 77% of our Apple devices that have apparently never done it.
But it gets more interesting. I found a log from just prior to wiping my iPhone, and it had been connecting to Exchange every two to three seconds on both Wi-Fi and cellular, all day and all night. No wonder the battery had been running down while the 3G data use had been climbing. When we surveyed the owners of our five over-communicative iDevices, they all confirmed that their battery life had been awful lately. One of them was using an iPhone 5 running just-released iOS 6.0.2. Wiping and reloading the devices has resolved their functional problems too. And our Exchange server logs confirm it.
So, we’ve narrowed the iPhone battery drain and data leak to a sporadic problem with iOS’s Exchange client as implemented in iOS 4.x, 5.x, 6.0.1 and 6.0.2, on both iPhone and iPad, while connecting to Exchange 2003. To be fair, we also have one very chatty Android 4.0.4 device in our environment, but nothing quite like the two to three second interval of the handful of runaway iOS devices. It’s completely possible that Apple’s Exchange client only has problems with certain versions of Exchange, and that some customers may not experience it at all. Going the other way, it could perhaps affect push e-mail more broadly, and not just Exchange. The hit-and-miss nature of it is troubling. At 22% percent of our devices affected at one time or another, this feels akin to a beta-quality experience; not what Apple is trying to sell.
Naturally I provided a summary of my most recent observations to Apple via e-mail, referencing case number 383766567. I received a call back the next day from our Senior Advisor iOS Tier 2. At first she asked me to wipe and reload a device, and put back Exchange accounts one at a time, as if to identify a problematic account. Of course several of our affected users only have a single Exchange account, and wiping the device has historically always resolved the problem for months at a time. This presents as a problem that, while sporadic, seems inherent to iOS, and manifests itself under an as-yet-unidentified set of potentially common circumstances. The support agent acknowledged that other companies may be similarly affected, and that they “haven’t put two and two together yet.” I offered to provide the log files so that Apple wouldn’t have to take my word for it regarding the connection intervals.
I expect to hear from Apple again next week. While I feel like I’m nearing the end of a 1.5 year odyssey – only to finally be taken seriously – my purpose in publishing this now is not to make anyone wrong. Rather, I want to share a summary of our iOS Exchange client problem – and the battery drain and 3G data consumption that accompany it – with a wider audience of IT peers, in order to better understand the true scope of the problem. As I mentioned to our Senior Advisor iOS Tier 2, I’d be thrilled if Apple simply fixed this problem in the next major iOS upgrade, and I never saw it again. Here’s hoping. I’ll share any newly-revealed information in a follow-up post. If you have anything to add to this conversation, please do so below.
[While I added a series of updates and a much easier fix here, and garnered some repeat visitors, I’ve since consolidated these into a second post.]