Problems Persist in iOS 6.1.2

In late December, 2012, I posted the saga of narrowing down a set of iPhone / iPad symptoms that periodically manifest themselves and usually arrive hand in hand.  Those symptoms include noticeably diminished battery life, surging 3G/4G data consumption, and devices that run warmer to the touch than normal.  If you’d like my history with this going all the way back to iOS 4.x, click here and then follow a link back to this post when finished with the first.  Today’s post is chapter two of what I hope is only a trilogy.

Quick Recap
When we left off in December, we’d narrowed our iPhone / iPad problems down to runaway interaction between problematic iDevices and my employer’s corporate e-mail server running Microsoft Exchange 2003 Service Pack 2.  These interactions are documented in the Internet Information Server (IIS) logs on our server running Exchange ActiveSync, and are stored by default in our case at c:\WINDOWS\system32\LogFiles\W3SVC1.  While a properly-functioning iPhone may connect to our Exchange server a few hundred times per day, a runaway device will connect up to tens of thousands of times per day.  This excess traffic will continue unabated on an offending device until manual intervention is taken.  If too many devices in an organization are doing this at once, it creates a sort of denial-of-service attack against the Exchange ActiveSync server.

It’s important to note that this issue may affect more than just the Exchange portion of the iOS e-mail client.  I’ve received and observed feedback regarding the client-side symptoms – battery drain, 3G/4G data surge and warm devices – from people connecting to other push e-mail services including Hotmail and iCloud as well.  Of course end-users and business network administrators like myself aren’t privy to what’s happening at the other end of those cloud services.

Since Then
Subsequent to my December post, this iOS / Exchange issue finally reached mainstream consciousness following the release of iOS 6.1.  Apple went as far as issuing a rare acknowledgement that an issue was known to occur when connected to Microsoft Exchange 2010 SP1 or later, and that it is triggered by responding to an exception to a recurring calendar event sent to a Microsoft Exchange account.  Apple said nothing of Exchange 2003 that I am aware of.

Apple Inc.
I talked about and documented several of my interactions with Apple in the December post. I shared my mild frustration in getting past their basic support to the point that they believed that I had a legitimate iOS issue.  On January 9th, 2013, my issue was escalated yet again to a person that I’ve been in fairly regular contact with since.  I provided raw log files at Apple’s request to be forwarded to their engineering team.  And I’ve had nothing but positive interaction with Apple since.  I know from Google Analytics that Apple is familiar with my original blog post.  As I’m writing this follow-up, my little blog has received 39 visits from a network labeled, ‘apple inc.’, another 8 visits from ‘apple inc. – 10g ashburn ide’, and 1 visit from ‘apple computer’ since the original post went live.

iOS 6.1.2
Apple released iOS 6.1.2 on February 19th, 2013.  The update description read in full, “Fixes an Exchange calendar bug that could result in increased network activity and reduced battery life.”  I received a few e-mails from friends as far away as China pointing out that our issue might now be fixed.  Three days later I sent out a memo to our iOS users asking any who had not already done so to upgrade to iOS 6.1.2.

Not Fixed
On Sunday, February 24th, a single CDMA iPhone 5 running iOS 6.1.2, noted in our IIS logs as ‘Apple-iPhone5C2/1002.146’, contacted our Microsoft Exchange ActiveSync server 69,878 times that day.  Keep in mind that there are only 86,400 seconds in a 24-hour period.  By way of comparison, my iPhone 4 checked in 336 times during the same span of time.  Upon discovery, I notified my contact at Apple and sent him the log at his request.

What To Do With Your Runaway iPhone
Since the December post, we’ve learned that fixing runaway iDevices is much simpler than our original course of action, which consisted of wiping them out and setting them up as if they were new.  Neither that action, nor simply deleting and re-adding an Exchange account, seems to be necessary.  In fact, resolving our latest runaway iPhone proved to be as simple as turning off the iPhone’s calendar sync on the Exchange account momentarily, and then turning it back on.
 

On your device, select Settings > Mail, Contacts, Calendars > (your account).  Turn off Calendars and 'Delete from My iPhone.'  Wait a moment, and turn Calendars back on.

On your device, select Settings > Mail, Contacts, Calendars > (your account). Turn off ‘Calendars’ and then ‘Delete from My iPhone.’ Wait a moment, and turn Calendars back on.


 
For Exchange Administrators
Some organizations are taking technical means to bar iPhones running iOS 6.1 through 6.1.2 from contacting their Exchange servers.  If you have a great many iPhones, this step may be absolutely necessary in order to keep your e-mail environment up for everyone else.  At the very least, you should consider alerting all users to stop accepting calendar invitations and updates via their iPhones and iPads.  For those of us with far fewer devices to manage, simply keeping abreast of the server logs and working with affected users may be enough.  As I write this, our IIS log for Saturday is already several times larger than it should be.  It wouldn’t be that difficult to automatically BULK INSERT each day’s log file into a SQL table and then query for the number of connection attempts by each user’s iPhone or iPad per day.  I work with a great DBA who may be called upon to do that for us if Apple doesn’t come out with a fix real soon.

Final Thoughts
While iPhones are wonderful pieces of technology that can do a great many things, there are exactly two functions that every smartphone must do reliably, bar none.  One is to make phone calls.  The other is to handle mobile e-mail, calendaring and contacts.  The fact that the premier device from one of the most well-regarded companies on earth has problems with one of these basic necessities is fairly disconcerting.  One wonders how hard it could possibly be to program in an upper limit to the number of attempts to update a single calendar invitation!?  I’m not yet at the point that I’m going to steer anyone away from the iPhone and iPad.  But I really hope that Apple can get this right in iOS 6.1.3 or soon thereafter.  Before I start giving BlackBerry a second look.

[While I hoped that this issue would ultimately be resolved by iOS 7, we’ve now seen this behavior in iOS 7 and 7.0.2 as well.  Follow this link to the next post.]

Tweet about this on TwitterShare on Google+Share on LinkedInShare on FacebookShare on RedditEmail this to someone

Comments

  1. Tumberry says:

    Your theory seems true, as the Usage and the Standby time appear to be the same even though the iphone is idle or locked.
    I actually even asked Apple in the support website, to create an app that can monitor the usage of cellular data generated from which specific app.
    E.g , i have sports app, or social media app, which always alerts me for events or whatsoever, now we do not know how frequently does this apps communicate with their servers.
    However , i personally feel that Apple will not do that, as already they know that their is some process running in the background that reports to them. I guess that’s how mostly in news u get to read that some criminal got caught by using an iphone.
    So if they were ever to make this app, we would definitely see some strange usage coming from the iphone that is not related to any app.

  2. Robert Parks r.a.parks says:

    On Saturday, March 2nd, one colleague’s new iPad running iOS 6.1.2 (Apple-iPad3C2/1002.146) contacted our Exchange ActiveSync server 79,818 times. The user resolved it by temporarily turning calendar sync off and turning it back on.

  3. Confirmed here as well.

    I see over a million instances of one username in one 24 hour (400MB) log file. That user is using an iPhone 5 – running iOS 6.1.2 – broken down further into what I think is the important number – 64,415 POST requests on 443…

    Like I said in comments on your first post on this – we’ve seen this since allowing iPads into the organization. 200, 300, 400MB logs generated by 5 or 10 users has been “normal” for these devices. I’d be happy to share with Apple if a higher level resource close to the issue needed another real world data point.

    For those who find this and want to quickly get a handle on how many requests are being processed – if you are not familiar with non-windows environments, I recommend grep (and head, and tail for that matter)… Grep is a available here: http://sourceforge.net/projects/gnuwin32/files/grep/2.5.4/ – pick either the setup or if you are like me download the bin and just extract the binary somewhere in your path.

    Once available in my path, I used the following command from the log directory to check for what I think represent unique transactions:

    type ex######.log | grep "443 domain\\username" -c

    Note: You need to escape the backslash character, so searching for DOMAIN\User turns into DOMAIN\\User in order to get the expected results. If you are having trouble with that, the following also seems to find the same posts in our logs:

    type ex######.log | grep "POST /Microsoft-Server-ActiveSync User=username" -c

    I’d be curious what others find in their logs and/or if there is a better indicator in the logs of what query constitutes of a “refresh” from the device?

  4. Robert Parks r.a.parks says:

    I received a call from Apple on the afternoon of May 4th, with a follow-up on the morning of May 5th. They would like to run a remote diagnostic on an iPhone or iPad while the runaway Exchange activity is occurring. So I spent a couple of hours sending myself meeting invitations and updates via both internal and external accounts in every scenario that I could think of. Neither my iPhone 4 (iPhone3C1) nor my new iPad (iPad3C3) are exhibiting the problem.

  5. Robert Parks r.a.parks says:

    Ethan, who provided great comments here earlier, today allowed Apple to run a remote diagnostic on a runaway iOS 6.1.2 device connecting to his Exchange environment. Ethan shared that after Apple was done with the device, he resolved the problem by turning calendar sync off and then back on.

  6. Hopefully the information provided to Apple will help them resolve this issue in 6.1.3 – for now we wait.

    I’m not sure about “resolving the issue” – the calendar tango is in my mind an effective workaround, but we don’t know what conditions actually cause the runaway situation with EXCH 2003, so there is nothing preventing the return of the issue on the device again…

    That said, the runaway device (that produced the logs for Apple) made 59,598 HTTPS POST requests yesterday from midnight ~ 3pm; today during the same period – 682… So yes, in my experience the workaround is very effective, at least in the short term. :)

    I have to wonder out loud how many (if not all) of the iPhone 5′s we returned under warranty for extremely poor battery life were actually experiencing this issue, undiagnosed.

  7. Jitendra Singh says:

    I have an iPhone 4 and updated to iOS 6.1.2 a few days ago. Shortly there after I started to experience a very shortened battery life and my iPhone getting quite warm to the touch. In trying to research the issue on the web I came across this post and took the high tech step of turning off MS Exchange Calendar and turning it back on. I also disable Push for the MS Exchange account. This action appears to have resolved the issue though it’s been only about 30 mins since I did this so I need to observe it longer to be sure. I too noticed the usage time to be excessive earlier. Hopefully someone at Apple can figure this issue out and resolve it or face the chorus.

  8. Apple just got back to me and I have to say I find the response a little disappointing.

    The agenda of the call was very clear: Point to a known Microsoft issue and end the call. Here is the information provided:

    http://support.microsoft.com/KB/967046

    Scanning the KB article quickly didn’t exactly put my concerns at ease. In fact it raised several questions. Most notably the KB states “Exchange 2003 treats the attempt to partner as a failure and prevents the device from synchronizing with Exchange 2003.”

    The devices for which I’ve experienced this issue with all were successfully synchronizing data. I’m curious how that can happen if this issue discussed in MSKB 967046 is indeed the root problem of the runaway apple devices. The representative was not able to speak to this concern.

    Not all devices have exhibited the problem. A fact that I also raised, thinking that if it was simply a server side issue, I’d be seeing this issue consistently with all devices of the same class/version/etc. Again, the representative was not able to speak to this concern.

    In all cases, once I track down the device that has the issue, turning off the calendar sync momentarily, and turning it back on has resolved the issue. Again, strange that a client side setting can correct the issue if it is a server side problem. The Apple representative was not able to speak to this concern. The user that produced the logs – today that device has connected 823 times, so 48 hours of “fixed” behavior and counting.

    I’m concerned that more cooperation between Apple / Microsoft / Verizon / AT&T / etc. would be prudent, since people are probably encountering this problem, not able to identify the root cause, and thinking that under 2 hours of battery life on a device that is hot to the touch is probably defective, which will result in needless warranty replacements, and lost revenue ultimately for Apple. I didn’t get the impression that voicing this concern to the Apple representative I was speaking with (which I did) was going to result in a change in policy. In fact, it was clear that beyond putting forth the statement that this was a server side issue, there was not much that was going to happen on the call.

    Since it is not a general release, the next step involves contact Microsoft Customer Service and Support to obtain the hotfix. I have a feeling that this will be a wild goose chase, and that the problem will persist until we eliminate either Exchange 2003 or Apple products from our environment.

    I wish I had better news to kick off the weekend.

    • Robert Parks r.a.parks says:

      Thanks for sharing this information! While I’m not convinced that a sporadic runaway iOS client can be solved by the server patch, I’ll definitely follow up on 967046 in order to rule it in or out.

  9. Robert Parks r.a.parks says:

    I opened a case with Microsoft on Saturday morning in order to get a copy of the Exchange Server 2003 hotfix mentioned in KB 967046. This required a $259.00 credit card commitment up front, though I was later told that the charge would be cancelled.

    Microsoft took no position as to whether this fix was appropriate to our situation. When they learned that our Exchange servers are virtualized on top of VMware 5, I had to acknowledge via return e-mail that we’re running an “unsupported environment.” We would apply the fix at our own risk.

    That evening, after first making a snapshot of our servers for recovery purposes, I installed the hotfix on the servers, made the specified manual changes, and rebooted. So far, everything appears to be performing properly since. We’ll not know for certain whether or not this update helped the situation for at least a couple of weeks to come.

    There is one reason to hope, however. Hours before I installed the updates, one of our iPhones cranked out 6,741 POST entries in 2 hours, 02 minutes and 55 seconds. The problem abated when the user shut her phone down. By the time she brought it back up, our servers were patched as described in KB 967046. So far, traffic between her iPhone and our Exchange server appears normal.

    If you’re experiencing iPhone woes and you happen to be connecting to an Exchange 2003 environment at your office, you might bring this post to the attention of your IT Department for their consideration.

  10. Robert Parks Robert Parks says:

    I’d hoped that Microsoft’s patch described in KB 967046 was our answer, but apparently it’s not. Yesterday a single iPhone 4S running iOS 6.1.2 connected to our Exchange ActiveSync server 47,600 times. The user’s battery declined from 100% to 65% this morning between the time that she got up, and her arrival at the office just after 8:00 AM.

    I talked with Apple today beginning at 11:20 AM Eastern. We installed a remote troubleshooting profile on the iPhone and then let it run for an hour. After that, we zipped up the phone’s diagnostic logs and uploaded them to Apple. By the time we were done, the iPhone had racked up another 47,350 connections to Exchange ActiveSync in today’s IIS log. I turned the phone’s Exchange calendar sync off and then back on to curtail the excess traffic.

  11. Robert Parks Robert Parks says:

    No word back from Apple yet regarding the diagnostic data that they collected on March 15th. Meanwhile, Apple released iOS 6.1.3 today. They don’t mention any fixes for Microsoft Exchange or any other e-mail connectivity.

  12. Dave Dickinson says:

    Very interesting – will be following this closely as we are having the same issues on our Exchange 2003 system with a few select iPhones and iPads. I’ve been using log parser and log parser lizard to track very high hits. Oddly enough, one phone we have which has already used 75% of its data quota by syncing doesn’t show abnormal amounts of hits from the IIS logs.

    I suspect we’ll have moved to Exchange 2010 before this is resolved.

  13. Robert Parks Robert Parks says:

    It’s now been a month since Apple released iOS 6.1.3. And while I’ve seen complaints about battery life generally, I’ve yet to see a single runaway iOS 6.1.3 device in our Exchange 2003 environment.

    [Edit: Spoke too soon regarding iOS 6.1.3. More details coming in a day or two.]

  14. Robert Parks Robert Parks says:

    On April 23rd, well over a month past the release of iOS 6.1.3, an iPad 2 (Apple-iPad2C3/1002.329) at my office connected to our Exchange 2003 Server 77,361 in one day. The same user’s iPhone only connected 163 times in that 24-hour period. The following day, the same user’s iPad and iPhone connected 76,220 and 266 times respectively.

    On April 24th, my new iPad (Apple-iPad3C3/1002.329) and iPhone 4 (Apple-iPhone3C1/1002.329) both began doing runaway connections to our Exchange server, connecting 74,604 times and 65,830 times respectively. After talking with Apple and reopening my prior case, I later turned Exchange calendar sync off and back on for each device to resolve the problem.

  15. Robert Parks Robert Parks says:

    On April 28th, another colleague’s iPhone 5 (Apple-iPhone5C2/1002.329) began the runaway traffic to our Exchange 2003 Server.

  16. zx9hippo says:

    any luck with improvements mitigating issues with 6.1.4?

  17. Thank you very very much!
    We have the SAME problem in our company!
    iPad or iPhone and Exchange 2003 SP2.
    Disable calendar it’s OK for us.
    Best
    Rick (from Italy)

  18. There are updates about this problem?
    Apple and / or Microsoft have FINALLY found a solution?
    Must necessarily wait for iOS 7?
    Thank you very much!
    PS: I want to open a ticket to Apple Italy, I can give your audience? You have a number of practice that I can show them? thanks again

  19. Pay attention!
    Today I’ve found this
    http://blogs.technet.com/b/exchange_it/archive/2012/02/24/script-per-risolvere-i-problemi-di-exchange-activesync.aspx
    and, in particular, this note, probably decisive!
    http://support.apple.com/kb/TS3398
    Today I’ll try on a e-mail (Exchange 2003) configured on an iPhone (can not remember if 4 or 4S with iOS 6.1.3) and let you know.
    Meanwhile, fingers crossed!
    See you soon!
    Rick
    P.S.: The Apple note is for iOS 4.0, but I installed the configuration profile correctly even on my iPhone 5 with iOS 7 beta 3, although I did not have this problem.

  20. Michael Helfrich says:

    we had these issues at my company and they seemed to drop off around August. Wondering if anyone else found the same or are your 6.1.3/4 users still having issues?

    Was very disappointed when iOS7 came out. No mention of a fix for this.

    • Robert Parks Robert Parks says:

      On September 28th, 2013, one user’s iPhone and iPad running iOS7 began exhibiting this behavior as well. A day or two later he upgraded the phone to iOS 7.0.2, and the problem didn’t abate. He was advised to turn his Exchange calendar sync off and back on, which seems to have resolved the problem.

Speak Your Mind

*

CAPTCHA Image

Reload Image