An Improved Liberal, Accurate Regex Pattern for Matching URLs

Tuesday, 27 July 2010

Back in November, I posted a regex pattern for matching URLs. It seems to have proven quite useful for others, and, even better, based on feedback from those who’ve used it, I’ve since improved it in several ways.

The problem the pattern attempts to solve: identify the URLs in an arbitrary string of text, where by “arbitrary” let’s agree we mean something unstructured such as an email message or a tweet.

So, here’s a pattern that attempts to match any sort of URL, using the extended multiline regex format that disregards literal whitespace and allows for comments, which explain a bit about how the pattern works:

(?xi)
\b
(                           # Capture 1: entire matched URL
  (?:
    [a-z][\w-]+:                # URL protocol and colon
    (?:
      /{1,3}                        # 1-3 slashes
      |                             #   or
      [a-z0-9%]                     # Single letter or digit or '%'
                                    # (Trying not to match e.g. "URI::Escape")
    )
    |                           #   or
    www\d{0,3}[.]               # "www.", "www1.", "www2." … "www999."
    |                           #   or
    [a-z0-9.\-]+[.][a-z]{2,4}/  # looks like domain name followed by a slash
  )
  (?:                           # One or more:
    [^\s()<>]+                      # Run of non-space, non-()<>
    |                               #   or
    \(([^\s()<>]+|(\([^\s()<>]+\)))*\)  # balanced parens, up to 2 levels
  )+
  (?:                           # End with:
    \(([^\s()<>]+|(\([^\s()<>]+\)))*\)  # balanced parens, up to 2 levels
    |                                   #   or
    [^\s`!()\[\]{};:'".,<>?«»“”‘’]        # not a space or one of these punct chars
  )
)

Here’s the same pattern in the terse single-line format:

(?i)\b((?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))

(And you thought the multiline version looked crazy, right?)

Here’s the test data I used while sharpening the pattern. Just like the pattern from November, it attempts to be practical, above all else. It makes no attempt to parse URLs according to any official specification. It isn’t limited to predefined URL protocols. It should be clever about things like parentheses and trailing punctuation.

In addition to being liberal about the URLs it matches, the pattern is also liberal about which regex engines it works with. I’ve tested it with Perl, PCRE (which is used in PHP, BBEdit, and many other places), and Oniguruma (which is used in Ruby, TextMate, and many other places). It should also work in all modern JavaScript interpreters. If you find a modern regex engine where the pattern does not work, please let me know.

Some of the advantages of the new pattern, compared to the previous one:

• It no longer uses the [:punct:] named character class. I thought this was universally supported in modern regex engines, but apparently it is not.

• It does a better job with URLs containing literal parentheses, correctly matching the following URLs that the previous pattern did not:

  http://foo.com/more_(than)_one_(parens)
  http://foo.com/blah_(wikipedia)#cite-1
  http://foo.com/blah_(wikipedia)_blah#cite-1
  http://foo.com/unicode_(✪)_in_parens
  http://foo.com/(something)?after=parens
  

• It now matches mailto: URLs.

• It correctly guesses that things like “bit.ly/foo” and “is.gd/foo/” are URLs. Basically: something-dot-something-slash-something.

Included in the parentheses-matching improvements is the ability to match up to two levels of balanced, nested parentheses — parentheses within parentheses. There are fancy ways of using dynamic or recursive regex patterns to match balanced parentheses of any arbitrary depth, but these dynamic/recursive pattern constructs are all specific to individual regex implementations. I.e., there’s one way to do it for PCRE, a different way for Perl — and in most regex engines, no way to do it at all. Hard-coding the pattern to support two levels of nested parenthesis should work everywhere, and, practically speaking, I only received two reports of actual real-life URLs that had a second level of parentheses, and none with more than two.

Lastly, I received several requests for a version of the pattern that only matches web URLs — http, https, and things like “www.example.com”. Here’s an extended format pattern that does this:

(?xi)
\b
(                       # Capture 1: entire matched URL
  (?:
    https?://               # http or https protocol
    |                       #   or
    www\d{0,3}[.]           # "www.", "www1.", "www2." … "www999."
    |                           #   or
    [a-z0-9.\-]+[.][a-z]{2,4}/  # looks like domain name followed by a slash
  )
  (?:                       # One or more:
    [^\s()<>]+                  # Run of non-space, non-()<>
    |                           #   or
    \(([^\s()<>]+|(\([^\s()<>]+\)))*\)  # balanced parens, up to 2 levels
  )+
  (?:                       # End with:
    \(([^\s()<>]+|(\([^\s()<>]+\)))*\)  # balanced parens, up to 2 levels
    |                               #   or
    [^\s`!()\[\]{};:'".,<>?«»“”‘’]        # not a space or one of these punct chars
  )
)

And here’s the same pattern in single-line format:

(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))

As before, suggestions and improvements are welcome, including just sending me example input where the current pattern fails. ★

Sorry, No, I’m Not Going to Write a Piece Arguing That Dan Lyons Is a Jackass

Thursday, 22 July 2010

So I get another call from my payola rep at Apple, and she’s like, “Hey, thanks so much for all the antenna-related links over the weekend. I just wanted to let you know how much we appreciate it, John.” They punctuate a lot of their sentences with your first name. Oldest trick in the book, but it works, even if you’re aware of it. Seriously. Anyway, I’m all like, “No problem, it’s my pleasure. You just keep those checks coming.” Laughs all around.

But then she gets serious, and says Apple will make it worth my while if I’d close out this antenna saga with a “Jackass of the Week” piece responding to Dan Lyons’s Antennagate story. So I say, “Who’d he write that for? I didn’t see anything on the Fake Steve blog about it.”

She says, “Newsweek, of course.”

“I thought they went out of business a few months ago.”

“No, they’re still around. I swear.”

She sends me a link to Lyons’s piece, “Apple’s Rotten Response”, which starts like this:

I wonder if panic has started to set in at Apple yet. If not, it should. Because today’s hastily called news conference — ostensibly to discuss problems with iPhone 4 and how Apple intends to fix them — only did further damage to Apple’s reputation.

Which is a polite way to start, because, well, we all know just how panicked everyone at Apple is feeling these days about the company’s prospects. (You’d be surprised at how many of the senior VPs at last week’s event reeked of booze, and it was only 10am. Nerves are frayed.) Lyons was kind to phrase it as a question.

So what should Apple be panicked about? Lyons’s thesis is that Steve Jobs has lost touch with reality, and the world is waking up to this. In this case, there’s something terribly wrong with the iPhone 4 antenna and Steve Jobs won’t admit it:

Some expected Apple might announce a recall of the phone. Others speculated it might announce some kind of software update that would improve reception problems. Instead, Apple CEO Steve Jobs came up with a two-part solution. Part 1: There is no problem. Part 2: Even though there is no problem, we’re going to give everyone a free case, which should insulate the antenna and prevent the interference that we just told you isn’t actually occurring.

I went back through my notes on the press conference, looking for the part where Jobs said there was no problem. I think it was right around the 13:00 mark in the video stream, where Jobs said, “And so the iPhone antenna went through all of this. We tested it. We knew that if you gripped it in a certain way, the bars are going to go down a little bit, just like every smartphone. We didn’t think it’d be a big problem, because every smartphone has this issue.” Or maybe it was a little later, toward the end, when Jobs said “A lot of people have told us, the bumper solves the signal strength problem,” where he claimed there was “no problem”.

So, Lyons concludes:

This is classic Apple behavior. No matter what the whole world can see with its own eyes, just keep saying that it isn’t true, and maybe, eventually, everyone will believe you. By refusing to acknowledge the problem, Jobs just reinforced the image of Apple as a company that is in deep denial and unable to admit a mistake — a company that has for so long been able to bend reality to suit its needs that it now has lost touch with reality itself.

I read that paragraph aloud to my payola rep, and told her that it was exactly why I’d be reluctant to criticize Lyons’s coverage. Apple really should have acknowledged reality. Sure, they’re giving out free cases because — in Jobs’s words — “A lot of people have told us, the bumper solves the signal strength problem”. Sure, Jobs said Apple “knew that if you gripped it in a certain way, the bars are going to go down a little bit”. Sure, the overwhelming majority of iPhone 4 owners seems delighted with it. But where’s the acknowledgement that the iPhone 4 antenna is this year’s Ishtar? Where’s the product recall? Where’s the “KICK ME” sign on Steve Jobs’s black shirt? Why does Bob Mansfield still have a job?

Jobs also said all other mobile phones suffer the same problems when you hold them in certain ways, and that “it’s a challenge to the entire industry.”

That’s ridiculous. It’s absurd. But that’s nothing new. Apple has a history of making ridiculous claims and having them accepted by an adoring fan base and worshipful press.

That’s the uncomfortable truth. The last honest man is Dan Lyons. Those videos from Apple showing other phones dropping bars? Fake. The similar videos on YouTube from owners of competing phones? Fake. Next thing Apple’s going to tell us, the Droid X has a flaky display.

With the launch of iPhone 4, for example, Apple pretended it had invented video chat — something that has been around elsewhere for years.

Lyons doesn’t name all those phones on which people are video chatting every day, because he doesn’t need to. Just look around and see them for yourself.

The real issue here is how the product is perceived. If you need to put a rubber case on a phone to make it work correctly, there must be something wrong with it, don’t you think?

Exactly. I mean, if the truth were that, in practice, the iPhone 4 works just fine without a case for most people — that it gets faster downloads and uploads and voice call quality is improved over the 3GS — well, that’d be a different story entirely.

Jobs clearly doesn’t. He seems scornful of customers who have complained.

It doesn’t show up on the video, but there was spittle coming out of Jobs’s mouth when he talked about the 0.55 percent of iPhone 4 owners who’d called Apple to complain about its reception. He seemed very upset about their gall.

Toward the end of the news conference, he blamed the media for blowing the problem out of proportion.

Apple’s rivals will have a field day with this.

Yes, one week out, this is looking like very good news indeed for Apple’s rivals. Apple’s goose is cooked and Lyons knows the score. People are going to look back at this piece a year from now and say, “By god, Dan Lyons saw it all along.” (Seriously: bookmark it.) It’s bad enough that I don’t have the courage to call Apple out on this blatant chicanery; the last thing I’m going to do is put my name on the line and argue that he’s a big dummy with a chip on his shoulder and that his work exemplifies the state Newsweek is in.

“You’re sure you won’t do it? We really think you could knock it out of the park,” my payola rep pleads.

“Sorry. Won’t touch it.” ★

Antennagate Addendum

Wednesday, 21 July 2010

A few follow-up points from yesterday’s “Antennagate Bottom Line”:

Regarding the Delta in Dropped Calls Between iPhone 4 and 3GS

One theory to explain this would be that the iPhone 4 antenna is simply worse than the 3GS’s antenna, perhaps even only because of skin coming into contact with the infamous spot.

Jobs’s “pet theory” is that iPhone 3GS owners were far more likely to have a case than 4 owners, which, interestingly, implies that even the iPhone 3GS gets better reception when in a case. (I asked about this during the tour of their antenna labs; the answer I got was “We don’t know.”)

Here’s another one, though, suggested by at least a dozen DF readers so far. Quoting from one such email:

In Antennagate Bottom Line, you mention the comparison of numbers of dropped calls, but I argue that this is not the right metric. What one needs to know is if the iPhone4 drops a call that would not be dropped by a 3GS. If the additional drops are in areas that the 3GS would have never connected in the first place, then the statistic isn’t telling us what everyone claims it is. All that would mean is that there is a large drop rate in regions that were previously regarded as dead zones. That’s an improvement, not a regression.

I’ll go out and acknowledge that this line of thinking is arguing that the iPhone 4’s higher dropped call rate is a good thing, which, on its face, sounds nutty. But is it outlandish? There are widespread reports — none better than Anandtech’s — that the iPhone 4 gets usable reception in areas where previous iPhones got none. Those may well comprise many of the extra dropped calls.

It’s also only fair to point out that I’ve also gotten many emails from DF readers who say they drop more calls with their iPhone 4 than their previous iPhones, from the same locations. I’ve gotten more such emails from readers claiming the iPhone 4 gets better reception, but for some, it’s worse. One thing I’d feel safe betting on: the extra dropped calls from the iPhone 4 are not evenly distributed among all iPhone 4 users. Some are getting a lot, and most are getting very few.

Regarding Apple’s Field Testing of the Antenna

Over the last few weeks I’ve probably gotten at least 200 emails posing the following theory:

We know from the Gizmodo stolen iPhone that the prototypes were disguised in cases when outside Apple’s campus. Maybe that’s why Apple missed this flaw in the antenna: they never noticed it on campus because they have a strong AT&T signal, and never noticed it off campus because the iPhones were always inside cases, and cases mitigate the skin-touching-the-spot problem.

That’s just not plausible.

For one thing, the strength of the AT&T coverage on Apple’s campus has no bearing on the testing they perform in their lab. There is no signal from AT&T inside those anechoic chambers. There is no signal from any external wireless source in those chambers. That’s the point of them. The way the chambers work is that they create their own little mini network inside the chamber. They run tests where they create strong signals, weak signals, and everything in between. They also run tests with people holding the phones being tested.

For another, they do test antennas in the field off-campus with no case. They do so using a fleet of about a dozen mobile testing labs. These are vans — more like small buses, maybe — which contain a slew of testing and measurement equipment.

The iPhone Gizmodo obtained was, in Apple’s internal lingo, a Design Verification Test (DVT) unit. These are one step below production units. My understanding is that when DVT units are deemed ready to go, the factories start churning them out as actual production units. Those DVT field tests are the final tests, certainly not the only tests. During the tour of Apple’s labs, Ruben Caballero — Apple’s senior antenna engineer, who led the tour — said the iPhone 4 antenna design had been in testing for two years.

Lastly, here’s what Steve Jobs said during the press conference:

Again you have to build these rooms, because if you don’t shield what you’re testing from all the outside interference, you don’t get accurate tests. And you can’t put your equipment in the room either. The equipment’s all got to be remoted outside the room. Now this is a state of the art antenna test facility. We have 17 anechoic chambers. These things are not cheap. We have invested over $100 million in antenna testing facilities over the past 5 years. We have 18 PhD scientists and engineers on our staff.

And so the iPhone antenna went through all of this. We tested it. We knew that if you gripped it in a certain way, the bars are going to go down a little bit, just like every smartphone. We didn’t think it’d be a big problem, because every smartphone has this issue.

Honestly, I thought the entire point of the lab tour was to reinforce this point: the iPhone 4 antenna is behaving exactly as Apple expected it would.

Laying It All on the Line

I posited yesterday that Jobs’s peevishness while announcing the free case giveaway had to do with the profits Apple is going to lose, which I estimated conservatively at $100 million. (On the analyst conference call yesterday afternoon, Apple estimated the cost at $175 million.) What I didn’t write about was whether I thought this was a good idea or not. I say yes.

Here’s the thing. Early last week this antenna story was spinning out of control. Letterman made a Top Ten list about it. Consumer Reports was posting updates every day, each getting a lot of traffic. CNN.com had a front page story stating that iPhone 4 owners could “fix” their phones with strips of duct tape.

It’s possible that if Apple had done nothing, the story would have died by now, perhaps drowned out by Apple’s spectacular quarterly results announced yesterday. I think they decided it wasn’t worth the chance — that if they did nothing, the fire might have gotten worse rather than died out.

And I think they decided, wisely, that if they were going to hit back in response to the story, they should hit back with everything they had. No use dribbling out responses one at a time. So: a live press conference, not just an open letter from Steve Jobs; a new section on Apple’s website specifically explaining Apple’s argument, and revealing their heretofore secret antenna testing facility; and, yes, free cases, for iPhone 4 users who are having signal problems that go away when the phone is encased.

In short: they weren’t going to take any chances. Except they did take a chance, insofar as Jobs mixed in a second message: media criticism. The message Apple needed to make was about the antenna (yes it has a weak spot, but it’s a worthwhile trade-off and it isn’t resulting in product returns, support calls to AppleCare, or a spectacular number of increased dropped calls) and about their concern for customers (we want them to be happy, so we’re waiving the restocking fee on returns and we’re giving a free case to any iPhone 4 user who wants one).

The extra message Jobs delivered was to the media: that they botched this story, that coverage was “so overblown it’s incredible”. Surely that’s what Jobs actually believes, and I think it’s the truth. I don’t think it was a good idea for Apple to make that case at the event, though. If the Antennagate PR problem was so dangerous that it warranted a significant response from Apple — and I think it did — then it was dangerous enough that it shouldn’t have been mixed with a message that might further antagonize the media — the very media whom Apple was clearly hoping would spread the facts Apple had presented regarding the antenna and its concern for customer satisfaction. It doesn’t matter whether Jobs was right; it distracted from the core message, which was all about dissipating the meme that the iPhone 4 antenna is severely flawed. ★