Why your browser still cannot open the photos your phone takes

Every iPhone in the world takes photos in a format that Chrome, Firefox and Edge refuse to display. That has been true for nine years and it is not an oversight anybody is going to fix. While running the last benchmark I hit a second, completely separate reason our own server could not read them either, and that one turned out to be more interesting: it has nothing to do with patents and everything to do with the fact that an iPhone does not store your photo as a photo.

What is actually inside a .HEIC

Start with the file rather than the politics. Here is the box structure of an unremarkable photo straight off an iPhone, 2,554 by 2,554, 704 KB on disk, dumped with a thirty-line reader rather than a library so nothing is hidden:

ftyp (32)
meta (2251)
  hdlr, dinf, pitm, iinf
  iref (102)
    dimg: item 26 -> 25 references
    cdsc: item 27 -> 1 references
    cdsc: item 28 -> 1 references
  iprp
    ipco
      colr, clli, ispe, ispe, irot, pixi, hvcC
  idat (16)
  iloc (464)
mdat (719077)

The interesting line is dimg: item 26 -> 25 references. dimg is a derived image: item 26 is not a picture, it is an instruction for assembling one out of 25 other items. The recipe sits in that 16 byte idat box, and decoded it says:

idat payload: 00 00 04 04 09fa 09fa

  version 0, flags 0
  rows_minus_one    = 4   ->  5 rows
  columns_minus_one = 4   ->  5 columns
  output            = 0x09fa x 0x09fa  ->  2554 x 2554

The hvcC box in that dump is the HEVC decoder configuration, which is where the patents come in, and we will get to them. But notice that nothing so far requires a decoder at all. Everything above is container structure, and it is where our own server fell over, several steps before any codec was involved.

Reason one: our server counted to sixteen

SwiftConverter processes images with sharp, which handles every other format the site touches. Handed one of these files, it produces this:

Input buffer has corrupt header: heif: Invalid input:
Security limit exceeded: Number of references in iref box (25)
exceeds the security limits of 16 references. (2.1000)

Twenty-five references against a limit of sixteen. That is the 5 by 5 grid from the previous section being counted and rejected. It is not a codec error, not a corrupt file, and not a licensing refusal. libheif hardens itself against malicious files by capping how many items a container may reference, because an attacker can otherwise hand you a file that references itself into a denial of service. Reasonable defence. The cap is just set below what a completely ordinary iPhone writes.

The number comes from libvips, which sharp is built on. In the version we ship it reads:

So this one is a bug with a fix already written, and the honest summary is that a library upgrade removes it. Until then our HEIC path runs through heic-convert, a pure JavaScript decoder that has no such limit and returns the same file as a 2,554 by 2,554 image in about a second. Two decoders for one format is not elegant, but it is the difference between HEIC to JPG working and not working, and users do not care which library was involved.

Worth saying plainly because I got it wrong first: I originally assumed this was the HEVC patent problem showing up in our stack. It is not. The two failures look similar from the outside and have nothing to do with each other, which is exactly why the licensing story below gets told badly so often.

Reason two: nobody wants to pay for the decoder

HEIC is a container. What is inside it is HEVC, also called H.265, and HEVC is one of the most thoroughly patent-encumbered pieces of technology in consumer computing.

H.264 before it had essentially one place to go for a licence, and that made it easy to ship. HEVC did not repeat that. A second pool launched in 2015 with different members, different terms and fees the first pool did not charge, several patent holders declined to join either pool and licensed directly instead, and the result is that a company wanting to ship an HEVC decoder cannot establish what it owes or to whom by reading one document. The industry term for this is royalty stacking, and it is the reason HEVC adoption on the open web stalled while HEVC adoption on hardware people buy did not.

Browser vendors reached the obvious conclusion. The Chromium request to support HEIC has been open since 2017 and is closed as will-not-fix. Mozilla's equivalent bug has been open since 2017 at priority P5 with nobody assigned, and the reasons given are patent licensing and the absence of any cross-browser agreement to move together.

So why does Safari manage it?

Because Safari is not doing the work. macOS and iOS already carry a licensed HEVC decoder, since Apple ships hardware that records HEVC video and someone had to pay for that anyway. Safari hands the file to the operating system. Apple is a device manufacturer paying device-manufacturer royalties on hardware it sells at a hardware margin, so the cost lands somewhere it can be absorbed. Google and Mozilla ship a free browser onto other people's hardware and have no such margin to bury it in.

Windows shows the same split more honestly than anywhere else, because Microsoft puts a price tag on it in public. The HEIF Image Extensions package is free. The HEVC Video Extensions package, which is the part that actually decodes the pixels, costs $0.99 in the Microsoft Store, while a functionally identical build is given to PC manufacturers to preinstall at no charge to the buyer. That dollar is the clearest statement anyone has published about why your browser does not do this.

What this means if you just have photos

The practical situation, stripped of history:

• On an iPhone or a Mac, everything works and you will never notice the format exists.
• In Chrome, Firefox or Edge, a HEIC will not render, and most upload forms reject it because the server behind them cannot read it either.
• On Windows, it depends entirely on whether the machine came with the manufacturer build preinstalled. Two otherwise identical laptops can disagree, which is why this problem feels random to people.
• On Android, support is common on recent devices and absent on older ones.

If you want the setting changed at the source: iPhone, Settings, Camera, Formats, choose Most Compatible. The phone switches to JPEG at capture and the problem stops arriving. It costs storage, and the previous article measured how much, which is more than most people expect.

Converting, and what it costs

For photos you already have, the target depends on why you are converting, and the benchmark gives real numbers for the decision. Eight of these files, 7.5 MB as shot:

JPG if you need it to open somewhere specific, which is the usual reason and costs you almost nothing in size. WebP if you want it smaller. PNG only if something downstream demands PNG, because as the table shows it will multiply the file by eleven while looking identical.

The format that learned the lesson

The interesting epilogue is that the industry already built the fix. AVIF is the same idea as HEIC, a still image pulled out of a modern video codec, except the codec is AV1 and AV1 was designed from the start by a consortium including Google, Mozilla, Netflix, Amazon and Apple specifically so that no one would have to negotiate for it. The whole reason AV1 exists is the licensing mess described above.

It worked. AVIF renders in Chrome, Firefox, Safari and Edge, and in our benchmark it produced the smallest file in all six categories, at a higher average quality score than JPEG at quality 80. It is genuinely better than HEIC at the thing HEIC is for. It is also slow to encode, around 2.1 seconds per image against 45 milliseconds for JPEG on the same machine, which is why we do not offer it in a live request.

So the honest ending is not that HEIC is bad. The last benchmark found it holds roughly as much data as a quality-80 JPEG at roughly the same size, which is a good result. HEIC is a fine format attached to a licensing arrangement that made half the industry route around it, and your photo library is where that decision landed.

Sources and how to check this

• The box dump, grid geometry and reference count came from reading real files. The parser is thirty lines and the grid layout is defined in ISO/IEC 23008-12.
• The max_items values were read from the v8.17.0, v8.18.3 and master sources of libvips heifload.c, and the check itself is in libheif box.cc.
• Browser positions: Chromium issue 41389159 and Mozilla bug 1402293.
• Pricing is the public Microsoft Store listing for HEVC Video Extensions. I have not seen the pools' actual licence terms, so this article describes the shape of the licensing problem and quotes no royalty figures.