Walkthrough to adding Lens IDs in Lightroom

Before you start:

You will need:

Lightroom – although this should work for anything else that can read XMPs, and for JPGs it should work for everything.
Exiftool Perl
Stuff to let you run perl scripts
A good text editor
Stickyback plastic


Non-RAW users - before you start!

Quick note for JPG/Tiff shooters before starting – this procedure is only reversible for RAW files - for everything else you're stuck with your changes. Lightroom seems to write EXIF directly to JPGs/JPEs rather than to XMP sidecars – if you do this for JPGs etc (which you can) then rather than editing a text file you’re editing your originals – be careful! When RAW users drag their XMPs into the BAT file to write LensID to XMP, JPG users instead drag their JPGs into the BAT to write LensID to the JPG – a backup is highly recommended!


Lens IDs in Lightroom

A nice little feature that Canon and Nikon users have had for donkeys’ years is the ability for programs such as Lightroom to take the lens used as another piece of exif data along with the likes of f-stop and camera body. While hardly essential it can help you consolidate a line-up by finding which lenses you never use and which constantly produce your best shots.

Minolta and Sony cameras do embed the lens used into the exif, but they do so in a different field to the Canon and Nikon files that Lightroom understands. Performing this little procedure reads the lens ID field produced by Minolta and copies it into the field understood by Lightroom.


Creating XMP files

The first step is to make Lightroom create what are called .xmp files for each and every shot that you want to read lens data from. These files are basically copies of the exif data embedded in the RAW files, without the picture information. Open one up in notepad and you’ll see a list of values tagged by Lightroom – everything from information on the shooting properties to who took the picture and with what camera is stored here. What’s not stored here (yet) is the lens type because Lightroom cannot read it from the RAW to place it into the file. XMPs are in addition to the RAW files themselves – they have the same filename and live in the same directory and if you later move a file in Lightroom it also moves its associated XMP. They don’t overwrite or interfere with the RAWs themselves in any way.

Creating XMPs is really easy – start by selecting all the files you want XMPing in Lightroom. If you have stacked shots then make sure that you 'expand all stacks' - if you don't then those stacked below the top level will be ignored when the XMPs are written. The Metadata tag at the top can be really helpful here – set it to filter by camera and you’ll get rid of all the compact camera snapshots that don’t need an XMP creating:





Secondly, ‘force’ Lightroom to write XMPs. For reasons that I don’t fully understand Lightroom won’t write an XMP unless it needs to – that is unless there’s metadata associated with the file that it can’t write to the RAW itself. So while Lightroom will quite happily write things like shutter speed and ISO value to the RAW, it doesn’t know how to write say copyright information or creator ID. To get around this add a little info to all of your RAWs to force XMP creation – I’m not sure exactly which fields will do the job but I use ‘Copyright’, ‘Copyright Status’ and ‘Creator’ which work fine and which I find useful anyway:





Once you’ve given all of your files additional data simply right-click the files in Lightroom and select ‘Metadata -> Write Metadata to file.’ Voila – XMP files for all of your RAWs.





Personalising your LensID table

The next stop is to read the Lens ID from the RAW files and write it into the XMP files. To do this we’ll be using a program called exiftool Perl. Exiftool comes in many variations and it’s apparently the most powerful exif editing tool around – it lets you edit the most fields. The big problem (for me!) is that it’s command line based – rather than say right-clicking and just typing into a lovely text box you have to write a command in a box telling it what to do – think dir/p, cd\... Yes I am old enough to remember all of that thank you! Someone did write a GUI for it, but unfortunately it doesn’t give access to the fields we need.

The huge advantage of the Perl version of exiftool over the standard version is that you get access to the Lens ID list. Sigma, Tokina, Tamron and the like use reverse engineering to give their lenses electronic compatibility with the Minolta mount – rather than creating their own little chips telling the camera what the lens attached is and what it does, they copy Minolta ones. This means that if we read the lens ID from the RAW file of say a Sigma 20mm f1.8, it won’t tell us that a Sigma 20mm f1.8 is attached – it’ll think that a Minolta 100-400 is. Unless you want to try to explain to your friends how you created crazy wide-angle effects with a 400mm lens I advise that you sort this out beforehand!

To do so go into the Exiftool/Lib/image/exiftool directory and find the file called Minolta.pm. Open this up in a decent text editor et voila – a list of Minolta lenses with their assigned ID.
What you need to do now is go through that list and work out where any conflicts might occur, tidying things up as you go. So let’s say that you own a Minolta 100-300 APO(D) – lensID 25. Sigma have used this lens ID on one of their lenses in the past, so in Lightroom it’s going to show up as:

'Minolta AF 100-300mm F4.5-5.6 (APO D) or Sigma Lens'

What I recommend you do is simply remove the reference to the Sigma Lens if you’ve only taken shots with a Minolta 100-300.

A bigger problem is when a lot of lenses use the same ID – shown by a decimal place after the lens ID. Take a look at LensID 25521:

25521 => 'Minolta AF 28-85mm F3.5-4.5 [New] or Other Lens',
25521.1 => 'Tokina 19-35mm F3.5-4.5', #3
25521.2 => 'Tokina 28-70mm F2.8 AT-X', #7
25521.3 => 'Tokina 80-400mm F4.5-5.6 AT-X AF II 840', #JD
25521.4 => 'Tokina AF PRO 28-80mm F2.8 AT-X 280', #JD
25521.5 => 'Tamron AF 19-35mm F3.5-4.5', #JD
25521.6 => 'Angenieux AF 28-70mm F2.6', #JD

Assuming that out of these you own an ultra-rare Angenieux AF 28-70mm F2.6 (what do you reckon one of those is worth now the a900’s out?), you’re probably going to be a little peeved when Lightroom tells you that you own a Minolta AF 28-85 instead. The solution is simple – delete all the other entries (25521.1 etc) and make entry 25521 read like this instead:

25521 => 'Angenieux AF 28-70mm F2.6', #JD

I don’t own two lenses in the same category, but if you did you could either edit the field to something like this:

25521 => ''Minolta AF 28-85mm F3.5-4.5 [New] or Angenieux AF 28-70mm F2.6'

or simply run exiftool in two batches – the first for files taken with the Minolta, the second for those taken with the Angenieux, changing the value of the LensID each time.

One more thing to note is that 3rd party lenses can show up as different lenses depending on the settings used. So my Sigma 18-125 would show up as a Minolta 24-105 if I took the picture at 24mm, or a Sigma 17-70 if I took it at 70mm… Great. However, don’t worry if you don’t get the lensIDs right the first time and you get weird lens IDs upon looking in Lightroom for the first time, there are ways to smooth things out afterwards. Also don’t worry about irreversibly writing the wrong lens ID to your RAW files – remember that all we ever edit in this procedure is the XMP files which can be deleted and re-created at any time – the RAW files are only read from and never edited!


Reading from RAW, writing to XMP

When you’ve edited the lens ID table to your satisfaction it’s time to let exiftool do its thing. As discussed earlier, there’s no easy buttons to click – it’s all text based. Fortunately we do have people at Dyxum who know how to write batch files – instructions that tell exiftool what to do, so huge thanks to Kiklop for preparing these.
Create a text file anywhere on your hard drive and paste the following:

@echo off
   :begin
   if _%1_==__ goto end
   echo Processing %1...
   D:\exiftoolperl\exiftool.pl -k -TagsFromFile %%d%%f.MRW "-lensId>lens" -ext XMP %1 -overwrite_original -r
   shift
   goto begin
   :end

Save the text file as a .bat – not as a .txt.

You’ll have to edit the above text to suit your own circumstances- if you didn’t install into D:\exiftoolperl then change that. Don’t use spaces in the directory name – exiftool doesn’t like it. There are also other variables that you can use – the above file will overwrite the XMP file we created earlier (but NOT the RAW – don’t worry!) – if you want a 2nd copy instead you can use the same code except “-overwrite_original” part of the code.

(If you want to learn more about exiftool commands you can visit here for some examples)

That file is for .MRWs (Minolta RAWs). If you want to do this for .ARWs or JPGs (read the warning at the start!!!) etc simply make the relevant changes - .ARW or .JPG instead of .MRW etc.

Now grab a bunch of XMP files (not the RAWs!) and drag them onto the bat file. Exiftool will search the RAW (or jpg or whatever) that goes with each XMP, find the lens ID and write it into the correct field in each XMP.


Done!!!

Now all we need to do is hope that we got things right! Back to Lightroom, select all the files that you just modified and right-click Metadata->Read Metadata from file to refresh the metadata. If all has gone well you’ll have a lovely list of Lens IDs just like mine!





If things went wrong - RAWs:

It’s quite likely that you didn’t correctly predict all the necessary changes in your Minolta.pm file. What we need to do now is to change anything that went wrong.

Having created the XMPs we’ve saved ourselves a lot of hassle. Even if we’ve written the wrong lensID, we’ve at least written it in a format that’s a lot easier to get at and simpler to understand than weird code embedded into a RAW file. With the XMPs written it’s now a simple case of editing them in a text editor and changing each false LensID to the correct one by typing it in. Back to UltraEdit.   Use search->replace in files to work on batches of files at one time. Here’s an example of UltraEdit in use:





In this case UltraEdit searches all of the XMPs in the selected directory, looking for instances of ‘T-mount or other lens’ (which is wrong) and replacing them with ‘Minolta 58mm f1.2’ (which is the lens that was actually used.) Because you only want to search the XMPs you can ignore the MRWs, ARWs etc in the directory – if you let UltraEdit search these too you’ll seriously slow things down – imagine converting a 24mp RAW file to text and searching it… No thanks!

Once the updates are complete go back to Lightroom and select 'read metadata from file' again to refresh the reading.


If things went wrong – JPGs/TIFFs:

Unfortunately your TIFFs and JPGs, not using xmp files, are not so easy to correct. The way to do it is still fairly simple, but you’ve got to be a little more careful. Instead of using UltraEdit to change text file sidecars, we’re going to individually write to the original file using exiftool – this time correctly.

Since you’ve now overwritten your original, one way to do things is simply to revert to your backup and re-process the Minolta.pm file using the knowledge that you learnt having got it wrong the first time. So say you missed a Sigma 17-70 that snuck in instead of your Minolta 24-105 – simply go back to the list of Lens IDs and make damn sure that there’s no mention of the 17-70!

If you’ve only got a few files to change, or if you’re confident with your ability to manually recognise each lens, then you can use exiftool to write directly to the jpgs, this time telling it exactly what to write instead of making it read from a file. Create a new .bat file for each lens that you want to add manually and add the following text:

D:\exiftoolperl\exiftool.pl -k -lens="Sigma 18-125mm F3.5-5.6 (D)" -ext jpe -overwrite_original %1 -r

Again, check your directory, lens-typeand file-type. The above example makes each file .jpe file dragged onto read Sigma 18-125 in Lightroom.

Again, once the updates are complete go back to Lightroom and select 'read metadata from file' again to update Lightroom's lens reading.

Really done!!!

And that’s it! Go through all your erroneous results like this and the job’s a good’un.

Good luck!!!


Edited by ABramley - 02 January 2009 at 17:13

Very well put together. Good job.

I shot a lot of my shots from a KM 5D in Adobe RGB, which leaves the files in .jpe format. These are the batch files I use to make the lens information readable; assuming that exiftool is installed at c:\Exiftool.

For JPE files
@echo
   :begin
   if _%1_==__ goto end
   echo Processing %1...
   C:\Exiftool\exiftool.exe -k -m -r "-lensId>lens" %1 -ext jpe -overwrite_original
   shift
   goto begin
   :end

For JPG files
@echo
   :begin
   if _%1_==__ goto end
   echo Processing %1...
   C:\Exiftool\exiftool.exe -k -m -r "-lensId>lens" %1 -ext jpg -overwrite_original
   shift
   goto begin
   :end

For DNG files
@echo
   :begin
   if _%1_==__ goto end
   echo Processing %1...
   C:\Exiftool\exiftool.exe -k -m -r "-lensId>lens" %1 -ext dng -overwrite_original
   shift
   goto begin
   :end

For ARW files and their respective XMP's
@echo
   :begin
   if _%1_==__ goto end
   echo Processing %1...
   C:\Exiftool\exiftool.exe -k -TagsFromFile %%d%%f.arw "-lensId>lens" -ext XMP %1 -r -overwrite_original
   shift
   goto begin
   :end

For MRW files and their respective XMP's
@echo
   :begin
   if _%1_==__ goto end
   echo Processing %1...
   C:\Exiftool\exiftool.exe -k -TagsFromFile %%d%%f.mrw "-lensId>lens" -ext XMP %1 -r -overwrite_original
   shift
   goto begin
   :end

When placed into the sendto directory in Windows, these can be run on entire directories (even nested) and can be run simultaneously if needed.

(for those using the perl format of exiftool just change exiftool.exe to exiftool.pl)

Hopefully this won't add any unnecessary confusion and will help someone out.

@ABramley: Thanks so much for this information! The Lightroom "help" (lower case and in quotes ) is anything but helpful on the subject. Just two questions.

1) I have to "press any key" to process each file in collection of files processed at the same time. Is there a way to get through a list of files with no intervention?

2) I can not process any more than 42 *.xmp files at a time without getting an error message. Is this a limit in the number of files that can be processed at once or a limit to the character length of %1?

Thanks... Brian

Well done ABramley

Is there a way of doing this using Mac OS X?

Yes, the -r key is set.

Kiklop wrote: I don't have any explanations about that one. Can you provide us with the error message you are getting?

The error message is:

The data area passed to a system call is too small.

Sounds like I need to expand the size of a data buffer, but which one and by how much is a mystery to me.

Cheers... Brian

I don't know...

When downloader pro runs that command it's running it on the directory of files that it's just renamed, placed in a folder and created XMPs for - so that's a lot of the stuff done for you. That command just says 'run this batch' which is the same batch file up above and the %d makes it do it on every file in the directory.

I don't know how to replicate that out of downloader pro - but tbh with us having to create XMPs to do this procedure and with it being fairly longwinded I'm happy to spend 20quid on a product that will do it automatically for me on import.

I think i need a little help here.

I downloaded all the items on the list using the links above, and Active Perl is running fine on my pc. i downloaded the exiftool perl and unpacked the tar.gz file. In this new directory there is no exiftool.pl file, just a file called exiftool with no extension. I then look at the Readme file, and there are instructions for compiling in windows using the nmake file they have linked to.

my problems arise when i try to nmake, it apparently calls on another file pl2bat.bat which is non existant and i get an error message:

--------------
D:\Exiftool>nmake

Microsoft (R) Program Maintenance Utility   Version 1.50
Copyright (c) Microsoft Corp 1988-94. All rights reserved.

        D:\Exiftool\perl\bin\perl.exe -MExtUtils::Command -e cp exiftool blib\sc
ript\exiftool
        pl2bat.bat blib\script\exiftool
'pl2bat.bat' is not recognized as an internal or external command,
operable program or batch file.
NMAKE : fatal error U1077: 'C:\WINDOWS\system32\cmd.exe' : return code '0x1'
Stop.
--------------------


do i need to get pl2bat.bat from somewhere, or am i going about this all wrong?

help is appreciated.

Thomas