FAQ FAQ  Forum Search   Events   Register Register  Login Login

DNG vs. ARW?

Page  <1234>
Author
pegelli View Drop Down
Admin Group
Admin Group
Dyxum Administrator

Joined: 02 June 2007
Country: Belgium
Location: Schilde
Status: Offline
Posts: 33669
Post Options Post Options   Quote pegelli Quote  Post ReplyReply Direct Link To This Post Posted: 09 July 2009 at 06:59
@ rovhazman: no copyright problems, don't think I'm the original owner of that workflow anyway.

@ polyglot: I also do that, I just make/store the DNG's for this << 1% probability that in the future there may no longer be software to open MRW/ARW's but still you would find programs that can handle DNG's

Don't know how I ever could sleep in the times you only had 1 negative or slide

Need to look into this par2 stuff. I've dabbled on it in the past but as my collection grows I really need to take the plunge and just DO it.
You can see the April Foolishness 2021 exhibition here Another great show of the talent we have on Dyxum
 



Back to Top
kevinbm View Drop Down
Senior Member
Senior Member

Joined: 23 September 2008
Country: United Kingdom
Location: Peterborough UK
Status: Offline
Posts: 1657
Post Options Post Options   Quote kevinbm Quote  Post ReplyReply Direct Link To This Post Posted: 09 July 2009 at 09:42
If I am storing video files on DVD backup (or even HD for that matter) I run them through winrar first to build a multipart archive and then create par files. I use a 20% redundancy setting so using this method I can lose up to 20% of the archive down to disc degradation / corruption and still rebuild it.

Edited by kevinbm - 09 July 2009 at 09:43
All the best

Kevin

Sony NEX-5
Nikon D4, D800 + glass

http://www.japonica-enterprises.co.uk/about.html
Back to Top
pegelli View Drop Down
Admin Group
Admin Group
Dyxum Administrator

Joined: 02 June 2007
Country: Belgium
Location: Schilde
Status: Offline
Posts: 33669
Post Options Post Options   Quote pegelli Quote  Post ReplyReply Direct Link To This Post Posted: 09 July 2009 at 12:33
Originally posted by Vivec Vivec wrote:

One thing you can do is use alphalensinfo on the original ARW file to generate an XMP file and then convert the ARW file to a DNG. Still, since the Sony "maker note" is discarded, there might be other meta information that is lost.


I was participating in a discussion on original raw vs. DNG on another forum where I brought up this point that not all data is transferred. In a response by another this was doubted very much and he claimed Adobe would never discard data but 'preserve it in an unparsed area'. However he's basing his knowledge on closer examining the differences between DNG's and their original NEF files (go figure), so he has no knowledge of our system and the way the DNG converter handles MRW/ARW files. So I seriously doubt his statement, but would also be interested in your reaction on it (as you are infinitely more knowledgable as I am on these kind of technical things)
You can see the April Foolishness 2021 exhibition here Another great show of the talent we have on Dyxum
Back to Top
sdm9465 View Drop Down
Senior Member
Senior Member

Joined: 29 February 2008
Country: Canada
Location: Nova Scotia
Status: Offline
Posts: 1180
Post Options Post Options   Quote sdm9465 Quote  Post ReplyReply Direct Link To This Post Posted: 09 July 2009 at 18:14
Originally posted by Vivec Vivec wrote:

A while ago, I was converting from ARW to DNG files since I liked the idea of a standard format. However, I found out that the dng conversion wasn't entirely lossless as the Sony metadata got lost -- in particular, the lens id is not preserved and one can no longer use programs like alphalensinfo to identify the lenses[/IMG]

This doesn't agree with my experience. I've been converting all of my .mrw and .arw miles to .dng on import using Lightroom since it was first released. Within the last six months I used ExifTool to modify all of my .dng files so that the lens used is shown in Lightroom. This worked for every single one of my .dng files so clearly the data wasn't stripped by the conversion. How were you converting to .dng?
Back to Top
Vivec View Drop Down
Senior Member
Senior Member
Knowledge Base Contributor

Joined: 17 October 2008
Location: United States
Status: Offline
Posts: 963
Post Options Post Options   Quote Vivec Quote  Post ReplyReply Direct Link To This Post Posted: 09 July 2009 at 20:13
Originally posted by pegelli pegelli wrote:

Originally posted by Vivec Vivec wrote:

One thing you can do is use alphalensinfo on the original ARW file to generate an XMP file and then convert the ARW file to a DNG. Still, since the Sony "maker note" is discarded, there might be other meta information that is lost.


I was participating in a discussion on original raw vs. DNG on another forum where I brought up this point that not all data is transferred. In a response by another this was doubted very much and he claimed Adobe would never discard data but 'preserve it in an unparsed area'. However he's basing his knowledge on closer examining the differences between DNG's and their original NEF files (go figure), so he has no knowledge of our system and the way the DNG converter handles MRW/ARW files. So I seriously doubt his statement, but would also be interested in your reaction on it (as you are infinitely more knowledgable as I am on these kind of technical things)


Well, I have to admit that I may be wrong about this. It makes sense after all to preserve all information.

Here is what I did: I parsed the DNG files and the Sony maker note is definitely not present any more. However, it may have been that it is hidden inside one of these "unparsed" areas in the DNG file. I remember looking for that too but I couldn't find it. Hmmm, I'll reread the DNG file format specification again this weekend and give it another try. Like you say, it may still be that the DNG conversion for NEF's preserves everything but that the Sony metadata is discarded.

Back to Top
Vivec View Drop Down
Senior Member
Senior Member
Knowledge Base Contributor

Joined: 17 October 2008
Location: United States
Status: Offline
Posts: 963
Post Options Post Options   Quote Vivec Quote  Post ReplyReply Direct Link To This Post Posted: 09 July 2009 at 20:19
Originally posted by sdm9465 sdm9465 wrote:

Originally posted by Vivec Vivec wrote:

A while ago, I was converting from ARW to DNG files since I liked the idea of a standard format. However, I found out that the dng conversion wasn't entirely lossless as the Sony metadata got lost -- in particular, the lens id is not preserved and one can no longer use programs like alphalensinfo to identify the lenses[/IMG]

This doesn't agree with my experience. I've been converting all of my .mrw and .arw miles to .dng on import using Lightroom since it was first released. Within the last six months I used ExifTool to modify all of my .dng files so that the lens used is shown in Lightroom. This worked for every single one of my .dng files so clearly the data wasn't stripped by the conversion. How were you converting to .dng?


Ha, this seems to confirm that I have been wrong since the exiftool could somehow find the lensid. I am very sorry for spreading misinformation and I have edited my original posts.

I converted using the dng converter command line tool. I only checked for the lensid using my own parser and clearly I have missed some specific section in the DNG file where the maker note is stored. I'll look into it again.
 



Back to Top
sdm9465 View Drop Down
Senior Member
Senior Member

Joined: 29 February 2008
Country: Canada
Location: Nova Scotia
Status: Offline
Posts: 1180
Post Options Post Options   Quote sdm9465 Quote  Post ReplyReply Direct Link To This Post Posted: 10 July 2009 at 18:00
Originally posted by Vivec Vivec wrote:

Ha, this seems to confirm that I have been wrong since the exiftool could somehow find the lensid. I am very sorry for spreading misinformation and I have edited my original posts.

I converted using the dng converter command line tool. I only checked for the lensid using my own parser and clearly I have missed some specific section in the DNG file where the maker note is stored. I'll look into it again.

I'm not sure if this helps but the method I used to modify my .dng files can be found in this thread.
Back to Top
Vivec View Drop Down
Senior Member
Senior Member
Knowledge Base Contributor

Joined: 17 October 2008
Location: United States
Status: Offline
Posts: 963
Post Options Post Options   Quote Vivec Quote  Post ReplyReply Direct Link To This Post Posted: 20 July 2009 at 09:01
Originally posted by Vivec Vivec wrote:

Originally posted by sdm9465 sdm9465 wrote:

Originally posted by Vivec Vivec wrote:

A while ago, I was converting from ARW to DNG files since I liked the idea of a standard format. However, I found out that the dng conversion wasn't entirely lossless as the Sony metadata got lost -- in particular, the lens id is not preserved and one can no longer use programs like alphalensinfo to identify the lenses[/IMG]

This doesn't agree with my experience. I've been converting all of my .mrw and .arw miles to .dng on import using Lightroom since it was first released. Within the last six months I used ExifTool to modify all of my .dng files so that the lens used is shown in Lightroom. This worked for every single one of my .dng files so clearly the data wasn't stripped by the conversion. How were you converting to .dng?


Ha, this seems to confirm that I have been wrong since the exiftool could somehow find the lensid. I am very sorry for spreading misinformation and I have edited my original posts.

I converted using the dng converter command line tool. I only checked for the lensid using my own parser and clearly I have missed some specific section in the DNG file where the maker note is stored. I'll look into it again.


Well, it turns out after all that I wasn't that wrong   

Here is the real story: in principle, the DNG converter tries keep all the original data of the original raw files, including the so-called "maker note" that contains manufacturer specific information (for Minolta and Sony, it contains for example the lens id).

The DNG converter stores such maker notes in a special area (DNG Private Data) in the DNG file -- so, yes, the lens id is still in the DNG file and I have upated the alphalensinfo utility to now work with DNG files too -- great!

Now for the bad news: since the maker notes are not standardized, the DNG converter cannot always preserve all the information since it doesn't always know where to find it. In general, it only preserves 'simple' values, like numbers, but not larger values like strings (*).



I have read that it is especially bad for Olympus raw files, but for Sony metadata the damage seems limited. For now, I have only confirmed that the DNG converter does not preserve the Sony "lens information section" as described by "Gigo" (see this thread. (alphalensinfo uses this to determine if this was a DT lens, and to construct a default lens description)

So, we can say that the DNG converter preserves at least most information but it is not entirely lossless with regard to metadata after all (which is of course really caused by the manufacturers that use all kinds of different standards for their maker notes)

(*) technically, the "base" of the maker note differs among different manufacturers, and any data larger than 4 bytes that is described by an offset to the base is not preserved in general (I guess Adobe can in theory add special code to handle this for different manufacturs). For Sony, it seems at first sight that all data larger than 4 bytes is not preserved and points to random data in the DNG file...

Edited by Vivec - 20 July 2009 at 09:13
Back to Top
chuck1856 View Drop Down
Groupie
Groupie

Joined: 10 February 2007
Country: United States
Location: MA
Status: Offline
Posts: 67
Post Options Post Options   Quote chuck1856 Quote  Post ReplyReply Direct Link To This Post Posted: 26 July 2009 at 15:21
Hi,

Possibly related question as part of this: How do you make sure that your program does in fact make a "physical" xmp side car file with the .ARWs so it can be moved?

I'm using lightroom and if I look in my file directory, I only see the .ARWs...no XMPs at all. I guess that data is kept in the lightroom catalog. I'd like to have it keep it with the files but don't know how to get it to put these files in explicitly...

thanks,
Chuck
A7ii, Sony 27-70/4 | M50/1.7 |M70-210/4 | M100-200 4.5-5.6 | LE-A4 | F43AM, F20
Back to Top
rovhazman View Drop Down
Moderator Group
Moderator Group
Knowledge Base Coordinator

Joined: 11 January 2008
Country: Israel
Location: Be'er Ora
Status: Offline
Posts: 9057
Post Options Post Options   Quote rovhazman Quote  Post ReplyReply Direct Link To This Post Posted: 26 July 2009 at 15:28
Hi Chuck,
If you want to create the XMP file (that's the only way to save your data outside of Lightroom catalog if you work with ARW files), you just need to select your files and press Ctrl+s (or save metadata from the menu).
Is this what you are asking?
Back to Top
chuck1856 View Drop Down
Groupie
Groupie

Joined: 10 February 2007
Country: United States
Location: MA
Status: Offline
Posts: 67
Post Options Post Options   Quote chuck1856 Quote  Post ReplyReply Direct Link To This Post Posted: 26 July 2009 at 18:50
That's sort of what I'm asking. I guess if this is all about "future-proofing", it seems you would want LR (or other file handler of choice) to keep your edits and data with the files. So that if your files are backed up and you do a recovery, you don't necessarily need to also successfully recover your lightroom catalog to get your photos "post-editing".

I guess this would be accomplished if LR would automatically make an .xmp by default and update it with any edits. If I have to remember to select the photos and do this periodically, chances are it will never stay up to date!

I also wonder about this if I want to move the files over to another drive. IE: Move off one pc to an external drive and then access that from lightroom on my laptop. Would it see the latest edits?

A7ii, Sony 27-70/4 | M50/1.7 |M70-210/4 | M100-200 4.5-5.6 | LE-A4 | F43AM, F20
Back to Top
rovhazman View Drop Down
Moderator Group
Moderator Group
Knowledge Base Coordinator

Joined: 11 January 2008
Country: Israel
Location: Be'er Ora
Status: Offline
Posts: 9057
Post Options Post Options   Quote rovhazman Quote  Post ReplyReply Direct Link To This Post Posted: 26 July 2009 at 19:19
As far as I know, there is an option to tell LR to automatically save XMP after every change, but I was told (and it makes sense) that it slows down the work. Just imagine writing to the disk every time you add keyword or slightly chaneg the exposure.
One thing that is useful, and I am using, is the smart collectoins in LR. You can ask LR to create a collection with all the files that were changed in the last xxx (it can be anytime between hours to years). I used the modified in the last 2 days collection (which I think it is there by default) and try to save every 2 days. In cases I am not sure, I am making a new collection of longer time and save it.

As for the external drive - if the file is on different drive, for LR it is a different file. So if you copy only the ARW without the XMP, you lose all your edits! (I did this mistake at the beginning...).
I am using the sofware Allwaysync to synchronize my 2 HDs. The software only copies the files that were changed.
Another way to keep the edits with the original file is to work with DNG, and that is the reason I started this thread.
Back to Top
Vamp898 View Drop Down
Newbie
Newbie

Joined: 07 December 2013
Status: Offline
Posts: 9
Post Options Post Options   Quote Vamp898 Quote  Post ReplyReply Direct Link To This Post Posted: 07 December 2013 at 19:02
I know this thread is a bit old, but i want to push a update here as there is no definitive answer yet on this.

I converted some of the ARW Files of my A99 using the KIPI DNG-Converter and i have to say, im positive suprised.

- I haven't lost any informations, i have pure RAW files which still need to be demoasiced and so on.
- The Files are smaller then the ARW (for example 24.3mb vs. 19.3mb)
- Lense Informations are all still there, as they have been in the ARW

I tested a lot and i could not find anything which goes wrong or worse when converting to DNG. I have the same quality than with the ARW.

It also brought some advantages. When converting ARW to DNG i can decide what size the embedded JPEG Preview should be (even FullSize is possible) and i can say i dont want any embedded JPEG preview at all which makes the files even smaller.

But i dont use Adobe to develop my RAW Files so i can not tell what Adobe does.

//Edit: Its also possible to embedd the full ARW File inside the DNG.

Edited by Vamp898 - 07 December 2013 at 19:08
Back to Top
jkp1 View Drop Down
Senior Member
Senior Member

Joined: 25 November 2005
Location: Denmark
Status: Offline
Posts: 2288
Post Options Post Options   Quote jkp1 Quote  Post ReplyReply Direct Link To This Post Posted: 07 December 2013 at 19:46
Personally, i couldn´t find any single valid argument at all for converting raw files to DNG.

Open standard, yea yea, but in the end who defines that open standard ? As defined in this so called open standard, other are allowed to add to and make whatever they want. This gives lot of possibilities for situations like fx the Android-situation.

Following Adobes cloud based PS and siblings, you never know when you have to pay more or whenever Adobe   make your life even harder.

I would advice not to mess around. Keep your original files. Those files contains all necessary informations. If …IF ….in the future, the original raw file manufacturer turns the key, you are always able to convert to whatever is used at that time.
Unused stuff in stock: A700, 24-105, 28-135, Sony A100, beercan, 100-200, Sony 70-200f2.8 + teleconverter
Back to Top
Dyxum main page >  Forum Home > Equipment forums > Digital Darkroom Page  <1234>

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.078 seconds.

Monitor calibration strip

Dyxum.com - Home of the alpha system photographer

In memory of Cameron Hill - brettania

Feel free to contact us if needed.