E-mount electronic protocol reverse engineering

As some of you might be aware, last year I started an effort to begin reverse engineering the E-mount electronic protocols.

That effort started over on dpreview at http://www.dpreview.com/forums/post/56133485 - However, certain limitations of that forum's architecture (inability to edit posts after they have been replied to, 200 post limit per thread at which point it gets automatically locked) makes it unsuitable to a long-term research/investigation project such as this, where things are continually discovered over the period of many months (at this point, 10 months and I still only have discovered a subset of what needs to be done if anyone wants to DIY an adapter or anything else that speaks the E-mount protocols).

Dyxum seems to be one of the most well-established forums for Sony cameras, so I figure this is the best place to organize my work somewhat better than that mess of a thread. (I can't even find my own pinout post and my own posts are highlighted in the thread view!!!)

Also, let me apologize in advance - I'm going to be making a few "reserved for X" posts in this thread with the intent of filling in the information later. I'd like to try to keep my initial data dump in a contiguous series of posts, but it's going to take me a while to fully fill everything in. Please bear with me as I organize/clean things over the next few days.

Edited by Micholand - 04 May 2016 at 17:49

High-level protocol and messaging - Initialization
Body sends message 0x29 to lens at 750kbps
Lens answers with message 0x29 at same speed - speed will not change unless noted otherwise
Body sends message 0x0A to lens
Lens answers with message 0x2B
Body sends message 0x0A to lens again with different data - this message appears to have multiple meanings
Lens answers with message 0x0A
Body and lens do some sort of speed negotiation handshake involving the CS lines without any data on the serial lines - this is the only time I've seen CS lines go high without data being transmitted
Body sends 0x0B to lens at newly negotiated speed (usually 1.5 Mbps)
Lens answers with 0x0B
[LA-EA3 adapter] Body sends message 0x10 to lens
[LA-EA3 adapter] Lens answers with a 0x10 message
Body sends message 0x11 to lens
Lens answers with 0xD2 - This has a large amount of information, and includes lens focal length ranges. Interestingly, these are stored as BCD values and are most significant byte first. For example, SAL55300 has 00 55 03 00. I am likely going to make an entire post just for this message as it has so much info
Body sends message 0x0D
Lens answers with 0x14
Body sends message 0x0A
Lens answers with 0x0A
Body sends message 0x0A again (different payload)
Lens answers with 0x0A
Body sends message 0x19
Lens answers with 0x19
Lens and body now enter operational state, continually looping the four-slot sequence described below

High level protocol and messaging - Ongoing operations
Once initialization is complete, the system moves into a state where there is a 60 Hz (for a US/NTSC body, ??? on an EU body) status and command loop. The high-level overview is as follows:

The loop starts when the body briefly drops the BODY_VD_LENS line to logic low for a very short period (TODO: Measure this and indicate actual time spent low here).

After this, there are two commands sent from the body to the lens.

Command slot 1 - Don't know yet, I'm suspecting aperture is in here but I've spent most of my time looking at command slot 2

Command slot 2 - Consists of one message from the set 0x16, 0x1B, 0x1E, or 0x27 - This command contains autofocus commands defined in <link to post with detail on message definitions for command slot 2 here>

Next there are two status messages sent from body to lens.

Status slot 1 - Don't know yet, I'm suspecting aperture status is in here.

Status slot 2 - This includes focus motor position and status. Messages seen so far are 0x30, 0x32, and 0x34 - Further detail in <link to post with detail on message definitions for status slot 2 here>

Edited by Entropy512 - 12 May 2016 at 02:27

Message types/definitions - Command Slot 2 (Focus commands)
Command Slot 2 will have one of four messages. In any given cycle, only one of these messages will be seen.

Message 0x16 - This appears to be a basic idle/housekeeping/placeholder message. It appears when the body is not commanding the lens to do something/waiting for the lens to finish a previously issued command.

TODO - put a few examples here

Message 0x1B - This handles basic absolute and relative positioning commands. It is the exclusive method for commanding an EA1/EA3 adapter or any adapter which emulates the EA1/EA3. It supports both absolute positioning (go to position X) and relative positioning (move Y steps) - Legacy adapted lens CDAF uses relative positioning exclusively, adapted lens PDAF (A7RII, A7II, A6300) uses absolute positioning exclusively. This is sometimes seen, although not often, when commanding native lenses. The criteria under which a body will issue a 0x1B to a native lens is TBD

TODO - Put some examples here, and some documentation on known byte fields

Message 0x1E - TBD, not seen often, may be a "stop what you're doing" command.

Message 0x27 - This appears to allow a body to command a lens to go into a semi-autonomous hunt mode - "move forwards/backwards at speed N, reversing when you hit the limit". This message is ONLY sent to native lenses and adapters that emulate them (such as Techart III) as far as I can tell. Techart III's handling of this command is incomplete - it seems to get the direction correct but does NOT handle speed control. As a result, lenses on a Techart III have a nasty tendency to overshoot/hunt - http://www.dpreview.com/forums/post/56692298

TODO - Post examples here, analyze further

Edited by Micholand - 04 May 2016 at 17:49

Message Types/Definitions - Status Slot 2 (Focus status)
This can contain one of three observed messages - ID 0x30, 0x32, or 0x34

Message ID 0x30 - Basic status message, typical during "idle/handling command" state - This information is currently summarized from http://www.dpreview.com/forums/post/56620518 - I'll flesh this out in more detail later

Bytes 7 and 8 contain the focus motor position in steps. This is the "current" focus motor position. This field is populated by both native lenses and EA1/3 adapters

Bytes 25 and 26 also contain a focus motor position in steps. This value usually leads the previous one by one cycle - it appears to be a "predicted motor position next cycle" field. It is ONLY populated by native lenses - EA1/3 adapters set this to zero

Bytes 31 and 32 contain what appears to be an absolute lens position in a form that allows calculation of subject distance - that is, this takes lens focal length into account. When a lens is focused to infinity (motor position as close to sensor as possible), this field is approximately 115*sqrt(focal length in mm) - For example it is 892 = 126.1*sqrt(50) when an SELP1650 is at 50mm and focused to infinity.

Message 0x32 - Appears to be issued by a lens when its status changes (such as direction change) - It has a few additional fields not present in 0x30, but has nearly all fields present in 0x30. Current motor position is bytes 9 and 10, "predicted"? motor position is at bytes 27/28. I think this may be a command ACK reply. On the EA1/3 adapter, the position fields are zero and this status is issued when the lens finishes moving. On native lenses, it seems to appear near the end of movement but not when movement is complete??? This is strange.

Message 0x34 - Appears to be issued by a lens when a command is complete. Like 0x32, it has a few additional fields not present in 0x30, but appears to have all fields that 0x30 has. Current motor position is bytes 11/12 - "predicted" position is never filled. This appears immediately after some focus motor commands, but not always even for the same command ID. Also, I have only seen it come from native lenses, never EA1/3 adapters and EA1/3 emulating adapters.

TODO: Provide examples, provide message lengths and more detail on 0x32 and 0x34

Edited by Entropy512 - 05 May 2016 at 13:39

Miranda F wrote: I have no experience of E-mount cameras and I'm not sure how you are checking the data flow*, but I tried something similar for A-mount and discovered that the camera can use different protocols according to the lens fitted. On the A-mount, it always starts with a basic legacy request for data on the original 5-pin protocol, and (possibly) depending on the result of this, the camera (certainly digital cameras and the later Dynax film cameras) sends requests from the camera on 8-pins for additional data which some lenses will reply to. The camera then determines how to talk to the lenses in future depending on the data and the conclusions it reaches.

Interesting. I also saw some information from bo-ming of Conurus (they do the electronics for metabones) indicating how they added SSM - http://www.dpreview.com/forums/post/57418698

So, it is important to appreciate that the camera can communicate in different ways according to what it is talking to. Does the E-mount use a completely different communications protocol from the A-mount, or is it just a superset?

Looks like I definitely need to prioritize finishing up the detail on the packet sequencing/basic message structure in that post (it's in the sigrok gitweb README but I haven't quite fully fleshed out/cleaned up that particular post - possibly tonight...) - It is completely different from A-mount based on what I've seen so far. Instead of being SPI-based,it's asynchronous serial with three additional handshaking lines. Two are for packet framing, one appears to kick off each cycle of the 60 Hz status and control loop.

As the E-mount cameras are capable of driving legacy Minolta lenses via an adapter, does the adapter do the whole task of translating between new and old protocols, or does the camera change mode to use the old protocol when it knows the adapter is fitted?

The adapter is a smart protocol translator. However, the protocol used by the adapter is a subset of the full native E-mount protocols - some messages are not used, and some fields in messages that are used are zero when talking to an adapter.

All Canon adapters currently on the market (if someone digs up this post years from now, it's May 2016 right now) except for the Sigma MC-11 when a GV lens is attached, the Metabones IV in Advanced mode, and the Techart EOS-NEX III in "Fn" mode emulate an EA1/EA3 adapter, so only implement that subset of the command protocols.

If you go up to my Command Slot 2 post, you'll see some indication of which commands are used by native lenses only. I'll be adding two command IDs that I just noticed last night to that post later. (they're used fairly rarely...)

In A-mount, there are a number of oddities which crop up with other manufacturer's products because they haven't fully understood the protocols or couldn't be bothered to implement them fully. So, for example, screw-drive Minolta lenses with 5 or 8 contacts will work with cameras with 5 or 8 pin contacts and TCs/extension tubes with 5 or 8 pin contacts, but some non-Sony/Minolta 8-pin screw-drive lenses won't work unless all 8 pins are present in the TC, evne though they have no SAM/SSM motor to need the extra contacts in the 8-pin connector.

Very interesting. Is there a place where a description of what is known of the 8-pin protocol is located?

Anyway I wish you success with what will probably be a very complex task!

Oh yeah... I've been doing this on and off for 10 months and still only know a tiny fraction of what is going on. Even for messages I've IDed, I only know the meaning of 10-15% of the fields in those messages. I'm scratching my head at two focus command messages which are quite long but seem to have "simple" effects - One always seems to drive the focus to minimum focus distance, one always seems to drive it to infinity - but the message is longer than the one used to drive absolute motor positions and relative motor positions...

*In my case I intercepted the comms by using an 8-pin extension tube to which I had added wires to an extension header.

Same approach here with a Meike extension. I'm actually surprised at how affordable A-mount extensions are despite the aperture lever. Pentax extension tubes are similar in complexity (pin passthroughs and aperture lever mechanical passthrough) and they're extremely expensive.

I think I am right in saying that the detailed byte-based analysis of lens data already published on Dyxum for A mount only includes the legacy (5-pin) data and doesn't cover the bidirectional SAM/SSM comms used by later non-screw-drive lenses.

Is there anywhere you know of that people have documented findings on the bidirectional comms?

Also, I wonder whether we need a new location for this kind of detailed hardware-based electronic discussion, particularly if it turns out that there are significant commonalities between the E-mount and A-mount protocols?

Good question - right now the two protocols are significantly different. I don't expect there to be more than 1 E-mount thread for the time being until there's enough known that people can try whipping up things that interface with E-mount lenses and bodies. We're a loooooong way from there right now. Although potentially at some point there may be merit to having threads involving hacking existing adapters. I've found out what microcontroller the Techart III uses (TI CC2540) and how to download firmware images. I did play around inside the Neewer NW-S-AF4 - which was a budget adapter that claimed to be firmware-updatable. There were some REALLY nice aspects of its internal hardware and construction that led me to believe it had huge potential for being a "hacktoy" adapter - until I discovered they miswired the USB D+ and D- pins to SWDIO and VSS on the microcontroller. OOOPS. At some point I want to see if there's a pad that I can hit SWDIO and SWCLK on the Viltrox EF-NEX II's LPC1114. FOTGA's adapter is cheap and poorly constructed, but it does use an ATMega8 with the ISP pins wired out to circuit board pads. Readback is disabled, but the adapter is so crappy I'm likely to just blow it away and workon my own firmware for it. (initially probably just playing with the Canon side of things, which is fairly well documented elsewhere such as https://pickandplace.wordpress.com/2011/10/05/canon-ef-s-protocol-and-electronic-follow-focus/ but I really need to play with it myself).

Miranda F wrote: Entropy512 wrote: Very interesting. Is there a place where a description of what is known of the 8-pin protocol is located? Not as far as I am aware. I discovered it by playing with TCs and mentioned it in one of my posts there, but as far I know, nobody here even posted that the 8-pin cameras actually send requests to the lens! (The traditional Minolta 5-pin was just unidirectional - clock out, data back). I guessed as soon as I saw the mis-identified SPI bus on the TC page here, so wasn't surprised it worked that way. re your dpreview link, I still don't think many people appreciate that SAM lenses communicate with a different protocol to Minolta screw-drive lenses, but send basic data out on the old protocol first.

At some point I want to get some A-mount extension tubes to look at how the EA1/3 commands the lens. Right now, my suspicion is that the A-mount SSM protocol and all current lenses only support relative and absolute motor position commands at full speed - I suspect there is no way to command a reduced-speed focus movement which is why I don't think we'll ever see REALLY good A-mount AF performance on anything mirrorless.

Miranda F wrote: Entropy512 wrote: Oh yeah... I've been doing this on and off for 10 months and still only know a tiny fraction of what is going on. Even for messages I've IDed, I only know the meaning of 10-15% of the fields in those messages. I'm scratching my head at two focus command messages which are quite long but seem to have "simple" effects - One always seems to drive the focus to minimum focus distance, one always seems to drive it to infinity - but the message is longer than the one used to drive absolute motor positions and relative motor positions... I would think that the focus to end messages contain complete setup information but the incremental focus messages are pared down to the bone for speed.

Probably unlikely... Most E-mount AF is tied pretty tightly to a 60 Hz BODY_VD_LENS pulse train, you can't issue multiple AF commands in a single cycle. It seems fairly timing-insensitive within a cycle - and the difference in message length at 1.5 Mbps is around .006 milliseconds per character. I need to check but the difference in length is maybe only 10 bytes, so .06 msec.

Miranda F wrote: Entropy512 wrote: I've found out what microcontroller the Techart III uses (TI CC2540) and how to download firmware images. I did play around inside the Neewer NW-S-AF4 - which was a budget adapter that claimed to be firmware-updatable. There were some REALLY nice aspects of its internal hardware and construction that led me to believe it had huge potential for being a "hacktoy" adapter - until I discovered they miswired the USB D+ and D- pins to SWDIO and VSS on the microcontroller. OOOPS. At some point I want to see if there's a pad that I can hit SWDIO and SWCLK on the Viltrox EF-NEX II's LPC1114. FOTGA's adapter is cheap and poorly constructed, but it does use an ATMega8 with the ISP pins wired out to circuit board pads. Readback is disabled, but the adapter is so crappy I'm likely to just blow it away and workon my own firmware for it. (initially probably just playing with the Canon side of things, which is fairly well documented elsewhere such as https://pickandplace.wordpress.com/2011/10/05/canon-ef-s-protocol-and-electronic-follow-focus/ but I really need to play with it myself). Interesting. I like the Atmel processors myself (designing a range of R/C speed controllers using one), but I note in passing that Kenko had to put a gate array in their 2x TC to make it work with SAM/SSM lenses - I guess because the delay in responding back to the camera in the bidirectional serial mode was too large. Which is why you need the alpha-AF types and not the M-AF ones, and even then not all the data is correctly filled out for all SAM lenses (See QuietOC's post on TCs, for example). I think one just has to be pleased that a legacy hardware option designed for other purposes (8-pins for xi lenses) actually manages to work as well as it does with modern SAM/SSM lenses, and not get too annoyed that 3rd party adapters aren't fully compatible (eg with data modification for FL, aperture).

Probably a limitation there imposed by SPI, where the body is clocking the data the lens is putting out - something wanting to modify the data would have to do so in less than a clock cycle. E-mount is async serial - as I mentioned above the timing of a given message seems not TOO sensitive and I'm fairly certain it could easily handle a delay of one or more characters. I'd love to see how the new E-mount TCs work.

Of note: Canon's EF teleconverters don't modify data, they instead signal to the lens that a TC is attached and the lens changes behavior. Much easier than altering SPI data on the fly.

overeema wrote: Miranda F wrote: ---snip--- Also, I wonder whether we need a new location for this kind of detailed hardware-based electronic discussion, particularly if it turns out that there are significant commonalities between the E-mount and A-mount protocols? I would favor a subforum to Lens Talk, which would enable to have separate threads for various subtopics. Great work!! btw my understanding is that the sensor size (APS-C, FF, ??APS-H??) also is communicated across the interface I would also suggest to reserve a thread on V-mount protocols, which are also at least partly reverse engineered

I'm not sure if there's a benefit to communicating the sensor size to the lens.

However we definitely know that image circle size (or at least, roughly quantized coverage, may only report FF vs. APS-C) is reported by the lens to the body, allowing for auto-crop. Also, we know based on the Sigma MC-11 and Phigment LM-NEX that the lens can report distortion/vignetting profiles to the body so that the camera does not need to maintain a database of lens profiles for correction.

I'm guessing that's somewhere in the init data, I'm hoping to put up some dumps of init sequences later this weekend.