USB Host UVC driver version 2 #84

tore-espressif · 2024-11-05T17:12:27Z

Introduction

This PR introduces version 2 of the USB Host UVC driver, aimed at enabling ESP SoCs to stream frames from connected USB cameras.

The driver provides frames to the user via a callback, offering a flexible interface that can serve as a foundation for more advanced frame processing applications.

Internally, the driver manages a FreeRTOS Queue containing multiple frame buffers, which helps prevent buffer overflows and underflows during streaming.

Detailed feature descriptions, future plans, and usage instructions are available in the README.md file.

Information for Testers

This section provides instructions for testing the driver’s functionality and public API.

Two examples are included for testing:

1. `basic_uvc_stream`

Test Setup: Tested on ESP32-P4 with a customer-specified dual-camera setup.
Functionality: Opens two UVC streams and continuously starts and stops them.

2. `camera_display`

Test Setup: Tested on ESP32-S3 with multiple compatible cameras.
Functionality: Initializes a display, opens a single UVC stream, decodes received JPEG frames, and displays them on the screen.

Information for Reviewers

This section provides guidance for those reviewing the internal code structure of the driver.

The driver includes a host_test folder with tests designed to run on Linux. These tests use a mocked version of the esp-idf/usb component, focusing on the following areas, which should need less in-depth review:

Configuration descriptor parsing – uvc_descriptor_parsing.c
USB transfer callbacks – uvc_isoc.c and uvc_bulk.c

The following file, focused on human-readable output, also requires only a light review:

Descriptor printing – uvc_descriptor_printing.c

Areas not yet covered by host_tests and requiring more detailed review include:

Frame buffer management – uvc_frame.c (manages queue of empty frame buffers)
Control requests – uvc_control.c (handles video format negotiation and other control requests)
Core driver functionality – uvc_host.c (includes driver installation, device management, stream initiation, etc.)

Related Information

This PR supersedes Feat(uvc2): support bulk mode (IEC-154) #60.

host/class/uvc/usb_host_uvc_2/README.md

host/class/uvc/usb_host_uvc_2/host_test/main/parsing/descriptors/old.hpp

host/class/uvc/usb_host_uvc_2/host_test/main/parsing/test_parsing_customer_dual.cpp

host/class/uvc/usb_host_uvc_2/host_test/main/parsing/test_parsing_elp_h264.cpp

host/class/uvc/usb_host_uvc_2/host_test/main/parsing/test_parsing_logitech_c270.cpp

host/class/uvc/usb_host_uvc_2/include/usb/usb_types_uvc.h

host/class/uvc/usb_host_uvc_2/uvc_descriptor_printing.c

tore-espressif

@peter-marcisovsky Thank you very much for such a thorough review!

I fixed most of the things, answered other and left a few for later.

I left the changes in separate comment for now, so it is easier to review

host/class/uvc/usb_host_uvc_2/include/usb/usb_types_uvc.h

host/class/uvc/usb_host_uvc_2/host_test/main/parsing/descriptors/old.hpp

host/class/uvc/usb_host_uvc_2/host_test/main/test_main.cpp

host/class/uvc/usb_host_uvc_2/examples/basic_uvc_stream/main/basic_uvc_stream.c

host/class/uvc/usb_host_uvc_2/uvc_host.c

roma-jam

Hey Tomas,

I did the first round and it works! Cool!

Regarding the changes, I did check the:

READMEs and Description
Folder struct and APIs

Couple of notes I've already put, but to proceed, I think that I might meed some additional information.
Like requirements. I prepared the list of the questions, answers to them will help me understand what should I check and how.

What are the targeting versions of esp-idf for examples? (I've tried the basic_uvc_stream on v5.2 and there is fatal error: sd_pwr_ctrl_by_on_chip_ldo.h: No such file or directory 😐
What are the requirements for the driver? Should it notify the user while the device is attached to the ESP32 and then user should start streaming? Or it just provides the possibility to open the stream, when the device is already connected and user knows the vid/pid? More like the general idea. Using with hubs, several streams on one device or not, hot detaching (detaching cameras during streaming) and so on
Single-thread/Multi-thread. Does it support opening several streams on one physical device from several different tasks?

I have checked the docs/ folder, thanks for the png! (also, if we need to provide special additional information, such as "create" above the arrow, maybe we need to think about renaming the name of the API)

I will proceed with the review, feel free to address current comments.

host/class/uvc/usb_host_uvc_2/README.md

host/class/uvc/usb_host_uvc_2/include/esp_private/uvc_control.h

host/class/uvc/usb_host_uvc_2/include/usb/uvc_host.h

host/class/uvc/usb_host_uvc_2/private_include/uvc_types_priv.h

host/class/uvc/usb_host_uvc_2/uvc_bulk.c

roma-jam

Actually, couple of new comments, but nothing serious.
I will try to verify it on a hardware, should I have some special camera or anything I should know before testing?

Also, did you do any measurements to what extend it is better than the implementation, based on libusb?

UPD: Is our current example in esp-idf is compatible with this new class driver?

host/class/uvc/usb_host_uvc_2/uvc_host.c

host/class/uvc/usb_host_uvc_2/include/usb/uvc_host.h

host/class/uvc/usb_host_uvc_2/examples/basic_uvc_stream/main/basic_uvc_stream.c

host/class/uvc/usb_host_uvc_2/uvc_host.c

roma-jam · 2024-11-27T14:26:12Z

@tore-espressif
Sorry, I'm stuck trying to ~~claim~~ open a stream from my camera on S3.
Currently, I'm trying to understand what did I do wrong:

Will proceed with review shortly...

tore-espressif · 2024-11-27T14:31:42Z

@tore-espressif Sorry, I'm stuck trying to ~~claim~~ open a stream from my camera on S3. Currently, I'm trying to understand what did I do wrong:

Will proceed with review shortly...

@roma-jam that is really weird, Can you share full CFG descriptor?

roma-jam · 2024-11-27T14:49:48Z

USB 2.0 Camera
Sonix Technology Co., Ltd.
CNE-CWC2

Seems like we have it in host_test, as CANYON_CNE_CWC2. Which probably means that I do smth wrong.

UPD: I open the stream on the same device again. But the stream was already opened and interface was claimed, so I don't get any error on that and driver tries to claim the interface one more time. But it is already claimed, that is why the EP already allocated. Good.

09 02 7D 02 04 01 00 80 FA 08 0B 00 02 0E 03 00
05 09 04 00 00 01 0E 01 00 05 0D 24 01 00 01 4D
00 C0 E1 E4 00 01 01 09 24 03 02 01 01 00 04 00
1A 24 06 04 70 33 F0 28 11 63 2E 4A BA 2C 68 90
EB 33 40 16 08 01 03 01 0F 00 12 24 02 01 01 02
00 00 00 00 00 00 00 00 03 0E 20 00 0B 24 05 03
01 00 00 02 3F 04 00 07 05 83 03 10 00 06 05 25
03 10 00 09 04 01 00 00 0E 02 00 05 0E 24 01 01
33 01 81 00 02 02 01 01 01 00 0B 24 06 01 05 00
01 00 00 00 00 32 24 07 01 00 80 02 E0 01 00 00
77 01 00 00 CA 08 00 60 09 00 15 16 05 00 06 15
16 05 00 80 1A 06 00 20 A1 07 00 2A 2C 0A 00 40
42 0F 00 80 84 1E 00 32 24 07 02 00 60 01 20 01
00 C0 7B 00 00 80 E6 02 00 18 03 00 15 16 05 00
06 15 16 05 00 80 1A 06 00 20 A1 07 00 2A 2C 0A
00 40 42 0F 00 80 84 1E 00 32 24 07 03 00 40 01
F0 00 00 C0 5D 00 00 80 32 02 00 58 02 00 15 16
05 00 06 15 16 05 00 80 1A 06 00 20 A1 07 00 2A
2C 0A 00 40 42 0F 00 80 84 1E 00 32 24 07 04 00
B0 00 90 00 00 F0 1E 00 00 A0 B9 00 00 C6 00 00
15 16 05 00 06 15 16 05 00 80 1A 06 00 20 A1 07
00 2A 2C 0A 00 40 42 0F 00 80 84 1E 00 32 24 07
05 00 A0 00 78 00 00 70 17 00 00 A0 8C 00 00 96
00 00 15 16 05 00 06 15 16 05 00 80 1A 06 00 20
A1 07 00 2A 2C 0A 00 40 42 0F 00 80 84 1E 00 1A
24 03 00 05 80 02 E0 01 60 01 20 01 40 01 F0 00
B0 00 90 00 A0 00 78 00 00 06 24 0D 01 01 04 09
04 01 01 01 0E 02 00 00 07 05 81 05 80 00 01 09
04 01 02 01 0E 02 00 00 07 05 81 05 00 01 01 09
04 01 03 01 0E 02 00 00 07 05 81 05 00 02 01 09
04 01 04 01 0E 02 00 00 07 05 81 05 58 02 01 09
04 01 05 01 0E 02 00 00 07 05 81 05 20 03 01 09
04 01 06 01 0E 02 00 00 07 05 81 05 BC 03 01 08
0B 02 02 01 00 00 04 09 04 02 00 00 01 01 00 04
09 24 01 00 01 29 00 01 03 0C 24 02 01 01 02 00
01 00 00 00 00 0B 24 06 02 01 02 01 00 02 00 00
09 24 03 03 01 01 00 02 00 09 04 03 00 00 01 02
00 00 09 04 03 01 01 01 02 00 00 07 24 01 03 01
01 00 0B 24 02 01 01 02 10 01 80 3E 00 09 05 84
05 20 00 04 00 00 07 25 01 01 00 00 00

roma-jam · 2024-11-27T15:41:07Z

Same board, same camera. Different drivers.

UVC:

UVC_2:

Why the initial sequence of Get Curr/Set Curr/Get Max/Get Min is so different?

UPD: Also, I can't get why the UVC set Iface = 1 and Alt = 3, whether UVC_2 set Iface=1 and Alt =4.

Any idea?

roma-jam · 2024-11-27T16:14:02Z

Yep, I opened the stream, but I keep getting the following error:
ESP_LOGW(TAG, "usb err %d", isoc_desc->status); with USB_TRANSFER_STATUS_ERROR

and I still have no idea why. In process.

host/class/uvc/usb_host_uvc/uvc_bulk.c

host/class/uvc/usb_host_uvc/uvc_control.c

host/class/uvc/usb_host_uvc/uvc_frame.c

host/class/uvc/usb_host_uvc/uvc_host.c

host/class/uvc/usb_host_uvc/uvc_isoc.c

tore-espressif · 2024-12-11T20:40:18Z

The last commit adds the following:

Sudden disconnection reaction improved (no more transfers are submitted to the disconnected device)
Allowed external events handling (similar to MSC or HID class)
uvc_host_stream_event_data_t now follow open-closed principle: Open for extension, closed for modification

Overall, this is Release candidate for v2. Other features and bugs will be addressed in bugfix releases

host/class/uvc/usb_host_uvc/docs/uvc_frames_state_transitions.png

roma-jam · 2024-12-12T12:02:53Z

Yep, here is the log:

I (1334) main_task: Calling app_main()
I (1344) UVC example: Installing USB Host
I (1374) UVC example: Installing UVC driver
I (1374) UVC example: Opening UVC device 0x0000:0x0000-0
	[email protected]...
I (1374) main_task: Returned from app_main()
I (6374) UVC example: Failed to open device
I (11374) UVC example: Opening UVC device 0x0000:0x0000-0
	[email protected]...
I (16374) UVC example: Failed to open device
I (21374) UVC example: Opening UVC device 0x0000:0x0000-0
	[email protected]...
I (26374) UVC example: Failed to open device
I (31374) UVC example: Opening UVC device 0x0000:0x0000-0
	[email protected]...
I (32424) UVC example: Device 0x0000:0x0000-0 OPENED!
I (32594) UVC example: Stream 0 start. Iteration 0
W (32634) uvc-isoc: usb err 1
W (32634) uvc-isoc: usb err 1
W (32634) uvc-isoc: usb err 1
W (32634) uvc-isoc: usb err 1
... 
W (33764) uvc-isoc: usb err 1
E (33764) USB HOST: Enqueue URB error: ESP_ERR_INVALID_STATE
W (33774) uvc-isoc: usb err 1
...
W (33894) uvc-isoc: usb err 1
E (33904) USB HOST: Enqueue URB error: ESP_ERR_INVALID_STATE
W (33904) uvc-isoc: usb err 1
...
W (34434) uvc-isoc: usb err 1
E (34434) USB HOST: Enqueue URB error: ESP_ERR_INVALID_STATE
I (34444) UVC example: Device suddenly disconnected
W (34454) UVC example: Stream 0: Frame not received on time
I (34454) UVC example: USB: All devices freed
E (34774) HUB: Root port reset failed
W (35464) UVC example: Stream 0: Frame not received on time
W (36464) UVC example: Stream 0: Frame not received on time
W (37464) UVC example: Stream 0: Frame not received on time
W (38464) UVC example: Stream 0: Frame not received on time
I (38464) UVC example: Stream 0 stop
I (40464) UVC example: Stream 0 start. Iteration 1
E (40464) USB HOST: Get EP handle error: ESP_ERR_NOT_FOUND
E (40464) uvc: uvc_host_stream_unpause(769): Could not submit transfer 0

Is there is anything, that I could print of add, let me know.

This driver does not use libuvc anymore, it is native to Espressif's USB Host Library

tore-espressif · 2024-12-18T09:13:52Z

The last commit adds the following:
* Sudden disconnection reaction improved (no more transfers are submitted to the disconnected device)

* Allowed external events handling (similar to MSC or HID class)

* `uvc_host_stream_event_data_t` now follow open-closed principle: Open for extension, closed for modification
Overall, this is Release candidate for v2. Other features and bugs will be addressed in bugfix releases

Meanwhile

Found and fixed bug in esp-idf about handling PING protocol.
Added FAQ entry about frame buffer size

tore-espressif · 2024-12-19T06:50:15Z

@roma-jam Thank you for all testing!

I tried with 3 more cameras and could not get the same error as you... I'll merge this not to delay any further.

I'll get the same camera as you and fix in next 2.0.1 release

tore-espressif self-assigned this Nov 5, 2024

tore-espressif marked this pull request as draft November 5, 2024 17:12

tore-espressif force-pushed the feature/uvc_2 branch 6 times, most recently from 7adb8ec to e9a9097 Compare November 12, 2024 09:55

tore-espressif marked this pull request as ready for review November 12, 2024 15:37

peter-marcisovsky reviewed Nov 18, 2024

View reviewed changes

tore-espressif force-pushed the feature/uvc_2 branch from e9a9097 to 4157f1b Compare November 20, 2024 13:44

tore-espressif commented Nov 20, 2024

View reviewed changes

roma-jam reviewed Nov 22, 2024

View reviewed changes

tore-espressif force-pushed the feature/uvc_2 branch 3 times, most recently from 430f7d0 to f2e7547 Compare November 25, 2024 14:22

roma-jam reviewed Nov 25, 2024

View reviewed changes

roma-jam reviewed Nov 26, 2024

View reviewed changes

host/class/uvc/usb_host_uvc_2/include/usb/uvc_host.h Outdated Show resolved Hide resolved

roma-jam reviewed Nov 26, 2024

View reviewed changes

host/class/uvc/usb_host_uvc_2/examples/basic_uvc_stream/main/basic_uvc_stream.c Outdated Show resolved Hide resolved

roma-jam reviewed Nov 26, 2024

View reviewed changes

host/class/uvc/usb_host_uvc_2/examples/basic_uvc_stream/main/basic_uvc_stream.c Outdated Show resolved Hide resolved

roma-jam reviewed Nov 26, 2024

View reviewed changes

host/class/uvc/usb_host_uvc_2/uvc_host.c Outdated Show resolved Hide resolved

tore-espressif force-pushed the feature/uvc_2 branch from f2e7547 to 104b5c1 Compare November 29, 2024 11:59

tore-espressif mentioned this pull request Dec 2, 2024

Configuration descriptor larger than control transfer max length (IDFGH-8401) espressif/esp-idf#9868

Open

3 tasks

tore-espressif force-pushed the feature/uvc_2 branch from 104b5c1 to a516b6c Compare December 9, 2024 10:11

github-advanced-security bot found potential problems Dec 9, 2024

View reviewed changes

roma-jam mentioned this pull request Dec 10, 2024

feat(msc_host): Add format MSC support #101

Merged

6 tasks

tore-espressif force-pushed the feature/uvc_2 branch from a516b6c to df83b49 Compare December 11, 2024 20:33

roma-jam self-requested a review December 11, 2024 21:09

leeebo self-requested a review December 12, 2024 02:21

roma-jam reviewed Dec 12, 2024

View reviewed changes

host/class/uvc/usb_host_uvc/docs/uvc_frames_state_transitions.png Outdated Show resolved Hide resolved

tore-espressif force-pushed the feature/uvc_2 branch from df83b49 to 999f316 Compare December 12, 2024 12:26

tore-espressif added 2 commits December 17, 2024 18:10

feat(uvc): USB Video Class driver version 2

7cedb0f

This driver does not use libuvc anymore, it is native to Espressif's USB Host Library

feat(uvc): Latest UVC updates

2e8904e

tore-espressif force-pushed the feature/uvc_2 branch from 999f316 to 2e8904e Compare December 18, 2024 09:11

roma-jam approved these changes Dec 18, 2024

View reviewed changes

tore-espressif merged commit 555db55 into master Dec 19, 2024
31 checks passed

tore-espressif deleted the feature/uvc_2 branch December 19, 2024 06:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

USB Host UVC driver version 2 #84

USB Host UVC driver version 2 #84

tore-espressif commented Nov 5, 2024 •

edited

Loading

tore-espressif left a comment

roma-jam left a comment

roma-jam left a comment •

edited

Loading

roma-jam commented Nov 27, 2024 •

edited

Loading

tore-espressif commented Nov 27, 2024

roma-jam commented Nov 27, 2024 •

edited

Loading

roma-jam commented Nov 27, 2024 •

edited

Loading

roma-jam commented Nov 27, 2024

tore-espressif commented Dec 11, 2024

roma-jam commented Dec 12, 2024

tore-espressif commented Dec 18, 2024

tore-espressif commented Dec 19, 2024

USB Host UVC driver version 2 #84

USB Host UVC driver version 2 #84

Conversation

tore-espressif commented Nov 5, 2024 • edited Loading

Introduction

Information for Testers

1. basic_uvc_stream

2. camera_display

Information for Reviewers

Related Information

tore-espressif left a comment

Choose a reason for hiding this comment

roma-jam left a comment

Choose a reason for hiding this comment

roma-jam left a comment • edited Loading

Choose a reason for hiding this comment

roma-jam commented Nov 27, 2024 • edited Loading

tore-espressif commented Nov 27, 2024

roma-jam commented Nov 27, 2024 • edited Loading

roma-jam commented Nov 27, 2024 • edited Loading

roma-jam commented Nov 27, 2024

tore-espressif commented Dec 11, 2024

roma-jam commented Dec 12, 2024

tore-espressif commented Dec 18, 2024

tore-espressif commented Dec 19, 2024

tore-espressif commented Nov 5, 2024 •

edited

Loading

1. `basic_uvc_stream`

2. `camera_display`

roma-jam left a comment •

edited

Loading

roma-jam commented Nov 27, 2024 •

edited

Loading

roma-jam commented Nov 27, 2024 •

edited

Loading

roma-jam commented Nov 27, 2024 •

edited

Loading