Testing a smart phone apps is a challenge. They are used by users who look at the screen and use a finger or two to interact.
You can plug a mouse into a smartphone, and use it to control the app as well.
Move the mouse and click as well as moving you finger and touching the screen.
It was easy to inject key strokes, this page explores using the same hardware to move the mouse and click.
It is easy to automate the mouse moves and clicks, however, it is REALLY DIFFICULT to do this correctly to control the app being tested.
A smartphone app was being used by the user for this service:
Some related pages:
I could use Arduino Leonardo to inject typing , and co-ordinate which Arduino was typing using a simple wire. I use the switch to disable the typing as required.
I set the call up, and in the picture above, I used my finger to select the input boxes so the input CARET is in the right place.
Once manually set up, I could turn on an automated stream of typing and they would type in turn.
If it kept working, all was well. I could go to lunch and hopefully it was still working when I got back! It does not know if the app has failed, and just keeps sending keystrokes!
It is also possible to program the Arduino Leonardo as a mouse. This is much harder than the keyboard, as the BIG UNKNOWN is how far to move the mose to the next click.
So this page is about exploring using the Arduino Leonardo as a mouse.
You can plug a mouse into a smartphone, and use it to control the app.
Move the mouse and click, as well as touching the screen with the finger.
The big unknown is how far to move the mouse to the next click.
I plugged a mouse using an OTG cable, and you get a cursor and you can click.
I could make the cursor bigs so that it was easier to see.
It is conceiveable that a camera could be used to look at the screen and see the position of the cursor.
A user is using a "closed loop" feedback system, as they can see what is going on, and adapt as required. It seems that using a HID device to automate is an "open loop" feedbacksystem, requiring dead reckoning. The HID spec allows a mouse to send absolute X & Y as well as the normal relative. If the mouse could be told where the cursor was, it may help. A way to force the the position to a known point would be helpful, so the errors associated with dead reckoning can be reduced.
https://www.testdevlab.com/blog/how-we-built-a-robot-for-automated-manual-mobile-testing
https://www.iphoneness.com/smart-robots/mobot-smartphone-tester-robot/
Knowing that using robots was not a realistic option in the short term,
The App needed a slow stream of repeatable text to soak test it and fill up scroll buffers, over many minutes.
When I tried to get two devices to type together I wrote a simulation on a webpage:
Simplex, Duplex Full Duplex and Half Duplex are terms from telegraphy that are relevant.
Instead of using an Arduino to inject keystrokes as a keyboard, use the Arduino as a mouse to type on the soft keyboard.
https://docs.arduino.cc/built-in-examples/usb/KeyboardAndMouseControl/
For the keyboard version, the built in library converts letters to key presses.
I just had to read each letter from a string, and print each letter and pause. I had to work out how to hold off typing if the other device was typing.
If you start at the DigiSpark? example, with little knowledge of Arduino Raspberry PICO and basic C knowledge, the proof of concept AutoType takes about 2 hours to get working. It provided active service and found many issues on the edge of normal use. One was using \n that simulated pressing < enter >. It lost the focus, so the CARET moved out of the input box. Simple workaround - have a version which did not send < enter > when it found \n in the string. One soak test went for three hours and that is a lot of test time.
Over six years later, I am exploring using Robots, and Arduino and Raspberry Pico as a mouse, and trying to understand the USB HID code, which is interesting, but serveral orders of magnitude harder to get a working useable solution.
If I want my sketch to act as a mouse, my sketch would need to map the letters onto positions on the soft keyboard and move the mouse.
Other people have used a delta-x robot and dabber.
They probably need to sse a camera to see where the mouse is on the smartphone screen. and detect if it has crashed.
You can have multiple keyboard and mice plugged into a USB hub plugged into the Device undertest.
https://www.usb.org/hid - usb documentation.
This Android documentation may help.
https://developer.android.com/reference/kotlin/android/view/MotionEvent
The USB spec supports RELATIVE and ABSOULTE mouse movements. The examples that come with Arduino only have RELATIVE examples. Many people report converting code to ABSOLUTE mouse movements.
The Arduino documentation does not have examples for HID absolute yet, but there are examples on the web. Windows supports ABSOLUTE mouse movements.
https://github.com/jonathanedgecombe/absmouse has an example for me to try. A modified version worked on the PC but not on my Smartphone
Searching HID ABSOLUTE throws up many results. I remembered trying sending keycodes using Tiny C and Windows.
https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mouse_event
If MOUSEEVENTF_ABSOLUTE value is specified, dx and dy contain normalized absolute coordinates between 0 and 65,535. The event procedure maps these coordinates onto the display surface. Coordinate (0,0) maps onto the upper-left corner of the display surface, (65535,65535) maps onto the lower-right corner.
This is an example, that could be explored.
https://github.com/NicoHood/HID/blob/master/examples/Mouse/AbsoluteMouse/AbsoluteMouse.ino
The device can be used Landsacpe or Portrait. There are many different softkey boards available on my smartphone.
AutoTypeWorkMoreNotes has a screen shot of the softkeyboard.
The letter needs to be converted to mouse co-ordinates
http://www.dougrice.plus.com/dev/keyboard.c - reading keyboards on Raspberry Pi
http://www.dougrice.plus.com/dev/DigiSpark/ has keyboardMouse.c which tries to send mouse codes using TinyC?.
It would be worth consulting the HID specs.
https://www.usb.org/hid - usb documentation.
https://github.com/microsoft/hidtools - a tool that has an example for a mouse.
https://github.com/digistump/DigisparkArduinoIntegration/blob/master/libraries/DigisparkMouse/DigiMouse.h - contains the HID descriptor and code for the move()
To summarise, when the mouse is moved it sends reports. These are small and do not contain if the mouse movements are relative or absolute.
When the mouse is plugged in, it sends a mouse_usbHidReportDescriptor? that states if the mouse sends absolute or relative increments.
On the DigiSpark? this is stored in PROGMEM, making it harder to change at runtime.
PROGMEM unsigned char mouse_usbHidReportDescriptor?[] = { /* USB report descriptor */
//The? XY movements are described here.
0x05, 0x01, // USAGE_PAGE(Generic Desktop) 0x09, 0x30, // USAGE(X) 0x09, 0x31, // USAGE(Y) 0x15, 0x81, // LOGICAL_MINIMUM (-127) 0x25, 0x7f, // LOGICAL_MAXIMUM (127) 0x75, 0x08, // REPORT_SIZE (8)
0x95, 0x02, // REPORT_COUNT (2) // 0x81, 0x06, // INPUT (Data,Var,Rel) 0x81, 0x02, // INPUT (Data,Var,Absolute) // dhr 20-05-20
Changing the Rel to Absolute makes the mouse absolute instead of relative.
You could have two mice, one set up for Relative and the other for Absolute.
However you still do not know where the App window is on the PC screen, and it does not seem possible to send the mouse a message to say where it is.
So the normal method is to move the mouse so that the mouse pointer hits the window edges.
It you move it left and assume it has hit the left edge, you can then move it right to the desired position. This probably needs visual calibration for each screen and device.
It might be possible to use Assistive devices for the visually impaired, but I need to find an example.
It would be nice if the mouse could be told when it hit a local screen boundary, like the edge of a button.
Using this example installed for the DigiSpark?:-
I disabled the mouse clicks and only tried move.
I modified it and made the DigiSpark? Absolute.
// 0x81, 0x06, // INPUT (Data,Var,Rel) 0x81, 0x02, // INPUT (Data,Var,Absolute) // dhr 20-05-20
0x09, 0x38, // Usage (Wheel)
However the sketch did not display the mouse on the smartphone, but worked on the PC
So use the DigiSpark? example for the mouse. The sketch below clicks each button on the softkeypad on my smartphone.
DO NOT RUN ON PC UNMODIFIED. It is "open loop" and does now what is is clicking.
http://www.dougrice.plus.com/dev/DigiSpark/MouseDigiSparkSmartphoneClick.ino
It calibrates the cursor by moving left and down so the cursor if off screen. It needs calibration for each device and soft keyboard.
It is a proof of concept - It needs more work to be used to autotype.
It works for the softkeyboard screenshot at the bottom of this webpage.
It clicks each letter on the Gboard English (UK) (PC) on a MOTO G14.
To convert the Arduino Leonardo mouse example from relative to absolute I tried this.
Add some mouse moves to the loop. We will see if they are relative or Absolute.
delay(1000); Mouse.move(100,100); delay(100); Mouse.move(50,50);
Save the sketch
Copy Mouse.cpp and Mouse.h from C:\Users\dough\AppData?\Local\Arduino15\libraries\Mouse\src into the sketch's folder.
Edit mouse.cpp to convert INPUT (Data,Var,Rel) to INPUT (Data,Var,Absolute)
static const uint8_t _hidReportDescriptor?[] PROGMEM = {
// Mouse //... near the bottom.
0x95, 0x03, // REPORT_COUNT (3)
// 0x81, 0x06, // INPUT (Data,Var,Rel)
0x81, 0x02, // INPUT (Data,Var,Absolute) // DHR
0xc0, // END_COLLECTION
0xc0, // END_COLLECTION
}
Check if you compile Mouse.cpp by adding a deliberate syntax error.
This worked for my Win11 and RaspberryPI? PC, but the Android smartphone does not display the cursor.
I am wondering if the Android Smartphone like the Ardurino Leonardo that enumerates as a Keyboard, mouse and serial port!
I tried the Raspberry PICO USBHID MOUSE example. It moves the mouse on my Smartphone and PC
void loop() {
// put your main code here, to run repeatedly:
delay(1000);
Mouse.move(10,0);
delay(100);
Mouse.move( 20,0);
delay(100);
Mouse.move( -30,0);
}
This moved the mouse two steps right, one step left, but did not return to the same place.
The manual mouse could be used to position the cursor and CARET. The AutoMouse? could be used to move RELATIVE around the soft keyboard and a switch could enable mouse clicks.
This could be a way forward for now. Using an AutoMouse? to tap away on the soft keyboard is closer to the human using the App.
I did try and find USBMouse.cpp and USBMouse.h
I copied USBMouse.cpp and USBMouse.h into the sketch's folder. It still needs some work to get it to build using the local copy. The implementation is more complex, so converting it from relative to absolute is harder.
I used a USB-C OTG dongle to plug a hub into my smartphone and I could plug a mouse in to the hub.
I seems to have installed Nico Hood's Absoulte Mouse but it needs more tinkering.
To make the mouse move absolute, this worked on the PC.
// USBMouse Mouse; //USBMouse? Mouse(true, ABS_MOUSE); USBMouse Mouse( false, ABS_MOUSE);
The sketch can move the mouse relative on the Android smartphone. If rebuilt to use ABS_MOUSE, the mouse moved on the PC but not on the smartphone. Larger movements were also tried.
USBMouse Mouse; // USBMouse Mouse(true, ABS_MOUSE); // USBMouse Mouse( false, ABS_MOUSE); void setup() {
// put your setup code here, to run once:} void loop() {
// put your main code here, to run repeatedly: delay(1000); Mouse.move(10, 10); delay(100); Mouse.move(20, 10); delay(100); //Mouse?.move(-30, 10); delay(1000); Mouse.move(10, 20); delay(100); Mouse.move(20, 20); delay(100);}
This Android documentation may help.
https://developer.android.com/reference/kotlin/android/view/MotionEvent
You can plug in a hub and a second mouse that moves the mouse cursor.
Other people have reported it not working:-
https://forum.pjrc.com/index.php?threads/absolute-mouse-with-teensy-3-2-android-ios.63582/
https://github.com/AaLl86/MiniKvm_public/tree/main/Specs
CH9329 is a UART to standard USB HID device (keyboard, mouse, custom HID) chip. According to different operating modes, it can be identified as the standard USB keyboard device, USB mouse device or custom HID device on the computer.
CH9329 receives serial port data sent from the client and according to the HID device specification, the data will be packaged and then transmitted to the computer via USB port.
Though the provided computer software, users can also configure the chip operating mode, serial communication mode, serial communication baud rate, various timeout periods, VID, PID, and various USB string descriptors.
The Arduino Leonardo , Raspberry PICO, an DigiSpark? can be programmed as a HID keyboard. This worked on Android and iOS Smartphones and Windows and LINUX PCs
The Arduino Leonardo , Raspberry PICO, an DigiSpark? can be programmed as a HID mouse sending relative X & Y. This worked on Android and iOS Smartphones and Windows and LINUX PCs
Sending ABSOULTE X & Y does not seem to work on Android.
A way to force the the position to a known point would be helpful, so the errors associated with dead reckoning can be reduced.
If the mouse could be told where the cursor was, it may help.
It seems that using a HID device to automate is an "open loop" feedbacksystem, requiring dead reckoning.
A user is using a "closed loop" feedback system, as they can see what is going on, and adapt as required.
The Smartphone is touch, so should I be searching for an external plugin touch emulation using the Leonardo or Raspberry Pico?
https://www.hackster.io/az-delivery/pi-pico-macropad-with-a-touchscreen-459db8
https://source.android.com/docs/core/interaction/input/touch-devices
https://www.usb.org/hid could help
Possibly, See https://www.usb.org/hid HID usage tables
https://usb.org/document-library/hid-usage-tables-17
4.1 Application Usages
UsgageID?: 09 UsageDescription?: TabletPCSystemControls? UsageTypes?: CA Sections 4.1
When I started learning about HTML and WEB pages, I found I could use JavaScript? to dump the DOM object structure, that could be used for testing. This has coloured my understanding of GUI
find decend() on http://www.dougrice.plus.com/hp/gbbook/index.htm
We tried AutoIt? - https://www.autoitscript.com/site/autoit/
It could describe some , but not all the controls on a screen.
A webcam could be used to look at the smartphone. I would need a machine vision system to say if the mouse is above a key to press.
The Arduino would need to be connected to the machine vision system.
All of this is a few orders of magnitude more complex than what I used in the picture above, which worked and I used a lot to soak test the app being tested. It found a number of useful bugs, that were fixed.
It is still worth exploring using the mouse. List the known solutions, and list what is manual and what could be automated.
solutions:-
This diagram tries to think through the way a Human User uses a smartphone. It shows a Machine Vision equipped Robot, and my AutoType.
GraphViz?:- http://www.dougrice.plus.com/images/autoType.dot - use saveAs
The user have to look at the screen and move the cursor over the input box and click on it so that the input caret is displayed. You can use multiple autoType, and realistically the user has to manage setting.
They then need to enable the typing using the switch. The autoType device can only type without feedback from the system under test. Pragmatic but it provides hours of typing.
All of this has been a bit of fun that uses devices brought to support a hobby interest in microcontrollers.
DigiSpark?, Arduino , Raspberry PICO have provides tools that enable soak testing the Smartphone App over lunch, and enabled significant improvements.
A screenshot of an app with the softkeyboard that could be pressed using a robot or AutoMouse? in relative or Absolute mode.
The keyboard is the Gboard English (UK) (PC) on a MOTO G14.