Testing a smart phone apps is a challenge. They are used by users who look at the screen and use a finger or two to interact.
You can plug a mouse into a smartphone, and use it to control the app as well. Move the mouse and click as well as moving you finger and touching the screen.
It was easy to inject key strokes, this page explores using the same hardware to move the mouse and click. I have yet to achieve a working solution.
A smartphone app was being used by the user for this service:
I could use Arduino Leonardo to inject typing , and co-ordinate which Arduino was typing using a simple wire. The switch also allows the typing to be turn on and off as required.
I set the call up, and in the picture above, used my finger to select the input boxes so the input CARET is in the right place.
Once manually set up, I could turn on an automate a stream of typing and they would type in turn. If it kept working, all was well. I could go to lunch and hopefully it was still working when I got back!
It is also possible to program the Arduino Leonardo as a mouse. This is much harder than the keyboard, as the BIG UNKNOWN is how far to move the mose to the next click.
So this page is about exploring using the Arduino Leonardo as a mouse.
You can plug a mouse into a smartphone, and use it to control the app.
Move the mouse and click, as well as touching the screen with the finger.
The big unknown is how far to move the mouse to the next click.
I plugged a mouse using an OTG cable, and you get a cursor and you can click.
I could make the cursor bigs so that it was easier to see.
It is conceiveable that a camera could be used to look at the screen and see the position of the cursor.
https://www.testdevlab.com/blog/how-we-built-a-robot-for-automated-manual-mobile-testing
https://www.iphoneness.com/smart-robots/mobot-smartphone-tester-robot/
Knowing that using robots was not a realistic option in the short term,
The App needed a slow stream of repeatable text to soak test it and fill up scroll buffers, over many minutes.
When I tries to get two devices to type together I wrote a simulation on a webpage:
Simplex, Duplex Full Duplex and Half Duplex are terms from telegraphy that are relevant.
Instead of injecting keystrokes as a keyboard, use the Arduino as a mouse to type on the soft keyboard.
https://docs.arduino.cc/built-in-examples/usb/KeyboardAndMouseControl/
For the keyboard version, the built in library converts text to key presses.
My sketch would need to map the letters onto positions on the soft keyboard and move the mouse.
Instead of using a delta-x robot and dabber , use the mouse and click.
Use a camera to see where the mouse is on the smartphone screen.
A webcam could be used to look at the smartphone. I would need a machine vision system to say is the mouse is above a key to press.
The Arduino would need to be connected to the machine vision system.
All of this is a few orders of magnitude more complex than what I used in the picture above, which worked and I used a lot to soak test the app being tested. It found a number of useful bugs, that were fixed.
It is still worth exploring using the mouse. List the known solutions, and list what is manual and what could be automated.
solutions:-
All of this has been a bit of fun that uses devices brought to support a hobby interest in microcontrollers.