Fingerspelling.xyz app helps people to learn the sign language alphabet

Creative studio Hello Monday has created an app that teaches and corrects hand positions in real-time to make it easier for people to learn the sign language alphabet.

Hello Monday worked with the American Society for Deaf Children to design Fingerspelling.xyz, an app that runs in web browsers.

Fingerspelling.xyz is an app that runs in web browsers

Not unlike the popular language-learning app Duolingo, Fingerspelling.xyz is able to assess its user's attempts to learn words in real-time.

The app makes use of a person's webcam to track their hand movements as they attempt the hand positions of the American Sign Language (ASL) alphabet, while an algorithm trained on images of correct hand positions analyses their accuracy.

The app gives instruction in the sign language alphabet tailored to the user

"The fingerspelling game is a great way to introduce the basics of ASL in a fun and playful way," said Hello Monday founding partner Anders Jessen.

According to Jessen, the technology used by the app offers a more engaging learning experience than the traditional methods of learning the sign language alphabet – through reading or watching videos.

The app targets parents of deaf children, who are unlikely to know sign language

"The game leverages advanced hand recognition technology, matched with machine learning, to give you real-time feedback via the webcam for each sign and word you spell correctly," Jessen said.

"It's fun to think about how this highly trained machine-learning model, which researchers have spent countless hours on training, now does the opposite – it now trains us back, and makes us better at fingerspelling."

To start, users are shown a word they will be learning to spell letter by letter

In ASL, fingerspelling is used for proper nouns, or when a person doesn't know the sign for a word.

On a purple and butter-yellow interface, Fingerspelling.xyz starts by giving the user a series of words, along with a 3D model of a hand showing the position required for each letter.

The app then analyses the user's attempt, gives feedback until they achieve the correct alignment, and then lets them progress through the levels.

Hello Monday promises that webcam data isn't stored or sent anywhere.

Fingerspelling.xyz makes use of the computer's webcam to track the user's gestures

The studio is especially targeting the app at parents of deaf children, so that they can encourage their child's development by introducing them to sign language at an early stage.

It points to statistics that show that two to three of every 1,000 children are born deaf or hard of hearing in the US, and 90 per cent of them are born to hearing parents.

A 3D model of a hand shows the correct shape, which users try to emulate

However, 72 per cent of families do not sign with their deaf children, and Hello Monday and the American Society for Deaf Children hope to change that.

"We are hoping that this can become useful for anyone wanting to learn Fingerspelling – and the next steps after this would be to take a course on Fingerspelling/ASL," said Jessen.

An algorithm analyses the user's hand shape and guides them to correct it

Hello Monday is a creative studio focused on digital products and experiences. Its recent work has included designing a virtual Summer Show exhibition for the Bartlett School of Architecture in 2020.

In 2011, a deaf couple had their house in Tokyo designed to enable them to sign to each other and their children through the walls. It is covered with nearly a hundred small windows.

The post Fingerspelling.xyz app helps people to learn the sign language alphabet appeared first on Dezeen.

from Dezeen https://ift.tt/3gC0P08