While real-time browser-based communications (like adding click-to-call to an instant messaging app) has become a solid use case for the likes of WebRTC, some developers are working on more advanced fare, like augmented reality (AR) hooks.
That’s because the use of JavaScript APIs like WebGL and WebRTC enables unexpected and unusual interactions between the browser and the user.
“For example, WebGL allows for hardware accelerated graphics and, with the help of the typed array specification, enables the JavaScript engine to execute number crunching with almost native performance,” said Alberto Mancini, a C Developer, in a post. “Similarly, with WebRTC, the browser is able to access video (and other data) streams directly from the computer hardware.”
Mancini and his team were able to use WebRTC and WebGL to harness a large Java library, NyARToolkit, which never intended to be used in the browser, to help them create Picshare, a marker-based AR application.
He explained that this type of AR application searches the scene for a specific, easily recognized geometric pattern, which provides information about an object’s position and orientation. That in turn allows software to project additional 3D scenery into the image in a realistic way.
To do that, the team used WebRTC to gain access to the camera from a browser: “Browsers…originally were not intended to allow websites to interact with most device hardware,” Mancini explained. “WebRTC breaks through this barrier using HTML5s media capture features, enabling the browser to access, among other things, the device camera and its stream.”
From there, the team analyzed each frame of the video stream to detect markers, and compute the marker’s position in the reconstructed 3D world, using NyARToolkit. To augment the video, the developers then used WebGL to draw the final, augmented scene onto the Web page. They also used a programming tool called GWT to link it all together, and a few other underlying technologies.
“Picshare is a long-term pet project for us,” Mancini said. “The first implementation dates back a few years, and even then it was fast enough to be impressive. At this link you can see one of our earlier experiments, compiled in 2012 and never touched….with the release of GWT 2.8 we have gained a much-improved interoperability layer with JsInterop, further boosting performance. Also, to the celebration of many, we also gained a much better development and debugging environment, Super Dev Mode. Oh yeah, and Java 8 support.”
Edited by
Kyle Piscioniere
