Hybrid app

[daya]

Is it possible to use elixir-desktop to create hybrid app to access device hardware functions like Camera, Location services etc?

If yes, then how and what kind of javascript framework would you package alongside ?

[dominicletz]

Hey @daya, definitely it's possible. Checkout the android or ios example https://github.com/elixir-desktop/ios-example-app in those you also have platform code in the Bridge.swift / Bridge.kt files there you can access all hardware functions and pass them to your Elixir app.

[daya]

@dominicletz Looking at the Bridge code it seems its based on and limited by wxWidgets capabilities, right? If yes, then its fair to conclude that mobile device specific functions like accessig camera or finger print sensor or location services won't be mapped by Bridge any time soon. Is that correct?

Is there any alternative approach or alternative Bridge implementation in the works ?

[dominicletz]

In our primary app we don't need these at the moment, so yes it's unlikely I will add those soon. But I'm happy to take any and all pull requests in this direction.

[daya]

@dominicletz I will be more than happy to create a PR, but I will need your guidance to familiarize with few concepts

Basically I want to know how should one think about this undertaking to support Camera, Location Services, Bio sensors.

1. Is there a better or alternative to the Bridge approach ? i.e. Ports and NIFs?
   1.1. Will the Bridge approach work for things like Camera or is it only good for URL navigation, launching browser, sending notifications etc ?

2. To start with should I identify the equivalent wxWidget function for Camera etc ?
   2.1. If yes, could you please point me to the some code samples showing this pattern ? as I am having hard time figuring this out end to end. FYI I have looked at Bridge.swift and Bridge.kt and I can see the method name pattern in handle function but how is Elixir side of code using these methods ?
   2.2. If no, then does it mean wxWidget compatibility/mapping is not required ? If yes, then we still need to map to some common set on Elixir side to support both Android and iOS, so what would that common set of functions be?

[dominicletz]

1. For the camera specifically, but maybe for other services as well it might make sense to use the HTML/Javascript API of these from within the Browser instead of using the path from Bridge to Elixir. Something to try (https://stackoverflow.com/questions/6336641/camera-access-through-browser)

2. On the wx side I don't really see any good mappings. So I would go with something new. Probably create a new module file called "BridgeServices.exor such and add the new functions there, so that they can work for both iOS and Android, the functions would of course just forward to the Bridge communication like all other wx* functions at the moment there. On the other side of this in theBridge.ktandBridge.swift` then catch those calls and fetch the corresponding data.

I would for now stay away from Ports and NIFs as those are a pain to compile for all the mobile platforms, but rather try to solve with existing Bridge. For live Video streaming the current bridge might now work though as it assumes JSON encoding (which makes binary data much larger) and it's not suited for large payloads like a video stream. For a video stream I would open a new TCP or UDP port and use the JSON Bridge communication just to agree on the ports used... But I'm getting ahead of myself.

[daya]

1. for other services as well it might make sense to use the HTML/Javascript API of these from within the Browser instead

That sounds reasonable but probably won't support older versions of mobile OS i.e. before Apple/Google started supporting these HTML APIs. Therefore going React Native route may be a better option but then one will need

1. Creating a release that packages elixir-desktop along with React Native code
2. And some kind of bridge between Elixir and React Native code

JSON encoding (which makes binary data much larger) and it's not suited for large payloads like a video stream

The new bridge doesn't have to use JSON it could simply use other more efficient protocols e.g. RTP, RTMP, WebRTC or even RTSP, but since the distance between the client & server will be just few silicons then JSON may not be a problem after all.

[oliver-kriska]

since the distance between the client & server will be just few silicons then JSON may not be a problem after all.

btw: It doesn't matter on distance in this case, it can be issue with encoding/decoding time on same device. I assume that it uses http protocol under the hood, so there is multiple encodings/decodings for example:
some server type -> json -> http "string" -> json -> some client type

Using some other protocol can save you some en/decodings and it can save you time and batter when we are talking about mobile devices. But I think this problem doesn't have to be solved now :)

btw good job with this project, thanks :)

[dominicletz]

Yes, encoding exactly. For Video content especially if it's a ton of it we should definitely skip any json encoding, and instead just push the raw bytes to save CPU cycles and battery on the phones.

The comment about react-native I don't understand but then I also don't know anything about react-native. I personally definitely stay away from as much javascript development as possible, so no plans from my side to include an additional js framework. That said if there is a pull request that allows us more access to hardware from the Elixir apps and happens to use react-native I wouldn't block it either.

Cheers!

[oliver-kriska]

Well, streaming video is a little bit complicated and different and I think you should check some solution for Elixir and Phoenix itself outside of this elixir-desktop "solution". For example there some older posts about it

https://littlelines.com/blog/2020/07/06/building-a-video-chat-app-in-phoenix-liveview
https://dockyard.com/blog/2020/09/25/live-streaming-with-liveview-and-mux-in-under-70-lines-of-code

[daya]

@oliver-kriska

I think you should check some solution for Elixir and Phoenix itself outside of this elixir-desktop "solution".

But how will that help with a mobile application that allows user to upload videos ? If a web app is rendered in a WebView then one is left to the mercy of OS to support HTML5 tags or hope JS hackery works.

It seems the only reliable way is to use device OS native API to access camera.

[oliver-kriska]

That's true but still you have two battle fronts:

1. OS native API for camera access
2. accept data in Elixir from native API