I recently started an improvement issue on github and @gareth kindly suggested to also bring the topic here so that we might have an interesting discussion.
I tried to remove ambiguity from my main post according to @gareth questions, hopefully it’s clearer like this, so with the git issue template:
What feature do you want to improve?
Have a way of synchronize data offline with other devices. E.g. Health worker A going to somewhere rural (being offline) would be able to synchronize data from other offline devices (health worker B, C, D, F etc…). Health worker A then could go to the main health facility, go online and synchronize with the main server.
Describe the improvement you’d like
Offline synchronization of data between two devices. Does not have to be bidirectional flow of data.
Describe alternatives you’ve considered
Download the data locally and the Health worker willing to collect the data would aggregate the data from every device in an USB key. He would then go to the main health facility and synchronize the collected data with the server.
Additional context
There would be an area with the main server (online) and then multiple areas offline where someone (health worker A from the previous example) would go, collect the data, come back, and synchronize with the main server
@magp18 thanks for giving this topic its own dedicated conversation here! This sounds very similar to the concept of “peer-to-peer” syncing that was briefly discussed here. I just wanted to link in that conversation to add some additional context here. Is your desired functionality similar to that “peer-to-peer” sync?
It does not necessarily need to be p2p but it would certainly be an option and solve the issue.
One way could also be having a button to download the data locally to a folder and then use the device’s bluetooth file sharing service to share the data (even if not as elegant as a smooth option with e.g. wifi direct).
Is it possible to download the data from the db whilst offline? I am looking into develop something on top of the cht-android, still looking how to get my hands on the data offline to then try to transport it
This is pretty far outside my range of experience so please consider all of this to be educated guesses + wild speculation.
For indirect syncning of data between two PouchDB instances, I would start by looking at this pouchdb-replication-stream util (it has not been updated in awhile, but it might still work or at least its source code would have some good hints on how to do things). Just remember that it is not enough to just get the doc JSON. For the CHT, you would also want to sync all the attachments too.
The big unknown for me is how hard it will be to connect some custom code to the PouchDB instance running locally on the device. Perhaps it is just a matter of having a custom version of cht-android where you would include your import/export code that connects to the Pouch running in the webview within the app…?
Thanks for your input, ideas are always highly appreciated! I think the link helps actually, I will take a look and let you know if I manage to get some progress. The minimum requirement would be to dump the database, but of course having sync of data automatically would be a lot fancier.
I was first looking into doing this at the level of cht-android, but then I wouldn’t have access to the db right? Unless I could inject JS but even then
I think it might be easier to modify cht-core but probably cooler to have it at cht-android level.
Once again I have not actually tried this, but I would think that injecting JS from cht-android to communicate with Pouch would be at least theoretically possible. Here is an example of where we actually inject some external JS code already.
I was looking into the same thing, so I guess I would have to start at android → inject JS to get the db and dump the json → back to android stream the file to the other device.
Probably something like this right I will look into this and let you know. If someone has some other ideas, I would love to hear them!
This is a fun thread to see being discussed! There’s some interesting problems to solve.
I was reading over the github thread where we talk about securing the data in transit by encrypting it. I realize however that if we use a publicly accessible accessible key, then anyone can send any data in to the CHT and we skip authentication. I haven’t fully thought this through, but I think both encryption and authentication likely need to both be part of the solution.
Yes indeed I agree that both will probably necessary. I haven’t thought this through either, I am still trying to download the data that is being stored locally. I have tried it from the android application as discussed with @jkuester but although we can inject JS, I can’t access the db from the android still, unless I am missing something. I have read about JS interface with android to access android elements from JS but not the other way around.
I might be wrong (not android dev here ) but I feel like it would be easier to create a button in JS access android widgets for a Toast when the download is over.
Not sure how helpful (or relevant) this actually is, but I had to satisfy my curiosity on this…
I tried to see if I could load data from PouchDB into the Android context. In a really simple example I was able to load the settings doc with the following code:
I added this method to MedicAndroidJavascript as a convenient callback for displaying info returned from JS:
@JavascriptInterface
public void toastResult(String result) {
Toast.makeText(parent, result, Toast.LENGTH_LONG).show();
}
Then, to actually get the document, I just had to run:
This is really cool and incredibly useful many thanks! Will try this out to download the data locally first (with a given user first and then trying to do it dynamically) and will keep you updated!
Just a short update:
It works now! Got a button to fetch all the documents on the db, your line PouchDB(‘db…’) alone clarified what I was struggling to find! I am now downloading the data to a folder in the phone (still hardcoded user)
So now I will try to update it to have dynamic user injected & look into how to import it.
Does anyone know how to find out which user is logged in to have access to the db dynamically?
I assume this is stored somewhere even offline right? I just can’t find where. I have been trying to read from cache but haven’t found anything yet, or to have the jsonclient pass the variable as we have url.getUserInfo() but didn’t work either. Also tried to access the SharedPreferences but it appears as if the username is not saved here either. Does anyone have any idea where I can grab the username from to have it in the Pouchdb()?
Sorry for the delayed response here! The user’s name gets stored in the userCtx session cookie. Sadly cookie parsing is a huge pain, but to get the value from within the Java Android context, you could use code like this (note that appUrl is the result of SettingStore.getAppUrl()) :
Awesome, this works like a charm, thanks! I now have a download button and import button with a file chooser. Now working on syncing the read file with the db and the databases should be synchronized!
So, again small update. Right now I am trying to use the upload button, and here we assume internet connectivity. So right now, I am reading from the file and trying to do a POST request to the server to update the data (if I read correctly, if ID is provided, the doc will be updated, otherwise a new one is created). However I am getting a 500 from the server, and according to the documentation, there are 3 possible reasons for that:
If required fields are not found return 500.
If invalid JSON return error response 500.
If submitting JSON and corresponding form is not found on the server you will receive an error.
I didn’t find anything regarding explicit uploading bulk documents but I saw something like /{db}/_bulk_docs/ and seems that it should be usable for POST requests. The three possibilities for error don’t seem to be valid here, since I am reading directly from the stringified dumped JSON file from db.allDocs({include_docs: true}, attachments: true). Does anyone know, if maybe the format is not supported in this endpoint?
Here is how I am doing it (user hard coded because pw retrieval still being checked) :
content = ("docs="+total.toString()).trim();
content = content.replace("\n", "");
Log.d("Content of the file ", content);
// Post downloaded data to the REST API / Main server
Log.d("APP uRL is ", appUrl);
URL url = new URL(appUrl+"/medic/_bulk_docs/");
Log.d("URL using", url.toString());
String userPassword = "medic" + ":" + "password";
String encoding = Base64.encodeToString(userPassword.getBytes(StandardCharsets.UTF_8), Base64.NO_WRAP);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Authorization", "Basic " + encoding);
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("Accept", "application/json");
con.setDoOutput(true);
con.setDoInput(true);
con.connect();
byte[] input = content.getBytes(StandardCharsets.UTF_8);
try(OutputStream os = con.getOutputStream()) {
Log.d("input is currently ", content);
os.write(input, 0, input.length);
//os.flush();
Log.d("input done ", "yes");
}catch (Exception e){
e.printStackTrace();
}
try(BufferedReader br = new BufferedReader(
new InputStreamReader(con.getErrorStream(), StandardCharsets.UTF_8))) {
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
Log.d("response of the call",response.toString());
con.disconnect();
Toast.makeText(getApplicationContext(), content,Toast.LENGTH_LONG).show();
}catch (Exception e){
e.printStackTrace();
}
Have you tried just posting some arbitrary JSON doc(s) (that you know are well-formed) via this code instead of whatever is in content? I was able to use basically the same code in this snippet to post these sample docs to a CHT Couch instance without issue:
I only saw a 500 error when I tried tweaking the JSON data and accidentally ended up with improperly formed JSON. I am specifically worried about the first line of your code snippet "docs="+total.toString()..., I might be missing something, but not sure how this ends up being valid JSON…
Yes that was a mistake, i was trying to send it as a parameter in the query, but not anymore. Thanks, great idea to check with the example file, helped debugging (total btw is the stringbuilder that helped me reading the file that was dumped from the user’s database with PouchDb.allDocs). So it is working now, I am getting
So I guess it is working. But by saving it as an offline person > log out > connect to internet > log in with other user > upload data (giving me the previous snipped response) is not showing me an offline created and assessed user for some reason.
Is it maybe not the /medic/ database I should be pushing to?
Seems like it was storing the whole thing as one document instead of seeing it as multiple individual documents. Realised that dumping with allDocs is having an extra entry with total_rows… This was now deleted and now the format is clearly just { “docs”: [{"id … }]} and Now I get an ok + id + rev per document which was not happening before. Still the files are not visible in the UI
