May an extension read the offline map store, and is there a supported way to get the road network ahead?
I built a coaching extension that computes per-segment power and speed targets from rider physics. Using karoo-ext 1.1.9, OnNavigationState gives me everything I need while a route is loaded: polyline, routeElevationPolyline and climbs all work well on a Karoo 2.
Many riders using it don't load a route. For them I'd like the road network ahead of the rider so I can look ahead to the next junction and estimate gradient. Two questions:
-
The offline maps live on shared storage at /storage/self/primary/offline/maps/*.map in standard Mapsforge format, readable with the normal storage permission. Is an extension permitted to read those files? Is that location something you'd consider stable across firmware updates, or should I treat it as private and not rely on it?
-
Is there a supported or planned karoo-ext API for the roads or elevation near the rider without navigation running, similar to what NavigatingRoute provides once a route is loaded? Even an elevation-along-heading lookup would cover my case.
If reading the map store is not something you want extensions doing, I'd rather know now and build my own road and elevation data instead. Thanks.
-
I think the Mapsforge data does not contain elevation if I remember correctly, as it's not required to draw the map (The recently added contour lines are precalculated polylines stored elsewhere I think). It does contain other useful data like surface types though.
It would be amazing if HH added access to the predicted route and its elevation to extensions as you suggest. Or even better, low level access to the routine engine and its data. They have not added new features to the SDK for more than a year or so though, and seem to be happy with what is there currently. They generally do not engage with third party developers here or on the defunct github issue tracker.
-
Thanks, that matches what I found. I opened the Karoo's ontario.map with mapsforge-map-reader: it has highway type, name, surface, tracktype, cycleway and oneway, but no elevation, as you say.
Two things others may find useful. First, the routing tiles beside the map (
.gph.gz, Valhalla format) do carry elevation per edge. I compared their per-edge mean elevation with my own surveyed profile along a 45 km ride and it agrees within a few metres, though a long rural edge gives only one value, so the shape inside the edge is lost.Second, karoo-ext 1.1.9 already gives an extension the loaded route's elevation: subscribe to OnNavigationState and the NavigatingRoute carries routeElevationPolyline, a 30 m profile of the whole route, plus the climbs list. On my Karoo 2 that worked, with the caveat that the state is delivered on subscribe and not pushed when navigation starts later, which I filed as karoo-ext issue #81. So the gap is only free riding without a loaded route.
Has anyone here parsed the Valhalla tiles on-device, or heard whether reading the offline map store is tolerated by Hammerhead? I would rather build on something supported than on files that may change.
-
Hi David, some progress, but not from the Karoo's own map data. The elevation in it turned out too coarse for what I need, and Hammerhead hasn't replied about access. So I went the other way and built my own road and elevation data. Elevation ahead without a loaded route is now running on a few test Karoos and looks promising so far.
What are you hoping to use it for?
-
Are you sure the karoo's own elevation data is too coarse? The karoo climber feature works fully offline without a loaded route so I thought the data has to be there. I did some tinkering with the routing data, and the data is reasonably easy to access with the nice valhalla-mobile project, which provides a jni binding and statically linked android library build of valhalla; I only patched in the valhalla locate api endpoint and some landmark handling where it crashed out before (not sure why). Map matching the route with trace_attributes allows to sample the elevation in a fixed interval, and the data looks good to me (I'd think this is the exact same data karoo-ext supplies in the elevation polyline when a route is loaded or you are navigation to a pindrop).
I prototyped elevation lookahead without a loaded route for my routegraph extension, but am not happy with the path prediction algorithm so far. I use the trace_attributes valhalla api with recent gps measurements to get the current way_id, and then use the locate api to look up the current edge's path and from then on, the next edge at each upcoming intersection, assuming the user continues to ride on the same road or does the least required turning if the current road ends.
It is not stable enough yet though; it snaps to adjacent roads too frequently and sometimes does not pick the correct edge at intersections.
The routing data also contains other useful stuff (surface conditions, tunnels, ferries, bridges) that are nice to have and show.
-
I come from wahoo and used the elevation profile ahead all the time. Imo it was super handy to know for descends or rolling terrain, etc, when I did not have a loaded route. In Wahoo you had 2km or so ahead predicted. More than enough to give you an idea of what was coming.
Please count me in for any testing. I really miss this feature so much. I find it odd that not more people are asking Hammerhead for it. Specially because it shouldn't be a big deal for them to implement it or let others do it.
-
@timklge thanks, that’s helpful, and you’re right that the data is there. Reading the tiles with Valhalla, we get per-edge elevation and grade, which is clearly what Climber uses.
“Too coarse” was about our use, not climb detection. Against 1 m LiDAR ground data the per-edge heights were typically a couple of metres off. That’s fine for spotting a climb ahead, but not for metre-level grade on short stretches.
One question: on the desktop, sampling at a fixed interval (elevation_interval) gave us no usable per-point heights, only the per-edge values. Which Valhalla version do you use in valhalla-mobile, and do you get real per-sample heights?
@Hammerhead, the original question still stands: is reading /sdcard/offline/nav from an extension permitted, and will the tile format and location stay stable across firmware updates?
-
@Andy Basacchi The graph edges encode deltas to the node elevation at the start point. HH have their own valhalla fork, and the node elevation seems to be shifted by a few bits compared to vanilla valhalla. I assume the elevation data baked into the tiles by HH is the SRTM data, so the elevation data has a 30 meter resolution with claimed accuracy of 6 meters. I am very happy with that, but it sounds like it does not meet your requirements. I'm not sure there is any publicly available dataset with higher precision though.
I have forked valhalla-mobile (and valhalla) with the changes applied so that it can read the karoo tiles with correct elevation: https://github.com/timklge/valhalla-mobile . The native library built by the CI in release mode and stripped of debug symbols is just 2.5 MB per arch, so would be fine to be shipped with an extension I think (The HH build on the karoo is 120 MB as it includes the symbols. I guess they want to be able to debug it when it inevitably crashes on user devices). HH seems to have implemented some of the karoo features in their fork (karoo climb detection, route lookahead etc). I found it interesting that they did that in native code instead of trying to have as much as possible on the jvm side.
I built a proof of concept with a datafield that displays an elevation graph of the predicted upcoming route (It just predicts you stay on the current road or ride as straight ahead as possible when it ends, shown as the orange polyline). Works better than I expected. Could be made smarter by looking at your past rides or even trying to read the raster heatmap tiles.
But I agree with you that it is not a very cool solution, because the nav tiles are not supposed to be a public interface. I think they can not reasonably make breaking changes to the format though, without also breaking it for users that do not install the newest system updates, but map updates. Downloading the full navigation tileset with hundreds of megabytes again in vanilla valhalla format for the extension also seems not that great though, at least not for this use case. Talking to the internal hammerhead navigation service (like the cool ant light control extension does with the sensor service) is also not a good option I think, as it relies much more on internals.
So I'm not sure I will continue iterating on this and try to build it into my routegraph extension. It would be an order of magnitude simpler for Hammerhead to just display the upcoming elevation profile in the native datafield when no route is loaded, as they already have it to do the climb detection on it anyway.
But it was fun tinkering on this.
You will not get a response from HH here I think.
-
@timklge thanks, that’s really useful, and the proof of concept looks great. The predicted-road elevation datafield is very close to what I’m after.
It also makes the case that Hammerhead’s own maps are the right foundation. The road network is already on every Karoo, it matches what the rider sees on screen, and the navigation and climb detection use it. An extension shouldn’t need its own copy of the roads or a second download of the tiles. The only weak spot is the elevation baked into them.
If that’s SRTM, it’s 30 m cells with errors of several metres, which is fine for a long climb profile but not for short, steep pitches. A 100 m ramp can disappear or move, and bridges and underpasses get the height of the ground below or above them. In North America much better data is free: USGS 3DEP (1 m LiDAR across most of the US) and Canada’s HRDEM (1–2 m). I’ve been building road profiles from those, and the gradients come out right where SRTM is off.
@Hammerhead, two questions:
Which elevation source goes into the Karoo’s navigation tiles, and do you plan to move to higher-resolution data where it’s available (3DEP, HRDEM, the European national LiDAR sets)?
Could extensions get the elevation ahead through a supported interface instead of reading /sdcard/offline/nav? For example, the upcoming profile on the current road when no route is loaded, which climb detection already works out.
@timklge one question for you: does Hammerhead’s fork keep the OSM way IDs in the edge info, as vanilla Valhalla does? If it does, an extension could use Hammerhead’s roads for “where am I going” and attach more accurate heights by way ID, until Hammerhead improves its own.
Please sign in to leave a comment.
Comments
13 comments