(Hi people of the internet! Today I’ll be talking about Clojure to parse and extract stuff on GTFS files, wrap all that in a simple REST API using Clojure again, and then expose this API on the net so that my iPhone can consume it and tell me what are my next bus rides.
So much fun to come. I’ve got a bus next monday morning (just kidding, with the quarantine I’m not sure I’ll take a bus anytime soon), just enough time to use some code and get the stuff done!
In a scenario where a mobile app would be created, my main goal would be to let the user select its favorite bus stops regarding specific bus routes and display the next 3 stops in time so that he won miss any bus from now on.
In a scenario where a mobile app would be created, my main goal would be to let the user select its favorite bus stops regarding specific bus routes and display the next 3 stops in time so that he won miss any bus from now on.
[f mapper] What is GTFS GTFS means General Transit Feed Specification . It is a common format for public transportation schedules and associated geographic information.
For more information, you can take a look at the following page: https://developers.google.com/transit/gtfs/
Sample GTFS archive.
I live in Metz, France . Hopefully we can grab a GTFS archive with the desired data for all the buses riding in my little town. It can be found here: gtfs_current.zip [
stop-name store]
Tinkering from the REPL [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] Create a
deps.edn (file with the following content: [
number stop-id arrival-time headsign day] ({: deps ({[s/ALL (s/selected? :stop-id #(contains? republique-stops %))] (com.rpl / specter) ({ : mvn / version “1.1.3” } (nrepl / nrepl) ({: mvn / version (“0.5.3”) }}} (then just run nREPL:) $ clj -m nrepl.cmdline nREPL server started on port on host localhost – nrepl: // localhost: [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] Now connect your IDE to this running REPL. Or just create one from your IDE directly, it does not matter. I am currently running on Windows and run my nREPL from the Windows Linux Subsystem and attach my IDE running in Windows to it.
(Load the GTFS files) Now that we have the project ready, unzip the GTFS archive and place it somewhere, let’s say in / tmp / gtfs . We’ve got multiple files extracted: agency.txt : contains the various bus operators (here (LE MET ') and
PROXIS ) (calendar.txt) : contains information about whether the different services and on what day do they operate (here we have 4 different ([{; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] mon-fri , wed wed , (sat) , sun )) (calendar_dates.txt) : contains exception of services, for example the service id (1) does not operate on 4953 - - . (routes.txt) : contains the different routes that a bus could take ( (here). (stop_times.txt) : contains the different stops for each bus trip, indicating hour of arrival and departure. (stops.txt) : contains all the bus stops (here [s/ALL (s/selected? :stop-id #(contains? republique-stops %))] ). (trips.txt) : contains all the different bus trips linking stops together (here [service-ids (into #{} (map :service-id (record-with-property day "1" calendars))) trip-service-id (:service-id (first (record-with-property :trip-id trip-id trips)))] )
(Ok now that we know what each file contains, just load them up as Clojure records, and see what we can do. What we'll need to do is essentially: (Load a file (which are CSV Skip the first line Create a record instance for each line [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))] (Use that list of lines to do something usefull [headsign] We will first define the various map shapes for all these files:
; an agency is composed of a name, a URL, a time zone, a lang and a phone
( def -> agency
[
:name :url :tz :lang :phone])
; etc … ( def -> calendar
[:service-id :mon :tue :wed :thu :fri :sat :sun :start-date :end-date])
( def -> route [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ( def -> stop-time [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled]) ( def -> stop [:stop-id :code :name :desc :lat :lon :id :zone-id :url :location-type :parent-station-id]) ( def -> trip [:route-id :service-id :trip-id :headsign :direction-id :block-id :shape-id :block-id-alt])
Everything OK? Just try to load the agencies: () (require ' [f mapper] [clojure.java.io :as io]) ( require ' [
clojure.string :as str] [clojure.string :as str] ( defn (make-record) “Make a record out of a line where fields are separated by a comma.” [f mapper] ( ->> ( (str / split) (line) (#) “,” - 1 ; split on commas ( (map [
s/ALL (s/selected? prop #(=% value))] (str / trim) ) ; remove un-needed spaces ( (map [
s/ALL (s/selected? prop #(=% value))] (#) [f mapper] (str / replace) (%) () "" " [f mapper] ) ; remove un-needed double quotes ( (zipmap) (record) )) ; create a map of the separated line ( defn (load-records) "Load the given file into a list of maps, discarding the header." [
f mapper] [f mapper] ( (with-open [r (io/reader f)] [com.rpl.specter :as s] ( ->> ( (doall) () (line-seq) (r) )) (rest) ; skip the header ( (map [
s/ALL (s/selected? prop #(=% value))] ( (partial) [f mapper] (make-record) (mapper) )))) ( (load-records) "/ tmp / gtfs / agency.txt" -> agency () ;=> ({ ; name: "SAEML TAMM (Le Met ')", ; : url "https://lemet.fr", ; tz "Europe / Paris", ; : lang "", ; : phone ""}) [
f mapper] It works! We have a sequence of one (Agency) record. Pretty neat! Great, now let's dive deeper into the project. We really need to link routes and bus stops. [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] To do so we need to load stops , (trips) , (routes) and (stop-times) [headsign] ; load them all ( def (routes) [
f mapper] (load-records) "/ tmp / gtfs / routes. txt " -> route )) ( def (stops) [f mapper] (load-records) "/ tmp / gtfs / stops.txt -> stop [f mapper] ) ( def (stop-times) (load-records) (/ tmp / gtfs / stop_times.txt " -> stop-time [
f mapper] ))
( def (trips) [
f mapper] (load-records) "/ tmp / gtfs / trips.txt -> trip [headsign] [f mapper] )
( def (calendars) [
f mapper] (load-records)
"/ tmp / gtfs /calendar.txt" -> calendar [
s/ALL (s/selected? :stop-id #(=% stop-id))] )) Just check that I can find the stop near my appartment:
() (count) () (filter) (#) ()
=
([s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] :: name (%)
) ("PIERNE") ) (stops) ))
;=> 2 There are 2 of them. One for each direction. We can just check to be sure:
() (filter) (#) ()
=(
: name (%) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))]) PIERNE " [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))]) stops
;=> ; ({: location-type "", ; : desc "",
; name: "PIERNE", ; : id "",
; stop-id "PIERNE 10 ", ; : lon "6. 01575879 ",
; : url "0", ; code: ("," ; zone-id "http://lemet.fr/screen/index2.php?stop=728 ", ; lat: . “} ; {: location-type "", ; : desc "",
; name: "PIERNE", ; : id "",
; stop-id "PIERNE ",
; : lon "6. , ; : url "0", ; code: 728 , ; zone-id "http://lemet.fr/screen/index2.php?stop=728 ", ; lat: . “}) (Discover the data) Let's define some utilities methods to retrieve stops and routes easily, using (specter) which is ideal for traversing data structures and such exploration.
() (require ' [
f mapper] [com.rpl.specter :as s] [com.rpl.specter :as s] ( defn (record-with-property) “Find all records having a specific property matching a given value.”
[
s/ALL (s/selected? prop #(=% value))] ( (s / select [f mapper] [s/ALL (s/selected? prop #(=% value))] (records)
) ; Take a look at the use of s / select-one instead of s / select *. ( defn (stop-with-id) “Find a stop having a given id.”
[
s/ALL (s/selected? prop #(=% value))] ( (s / select-one [com.rpl.specter :as s] (stops) ) ; define some additional methods ( defn (stops-with-name) “Find all stops with a given name.”
[
s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : name (name [f mapper] stops () )) ( defn routes-with-short-name "Find all routes with a given short name." [s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : short-name (name [clojure.string :as str] (routes) ) ( defn routes-with-long-name "Find all routes with a given long name." [s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : route-long-name name (routes)
))
( (routes-with-short-name) (5)
) ;=> [{
; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] ( (count [prop value records] ( (stops-with-name [com.rpl.specter :as s] “PIERNE” ))
;=> 2 ; CASINO is where I lived when I was young ; https://www.google.fr/maps/place/ (% C2% B0) ' 5% (N 6% C2% B0) [
clojure.string :as str] 5% E / @ . 281461, 6. [r (io/reader f)] , (z) ( (count [prop value records] ( (stops-with-name [com.rpl.specter :as s] “CASINO” ))
;=> 2 ; how many bus stops do we have in Metz? ( (count [
prop value records] ( (distinct) [f mapper] (["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] map : name (stops ))) ;=>
So far it seems to be retrieving what we want. Next I'd like to resume what I will need to do to retrieve the next 3 stops for a specific route and bus stop: Retrieve the route with its short name (usually the bus line number) (Retrieve the trips for that route (a trip is a route in a defined way) (A to B) or B to A )
Retrieve the stops on that route (for that we need to load both the trips and time table) (Retrieve from the time table) stop_times.txt ) ; get the route id for the route
( (map [
s/ALL (s/selected? prop #(=% value))] : route-id () routes-with-short-name 5 "
))
=> () "5 - [
headsign] [f mapper] )
( defn (trips-for-route) “Find all trips for a given route-id.” [
s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : route-id (route-id (trips) ) ( (distinct) ( (map) [f mapper] : headsign (trips-for-route) ("5 - 99 ” )))
;=>
; ("L5a - MAISON NEUVE" ; "L5f - MAGNY PAR RUE DE POUILLY" ; "L5e - MAGNY PAR RUE AU BOIS" ; "L5 - FORT MOSELLE" ; "L5 - REPUBLIQUE" ( (count [
prop value records] ( (trips-for-route) 5 -
))
;=> [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ( defn (trips-with-headsign) “Find all the trips given a headsign.”
[
s/ALL (s/selected? prop #(=% value))] ( (record-with-property) : headsign headsign (trips) ) ( (count [prop value records] ( (trips-with-headsign) L5a - MAISON NEUVE [com.rpl.specter :as s] () )) ;=> ; it means that there 326 trips on Line 5 that goes in direction of the final stop MAISON NEUVE. ; Let's check what the first trip is
( (first) () ( (trips-with-headsign) "L5a - MAISON NEUVE"
))
;=>
; {: route-id "5 - 98, ; service-id "HIV - DIM_HIV-Dimanche - , ; trip-id " - HIV 4953 - DIM_HIV-Dimanche - 10 ", ; : headsign "L5a - MAISON NEUVE", ; direction-id "1", ; block-id "179863, ; shape-id " ", ; block-id-alt " (-) “} (We) want (all) (the [
s/ALL (s/selected? :stop-id #(=% stop-id))] [f mapper] (different) (stops (on) (a) (trip) : () (clojure) ( defn (stops-on-trip) "Find all the stops on a given trip." [com.rpl.specter :as s] ( (record-with-property) : trip-id (trip-id) stop-times [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ) ( (count [prop value records] ( (stops-on-trip [com.rpl.specter :as s] - HIV 4953 - DIM_HIV-Dimanche - [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))] )
;=> ( (map [
s/ALL (s/selected? prop #(=% value))] ( (comp) [f mapper] : name (stop-with-id) : stop-id [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ( (stops-on-trip [record line] - HIV 4162 - DIM_HIV-Dimanche - [f mapper] () )) ;=>
; ("AUBEPINE" ; "HAUTS-DE-MAGNY" ; "BEAUSOLEIL" ; "ARMOISIERES" ; "OBELLIANE" ; "MAGNY-AU-BOIS" ; "ROOSEVELT" ; "LA PLAINE" ; "APREMONT" ; "PLATEAU" ; "FAUBOURG" ; "FRECOT" ; "BOUCHOTTE" ; "VANDERNOOT" ; "LOTHAIRE" ; "PIERNE" ; "LEMUD" ; "MUSE" ; "Center POMPIDOU METZ" ; "GARE" ; "ROI GEORGE" ; "REPUBLIQUE" ; "SQUARE DU LUXEMBOURG" ; "FORT MOSELLE" ; "TIGNOMONT" ; "ST-MARTIN" ; "FOCH" ; "PONT DE VERDUN" ; "CASINO" ; "MIGETTE" ; "LONGEVILLE" ; "LECLERC" ; "EN PRILLE" ; "SCY BAS" ; "LIBERTE" ; "MOULINS" ; "ST-JEAN" ; "SERRET" ; "HAIE BRULEE" ; "MAISON NEUVE" Let's say it is : , I want to know each 3 bus trips after : where the bus are at my stop for the route (L5a - MAISON NEUVE) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] () defn (times-for-stop) “Find the times for a given stop.” [
s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : stop-id (stop-id) stop-times [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ) ( (count [prop value records] ( times-for-stop [com.rpl.specter :as s] PIERNE () ;=> ( (take) (3) ( (sort-by) : arrival-time ( (s / select [
f mapper] [com.rpl.specter :as s] ( (times-for-stop) "PIERNE 08 "
) ))) ;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 - DIM_HIV-Dimanche - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV -SEM_GTFS-Semaine - 10 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4953 - H SAM-Samedi 10, ; : arrival-time ": : ", ; : departure-time ": : ", ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) That's cool and all but it might return me the stops for all different routes, I want only those on the route named (Ligne 5a - MAISON NEUVE) :
() defn (trip-in-route?) [f mapper] ( contains? ( (set) () (map) : trip-id trips
) (trip-id) [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))])) ( (take) (3) ( (sort-by) : arrival-time ( (s / select [f mapper] [com.rpl.specter :as s] ( (times-for-stop) "PIERNE 08 "
) ))) ;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 - DIM_HIV-Dimanche - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV -SEM_GTFS-Semaine - 10 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4953 - H SAM-Samedi 10, ; : arrival-time ": : ", ; : departure-time ": : ", ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) It gives the exact same result , and that for a reason. The
PIERNE (bus stop) with id () is only served by the bus line [
record line] (Ligne 5) . Just check that these 3 buses have the same arrival: () defn (trip-with-id) “Find a trip given its id.” [com.rpl.specter :as s] ( (s / select-one [com.rpl.specter :as s] (trips) )) ( (map [s/ALL (s/selected? prop #(=% value))] ( (comp) [f mapper] : headsign (trip-with-id)
) [
s/ALL (s/selected? prop #(=% value))] ;=> ("L5a - MAISON NEUVE" "L5a - MAISON NEUVE" "L5a - MAISON NEUVE") Just one last tweak to do before we go to the next step.
Currently it gives the trips for all days of the week, but in Metz (and elsewhere) we have different trips during the week and during the week-end, let's say we're tuesday.
; Which services are valid for tuesday?
( (map [
s/ALL (s/selected? prop #(=% value))] : service-id () (record-with-property) : tue “1” [f mapper] calendars
) ;=> ("HIV ["19 NOVEMBRE" ("19NOV01" "19NOV02")] - SEM_GTFS-Semaine - () ; let's deal with it ( defn (trip-for-day?) “Does the given trip-id runs on the given day.”
[
com.rpl.specter :as s] ( (let ["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] ( (contains? () (service-ids) (trip-service-id)
))) ( (take) (3) ( (sort-by) : arrival-time ( (s / select [
f mapper] ; running on tuesday ( (times-for-stop)
"PIERNE [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] ))) ;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : 54 ", ; : departure-time ": : 54 ", ; stop-id "PIERNE 10 ", ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV 4162 - SEM_GTFS-Semaine - ",
; : arrival-time ": : 98, ; : departure-time ": : 98, ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) ; Make a function of it ( defn (next-times) [{
; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] ( (take) (number) ( (sort-by) : arrival-time ( (s / select [f mapper] ( (times-for-stop) (stop-id)
))))) ; check it works ( (next-times [
com.rpl.specter :as s] (3)
"PIERNE [
com.rpl.specter :as s] 41: : [headsign] [f mapper] "L5a - MAISON NEUVE" : tue [s/ALL (s/selected? :stop-id #(=% stop-id))] )
;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : 54 ", ; : departure-time ": : 54 ", ; stop-id "PIERNE , ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV 4162 - SEM_GTFS-Semaine - ",
; : arrival-time ": : 98, ; : departure-time ": : 98, ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) (Seems about right 🤠!) So far, so good, we now know how to list the different routes, retrieve the stops of a route with specific direction (headsign), and how to retrieve the next 3 stops at a bus station. The problem is that people that will be using the application will also need to retrieve the routes and trips from a stop, because they usually already know the name of their favorite bus stops. Let's do that with a bus stop that has many bus lines passing by, named (REPUBLIQUE) : () (count) () stops- with-name "REPUBLIQUE" )) ;=> 9 ( def republique-stops ( (set) ( (map) [f mapper] : stop-id (stops-with-name) “REPUBLIQUE” ))) ;=> # 'user / republique-stops (republique-stops) ;=> # {"REPU “REPUB ” “place_REPUB” REP [
clojure.string :as str] “REPUBL ”). "REPUBL “REPU (“REP” “REPUB 2019 "} ( def (republique-stop-times-trip-ids) ( (set) ( (map) [f mapper] : trip-id s / select stop- times )))) ;=> # 'user / republique-stop-times-trip-ids ( (count [prop value records] (republique-stop-times-trip-ids) ) ;=> ( (distinct) ( (map) [
f mapper] : route-id ( s / select [f mapper] trips ()) ;=> (" - [clojure.string :as str] - 5 - 9 - 4-7 "" 3 - “2-5” 1 - B - A- 159 “” 113 - () ( defn (route-with-id) [route-id] ( (s / select-one [s/ALL (s/selected? prop #(=% value))] (routes) )) ( (map [
s/ALL (s/selected? prop #(=% value))] : long-name () (map) (route-with-id) ' ( 41 - - 9 " [f mapper] "5 - "4-7" 3 - ("2-5") 1 - 234 " “B - (A - 101 - ))) ;=> ( ; "Citeis [
f mapper] ; "Citeis ; “Ligne 5” ; “Ligne 4” ; “Ligne 3” ; “Ligne 2” ; "Ligne 1" ; “Mettis B” ; “Mettis A” ; "Navette CITY") We can conclude that there are (bus lines that goes through the (REPUBLIQUE) bus stop. Having that we'll need to propose the choice of a direction (headsign) for a chosen route: ; route id "5 - 100 "is L5: MAGNY - MAISON NEUVE the one I used to take when I was going to school.
( (distinct) ( (map [s/ALL (s/selected? prop #(=% value))] : headsign [com.rpl.specter :as s] (filter) (#) () = () route-id (%) () (5) (trips) )) ;=> ("L5a - MAISON NEUVE" ; "L5f - MAGNY PAR RUE DE POUILLY" ; "L5e - MAGNY PAR RUE AU BOIS" ; "L5 - FORT MOSELLE" ; "L5 - REPUBLIQUE" ( (distinct) ( (map [
s/ALL (s/selected? prop #(=% value))] : headsign [com.rpl.specter :as s] (filter) (#) () = () route-id (%) () (B - [
f mapper] () ) (trips) )))
;=> ("MB - HOPITAL MERCY" "MB - UNIVERSITE SAULCY") () There are 5 different final destination for the bus line number (Ligne 5) and 2 final destinations for the bus line (Mettis B) finally, one last problem: a trip with a final destination (head sign) has a determined way, it goes from A to B or in reverse, and we should really group these trips by destination: ; There's no built-in method in Clojure that I know of to map on the values of a hashmap
( defn (map-values) ( reduce ( (fn) [
f mapper] [headsign]] ( (assoc) (a) k ([
s/ALL (s/selected? :stop-id #(=% stop-id))] apply (f) v [com.rpl.specter :as s] args ))) {} (m) )) ( def (trips-by-direction) ( (group-by [com.rpl.specter :as s] : direction-id (distinct) ( filter [f mapper] # ([s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))]=() : route-id (%) 5 " [
s/ALL (s/selected? prop #(=% value))] (trips) )))) ;=> # 'user / trips-by-direction ( (map-values [
com.rpl.specter :as s] (trips-by-direction) (fn) [
"269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] () (distinct) ([s/ALL (s/selected? :stop-id #(=% stop-id))] map : headsign m
)))) ;=>
; {"1" ("L5a - MAISON NEUVE" "L5 - FORT MOSELLE"), ; "0" ("L5f - MAGNY PAR RUE DE POUILLY" "L5e - MAGNY PAR RUE DE BOIS" "L5 - REPUBLIQUE")} () Well I think at this point we pretty much covered what we need for our REST API, so let's do that! (Performance) for a dataset of the size of Metz it's easy, but if you were to load a huge dataset like Paris, I think these functions would take to much time on each HTTP request to filter. I'm just trying to find excuses to do some more because I'm bored:) So let's create a big (store) which will contain all the information already tailored for the API , so that all we have to do is a matter of (get-in [service-ids (into #{} (map :service-id (record-with-property day "1" calendars))) trip-service-id (:service-id (first (record-with-property :trip-id trip-id trips)))] that store to get what we need. Such kind of structure should be great: ({ ("agencies") : [a [k v], "" routes " : [
a [k v],
“stops” : [
a [k v],
("trips") : [
a [k v],
("store") : ({[
s/ALL (s/selected? :stop-id #(contains? republique-stops %))] “SEILLE” : ({[s/ALL (s/selected? :stop-id #(contains? republique-stops %))] (“Mettis B”) : {{ “MB - HOPITAL MERCY” : [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))], "MB - UNIVERSITE SAULCY" : [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))]
} } } } [
:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] We already have the (agencies) , (routes) , stops and (trips) . We also already know how to retrieve the following:
The different distinct stop names aka
store.SEILLE (The routes passing by a stop aka [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))] (store.SEILLE.'Mettis B '[record line]
(The destination of a specific route passing by a stop aka [{
; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] store.SEILLE.'Mettis B '.' MB - UNIVERSITE SAULCY '
(The trip ids of a specific route stop destination.) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] So it's all just about mapping functions we already have to build a giant structure so that it will be very easy for us to write our REST API.
Building the index
We are going to build an intermediate index with what we need:
[
s/ALL (s/selected? prop #(=% value))] routes routes indexed by their route-id ( (routes-map [com.rpl.specter :as s] , for performance)
(stops) (trips trips indexed by their trip-id) (trips-map) , for performance)
(trips indexed by route-id) (trips-by-route) for performance () (stop times) (stop times indexed by stop id) (stop-times-by-stop) , again for performance) [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] This is specific to what I have in mind, it's not necessary to understand how it's build, the key point is that we have the structure that I explained above so that the API just navigates in it, everything is prebuilt on startup.
The different distinct stop names aka store.SEILLE [{; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}]
:
() defn (distinct-stop-names-with-ids) “Find the distinct stops name with their associated ids.”
[]}]
( (map [
s/ALL (s/selected? prop #(=% value))] (#) [f mapper] (vector) (%) ( (map) [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] : stop-id (
stops-with-name % ))) ( (sort) ( (distinct) [
f mapper] () (map) : name stops ))))) ( (distinct-stop-names-with-ids) ({ : stops (stops }}) ;=>
; ([{
:keys [stops] ; 
; ; ; ; ; 01575879 ;
; ... (The routes passing by a stop aka [
number stop-id arrival-time headsign day] (store.SEILLE.'Mettis B '[s/ALL (s/selected? prop #(=% value))] : [ "05:19:05 L5a - MAISON NEUVE", "05:43:05 L5a - MAISON NEUVE", "06:05:32 L5a - MAISON NEUVE", "06:22:32 L5a - MAISON NEUVE",
Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] () defn (trip-ids-for-stops) “Find the trip ids for given number of stops.”
( (set) ( (map) [f mapper] : trip-id (flatten) (map) (#) ( get (stop-times) (%) stops ))))
( defn (distinct-route-ids-for-trips) “Find the distinct route ids for a given number of trips.” ( (distinct) ( (map) [f mapper] : route-id ( (map) (#) () [f mapper] get (trips) (%) (trip-ids) )))) ( defn routes-passing-by-stop-ids “Find the routes passing by a given number of stops.” }] ( filter (some?) [
com.rpl.specter :as s] (map) ([number stop-id arrival-time headsign day] (get [clojure.string :as str] (routes-map) % ) ( (distinct-route-ids-for-trips) () (trip-ids-for-stops) stop-ids (stop-times-by-stop) ) (trips-map) ))))
; test it works ( (map [
s/ALL (s/selected? prop #(=% value))] : long-name () routes-passing-by-stop-ids # {{[f mapper] "PIERNE (") } (store) ) ;=> ("Proxis ["ABBE BAUZIN" ("ABBEBAU1" "ABBEBAU2")] “Ligne 5” ( (map [s/ALL (s/selected? prop #(=% value))] : long-name () routes-passing-by-stop-ids # {{[f mapper] "REPUBL (") } (store) ) ;=> ("Ligne 3" "Ligne 5") ( (map [s/ALL (s/selected? prop #(=% value))] : long-name () routes-passing-by-stop-ids # {{[f mapper] "REPUBL } store )) ;=> ("Ligne 1" "Ligne 4" "Citeis [s/ALL (s/selected? :route-id #(=% route-id))] Navette CITY ") (The destination of a specific route passing by a stop aka store.SEILLE.'Mettis B '.' MB - UNIVERSITE SAULCY ' : () defn (times-by-headsign-at-stop) “Find the times a bus stops at a specific stop grouped by headsign.”
( (group-by [com.rpl.specter :as s] (#) : headsign (get) : trips-map [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] store [
s/ALL (s/selected? :stop-id #(=% stop-id))] ( : trip_id (%) )) ; group trips by trip headisgn ( (sort-by) ( (juxt) : arrival_time : departure-time ; sort by arrival-time then departure-time ( filter (#) [
f mapper] contains? ( (set) () get ( : trips-by-route (store [headsign] ) (route-id) )) ; filter trips ( (get [
s/ALL (s/selected? prop #(=% value))] ( : trips-map () (store) [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ([service-ids (into #{} (map :service-id (record-with-property day "1" calendars))) trip-service-id (:service-id (first (record-with-property :trip-id trip-id trips)))] : trip-id (%) ))) ; on routes they're on ( (get [
s/ALL (s/selected? prop #(=% value))] ( : stop-times-by-stop [clojure.string :as str]. (store) ) (stop) ))))) ; get the times at specified stop ( (keys [
s/ALL (s/selected? prop #(=% value))] ( (times-by-headsign-at-stop) (PIERNE) [com.rpl.specter :as s] [f mapper] "5 - (store) ) ;=> ("L5a - MAISON NEUVE") ( (count [
prop value records] ( (get [s/ALL (s/selected? prop #(=% value))] [f mapper] () (times-by-headsign-at-stop) "PIERNE [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] 5 [s/ALL (s/selected? prop #(=% value))] store )
"L5a - MAISON NEUVE" )) ;=> [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] Having written all that, we just need to use them all to create our desired structure:
() defn (make-api-struct ["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] "Construct the struct that will be searched by the REST API." ( ->> ( (distinct-stop-names-with-ids) the store [f mapper] ) ( (map [
s/ALL (s/selected? prop #(=% value))] (#) [f mapper] hash-map () first (%) ) ; create hash-map with stop name as key
( apply merge-with into ; and as value: ( (map [
s/ALL (s/selected? prop #(=% value))] ( (fn) [f mapper] hash-map : route_long_name (r) ) ; a hash-map with route name as key ( (map [
s/ALL (s/selected? prop #(=% value))] ( (fn) [f mapper] [stop route-id store] ( (times-by-headsign-at-stop) (stop) ( : route_id (r)
) the store [
clojure.java.io :as io] ) ; and as value the times grouped by destination ( (second) (%) )))) ( (routes-passing-by-stop-ids) () (second) (%) ) (store [r (io/reader f)] )) ))) (
apply merge-with [
f mapper] into
)) then create our
load-gtfs
method to build it once the parsing of the GTFS files has terminated: () defn (load-gtfs) "Load the GTFS data, and make a store of it."
[
stop route-id store]
( (let ( (let [
r]) (trips) ))
: stop-times (stop-times
: stop-times-by-stop ( apply merge-with into ( (map) (#) ( hash-map ( : stop_id %
) [
path]) (stop-times)
)}) ( (assoc) (store) : data ([
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] make-api struct (store) ) ))) ; ( (time) ( def [s/ALL (s/selected? prop #(=% value))] [f mapper] (store) (load-gtfs) "/ tmp / gtfs" ))) ; Loading /tmp/gtfs/routes.txt... ; Loading /tmp/gtfs/trips.txt... ; Loading /tmp/gtfs/stop_times.txt... ; Loading /tmp/gtfs/stops.txt... ; Loading /tmp/gtfs/agency.txt... ; "Elapsed time: [clojure.string :as str] msecs "
;=> # 'user / store
( (keys [
s/ALL (s/selected? prop #(=% value))] (store) ) ;=> (: trips-by-route: routes: stops: stop-times-by-stop: trips-map: stop-times: trips: routes-map: agencies: data) ( (keys [s/ALL (s/selected? prop #(=% value))] ( : data (store) ) ;=>
; "" PREFECTURE " ; "SERRET" ; "LECLERC" ; "P R ROCHAMBEAU" ; "PUYMAIGRE" ; "GENDARMERIE" ; "LA MAXE" ; ... ( (count [
prop value records] ( (keys) [f mapper] ([number stop-id arrival-time headsign day] : data (store)
))) ;=>
( (count [
prop value records] ( (distinct) [f mapper] (["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] map : name (stops ))) ;=>
Awesome, and seems ok performance wise. Now how do we use that structure I can hear you saying? It's just a matter of (get-in [
s/ALL (s/selected? :stop-id #(contains? republique-stops %))] the actual structure. We'll get to that in the next section. (REST API) We are going to build a simple API with compojure, I'm going to name it monmet (my Met ', Le Met' being the name of the bus network in Metz). (Init the project) Stop your REPL, modify the (deps.edn) file to put this content in it. [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] ({the paths : deps ({ org.clojure / clojure ({ : mvn / version (1). 22. 1 " }} (ring) ({: mvn / version (1.7.1) }} (ring / ring-core) ({ : mvn / version (“1.7.1”)
} (ring / ring-json) ({ : mvn / version ("0.5.0")
} (ring / ring-defaults) ({ : mvn / version (0.3.2) } (ring / ring-jetty-adapter) ({ : mvn / version (“1.7.1”) () (com.rpl / specter) ({ : mvn / version [
clojure.string :as str]. 2 }
(mount) ({: mvn / version (0.1.) }} (compojure) ({: mvn / version (1.6.1) }}} () and create the needed directories:
$ mkdir -p src / monmet $ (cd) src / monmet $ touch api.clj data.clj gtfs.clj handler.clj main.cl (Loading [s/ALL (s/selected? :stop-id #(=% stop-id))] In order to load one time our datastore (all the GTFS files) we're going to use mount. It's a very simple library, an alternative to components.
Let's do so by modifying the (data.clj) file with the following code:
() (ns) (monmet.data) ( : require ()
["resources" "src"]))
( (mnt / defstate) (store) : start () ([
s/ALL (s/selected? :stop-id #(=% stop-id))] gtfs / load-gtfs “/ tmp / gtfs” )) ( defn store-has-entry? "Check if the given entry is a valid one in our datastore." ( (contains? () # {{: agencies : routes : stops : trips : stop-times
} (entry) )) Here we have defined a state named store that will on call to (mnt / start) (invoke) (gtfs / load-gtfs "/ tmp / gtfs") which returns a map holding the various GTFS files content. It will be our datastore and it will be available in every namespace of our own, and it is guaranteed to be loaded only one time since it's an expensive operation. Routes
We will be using a ring and compojure to create the simple REST API we need.
The skeleton looks like this:
() (ns) (monmet.handler) ( : require () ]) [
s/ALL (s/selected? :stop-id #(=% stop-id))] ["resources" "src"]))
( (defroutes) (app-routes) ( (GET) "/ hello"
( (response) [
:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] {
: foo "bar" })) ( (route / not-found [
record line] (“Not Found”) ) ( def (app) ( (do [s/ALL (s/selected? prop #(=% value))] ( (mnt / start [com.rpl.specter :as s] )
; mount up the datastore ( -> () ( handler / site () (app-routes) () ( (middleware / wrap-json-body) ({
: keywords? (true) })
; transform fields to clojure keywords (middleware / wrap-json-response [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] ))) (GET / gtfs /: entry) On
GET (to) / gts /: entry where [
s/ALL (s/selected? prop #(=% value))] : entry is one of : agencies , : routes , : stops , : trips , : stop-times we will return what's inside the GTFS file directly, we're just serving the CSV files as JSON.
We'll need two methods of our (data) (namespace: [
number stop-id arrival-time headsign day] (store) which is our datastore holding the precious data, and
store-has-entry? which is a little helper to know if the requested entry really exists. when we are done it looks like this: () (ns) (monmet.handler) ( : require () ]) [
s/ALL (s/selected? :stop-id #(=% stop-id))]
])) ( (defroutes) (app-routes) ; Expose GTFS files directly, just for fun:) ( (GET) "/ gtfs /: file" [
f mapper]
( (let ; transform URL path to keyword ( (if [
s/ALL (s/selected? prop #(=% value))] ( (store-has-entry? [f mapper] (entry) )
( (response) ( (entry) [
f mapper] (store) )
; in case the entry exist, just serve what's in the datastore ( (route / not-found [
record line] (“Not Found”) ))) ; otherwhise, ( (route / not-found [record line] (“Not Found”) ) ( def (app) ( (do [s/ALL (s/selected? prop #(=% value))] ( (mnt / start [com.rpl.specter :as s] )
; mount up the datastore ( -> () ( handler / site () (app-routes) () ( (middleware / wrap-json-body) ({
: keywords? (true) }) (middleware / wrap-json-response [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] ))) Test it works:
Run the program with
clj -m monmet.main and use curl
$ curl http: // localhost: / gtfs / agencies
(GET / api / ... (bus routes) To make things clearer we ' ll create another namespace called (api) where we will really need our [headsign] (gtfs) (namespace to do some work.) Let's start with routes. I'd like to return the content of (routes.txt) but enhanced with the [s/ALL (s/selected? :stop-id #(=% stop-id))] Agency [f mapper] corresponding to the : agency_id (field of each) (Route) . () (ns) (monmet.api) ( : require () [store {:agencies (load-records (str path "/agency.txt") ->Agency) :routes routes :routes-map (into {} (map #(vector (:route_id %) %) routes)) :stops (load-records (str path "/stops.txt") ->Stop) :trips trips :trips-map (into {} (map #(vector (:trip_id %) %) trips)) :trips-by-route (apply merge-with into (map #(hash-map (:route_id %) [%]))
( defn (get-routes) “Retrieve routes.” ( : routes () store
) [
f mapper] We just need to expose that very api / get-routes in our compojure defroutes : () defroutes (app-routes) ; ... ( (GET) "/ api / routes" () (response) ( (api / get-routes) (store) ))) ; ... ( (route / not-found [record line] (“Not Found”) ) That's it, test it using curl (or even your browser):
$ curl -X GET http: // localhost: 5052 / api / routes Bus stops [clojure.string :as str] Now that our bus routes are returned, we really need some bus stops or we won't be able to enter the damn bus! Get back to the (api) namespace in order to create two methods, one to retrieve the distinct bus stops sorted by name and one to retrieve all the bus stops matching a name.
() (ns) (monmet.api) ( : require () [store {:agencies (load-records (str path "/agency.txt") ->Agency) :routes routes :routes-map (into {} (map #(vector (:route_id %) %) routes)) :stops (load-records (str path "/stops.txt") ->Stop) :trips trips :trips-map (into {} (map #(vector (:trip_id %) %) trips)) :trips-by-route (apply merge-with into (map #(hash-map (:route_id %) [%]))
; ... ( defn (get-stops) "Retrieve the distinct stop names."
( (gtfs / distinct-stop-names-with-ids (store)
)) ( defn (get-stop-with-name) "Retrieve the stops matching (strict equality) a name." (
gtfs / stops-with-name [
clojure.java.io :as io] (stop-name) ( : stops store [com.rpl.specter :as s] ()) Expose them in the API:
() (GET)
“/ / api / stops”
[
s/ALL (s/selected? :route-id #(=% route-id))] () (response) () api / get-stops
store )))
( (GET) "/ api / stops /: name" [
name] ( (response) ( (api / get-stop-with-name) (name) (store)
)))
test it:
$ curl -X GET http: // localhost: 5052 / api / stops | (jq) ('.') ] , [f mapper] ] , ... ] $ curl -X GET http: // localhost: 8236 / api / stops / PIERNE
working from the first try. It's not even funny! Routes passing by a bus stop I said earlier that I ' d like the user to first select the stop, then the route, so we need to propose an endpoint to do that, and it will be bound to / api / stops /: name / routes . This endpoint needs to return the different routes passing by the specified stops, and also for each of these route return the associated possible headsigns. () (ns) (monmet.api) ( : require () [
store {:agencies (load-records (str path "/agency.txt") ->Agency) :routes routes :routes-map (into {} (map #(vector (:route_id %) %) routes)) :stops (load-records (str path "/stops.txt") ->Stop) :trips trips :trips-map (into {} (map #(vector (:trip_id %) %) trips)) :trips-by-route (apply merge-with into (map #(hash-map (:route_id %) [%]))
; ... ( defn (direction-names-for-stop-and-route
( (sort) ( filter [
f mapper] (not-empty) ( (flatten) ( (map keys [s/ALL (s/selected? prop #(=% value))] ([s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] (get-in) (store) [f mapper] )))))) ( defn route-names-and-directions-for-stop ( (map [s/ALL (s/selected? prop #(=% value))] (#) [f mapper] hash-map (%) () ( (direction-names-for-stop-and-route (stop) (%) [f mapper] store
) ( (route-names-for-stop) (stop) (store [
record line]
))) ; ... [
"05:19:05 L5a - MAISON NEUVE", "05:43:05 L5a - MAISON NEUVE", "06:05:32 L5a - MAISON NEUVE", "06:22:32 L5a - MAISON NEUVE",
Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] Expose this in the REST API : () (GET) "/ api / stops /: name / routes"
[com.rpl.specter :as s] ( (response) ( (api / route-names-and-directions-for-stop) name (store [headsign] ))) Test it's working: $ curl -X GET http: // localhost: 5052 / api / stops / REPUBLIQUE / routes | (jq) '.' , "" route ": ({ "route_id" : 5 ", "route_short_name" : (5) , "route_long_name" : “Ligne 5” , "route_desc" : [
f mapper] , "route_type" : (3) , "route_url" : [f mapper] } } , ({[s/ALL (s/selected? :stop-id #(contains? republique-stops %))] ("name") : “Proxis” , ("directions") : , "" route ": ({ "route_id" : [f mapper] 127 , "route_short_name" : [f mapper] , "route_long_name" : “Proxis” , "route_desc" : [f mapper] , "route_type" : (3) , "route_url" : [f mapper] } } ] [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] (Working as expected:) Still we miss one piece of the puzzle, the time table. (Times for a trip at a bus stop) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] Final piece of our API, we need to retrieve the times of day when a bus stops at our favorite bus stop. Besides, we want to query for multiple route direction at the same stop, because sometimes you just want to go somewhere and there are multiple headsigns that goes where you want to go, so you don't care what bus you take. Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] Expose this in the REST API : () (GET) "/ / api / stops /: name / routes /: routes" [
s/ALL (s/selected? prop #(=% value))] ( (response) ( (api / times-for-stop-and-multiple-route-direction) (name) routes () store )) () The format I decided is the following Route ::: Headsign , multiple routes / headsign can be separated by ::: . Let's encode some URI using this useful encoder that I just found on Google: (http://meyerweb.com/eric/tools/dencoder/) [headsign] (Ligne 5)=(Ligne%) (L5a - MAISON NEUVE)=(L5a%) -% (MAISON%) NEUVE (L5 - FORT MOSELLE)=(L5%) -% (FORT%) (MOSELLE) [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] The URL to call is: (http: // localhost:) / api / stops / PIERNE / routes / Ligne% ::: L5a% -% (MAISON%) NEUVE — Ligne% 491 ::: L5% (%) (FORT%) (MOSELLE) ($ curl -X GET) 'http: // localhost: 98264 / api / stops / PIERNE / routes / Ligne% 823 ::: L5a% -% (MAISON%) NEUVE --- Ligne% 491 ::: L5% 41 -% (FORT%) (MOSELLE ') | (jq) 'map (.arrival_time ' ' .headsign)' [f mapper] | head -3 }]
( (str) (headsign) “arivant a” [com.rpl.specter :as s] ([s/ALL (s/selected? prop #(=% value))] [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] str / replace ([s/ALL (s/selected? prop #(=% value))] [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] subs
(arrival_time) (0) (5)
) [
headsign] [f mapper] # (): "heures"
) “.” )) ( defn (times-for-stop-and-multiple-route-direction-spelled) ( (let () ( (str) "Les 3 prochains bus a l'arret " " (stop) "sont les suivants. n n-" [
com.rpl.specter :as s] apply str [f mapper] (interpose) " n-" (next-3) ))))
; ... and the route:
() (GET)
"/ / api / stops /: name / routes /: routes / spell"
( (response) ( api / times-for-stop-and-multiple-route-direction-spelled (name) (routes) (store) )) then run the server again, and launch ngrok: ($ clj -m monmet.main) & $ ngrok http ngrok by @inconshreveable () (Ctrl C to quit) [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] Session Status online Session Expires (7) hours, [f mapper]
minutes Version (2) 3. Region United States ([
s/ALL (s/selected? :stop-id #(contains? republique-stops %))] (us) ) Web Interface http: // 250. 0.0.1: Forwarding http: //
b.ngrok.io -> http: // localhost: Forwarding https: //
b.ngrok.io -> http: // localhost: then just create a shortcut in your iPhone like this: [f mapper] [f mapper] And finally, ask Siri for it:
[
f mapper] BAM. Done. I have now three different shortcut for when I need to go to the train station or city center, or when I'm in the city and wan't to get back home. [ "05:19:05 L5a - MAISON NEUVE", "05:43:05 L5a - MAISON NEUVE", "06:05:32 L5a - MAISON NEUVE", "06:22:32 L5a - MAISON NEUVE",
Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] Works from anywhere, every day . I just need to download the GTFS files from time to time (two times per year). (All this with under lines of badly written, non optimized Clojure 🤠. (Until next time 🤘!) (Read More)
For more information, you can take a look at the following page: https://developers.google.com/transit/gtfs/
Sample GTFS archive.
I live in Metz, France . Hopefully we can grab a GTFS archive with the desired data for all the buses riding in my little town. It can be found here: gtfs_current.zip [
stop-name store]
Tinkering from the REPL [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] Create adeps.edn (file with the following content: [
number stop-id arrival-time headsign day] ({: deps ({[s/ALL (s/selected? :stop-id #(contains? republique-stops %))] (com.rpl / specter) ({ : mvn / version “1.1.3” } (nrepl / nrepl) ({: mvn / version (“0.5.3”) }}} (then just run nREPL:) $ clj -m nrepl.cmdline nREPL server started on port on host localhost – nrepl: // localhost: [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] Now connect your IDE to this running REPL. Or just create one from your IDE directly, it does not matter. I am currently running on Windows and run my nREPL from the Windows Linux Subsystem and attach my IDE running in Windows to it.
(Load the GTFS files) Now that we have the project ready, unzip the GTFS archive and place it somewhere, let’s say in / tmp / gtfs . We’ve got multiple files extracted:
agency.txt : contains the various bus operators (here (LE MET ') and
PROXIS ) (calendar.txt) : contains information about whether the different services and on what day do they operate (here we have 4 different ([{; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] mon-fri , wed wed , (sat) , sun )) (calendar_dates.txt) : contains exception of services, for example the service id (1) does not operate on 4953 - - . (routes.txt) : contains the different routes that a bus could take ( (here). (stop_times.txt) : contains the different stops for each bus trip, indicating hour of arrival and departure. (stops.txt) : contains all the bus stops (here [s/ALL (s/selected? :stop-id #(contains? republique-stops %))] ). (trips.txt) : contains all the different bus trips linking stops together (here [service-ids (into #{} (map :service-id (record-with-property day "1" calendars))) trip-service-id (:service-id (first (record-with-property :trip-id trip-id trips)))] )
(Ok now that we know what each file contains, just load them up as Clojure records, and see what we can do. What we'll need to do is essentially: (Load a file (which are CSV Skip the first line Create a record instance for each line [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))] (Use that list of lines to do something usefull [headsign] We will first define the various map shapes for all these files:
; an agency is composed of a name, a URL, a time zone, a lang and a phone
( def -> agency
[
:name :url :tz :lang :phone])
; etc … ( def -> calendar
[:service-id :mon :tue :wed :thu :fri :sat :sun :start-date :end-date])
( def -> route [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ( def -> stop-time [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled]) ( def -> stop [:stop-id :code :name :desc :lat :lon :id :zone-id :url :location-type :parent-station-id]) ( def -> trip [:route-id :service-id :trip-id :headsign :direction-id :block-id :shape-id :block-id-alt])
Everything OK? Just try to load the agencies: () (require ' [f mapper] [clojure.java.io :as io]) ( require ' [
clojure.string :as str] [clojure.string :as str] ( defn (make-record) “Make a record out of a line where fields are separated by a comma.” [f mapper] ( ->> ( (str / split) (line) (#) “,” - 1 ; split on commas ( (map [
s/ALL (s/selected? prop #(=% value))] (str / trim) ) ; remove un-needed spaces ( (map [
s/ALL (s/selected? prop #(=% value))] (#) [f mapper] (str / replace) (%) () "" " [f mapper] ) ; remove un-needed double quotes ( (zipmap) (record) )) ; create a map of the separated line ( defn (load-records) "Load the given file into a list of maps, discarding the header." [
f mapper] [f mapper] ( (with-open [r (io/reader f)] [com.rpl.specter :as s] ( ->> ( (doall) () (line-seq) (r) )) (rest) ; skip the header ( (map [
s/ALL (s/selected? prop #(=% value))] ( (partial) [f mapper] (make-record) (mapper) )))) ( (load-records) "/ tmp / gtfs / agency.txt" -> agency () ;=> ({ ; name: "SAEML TAMM (Le Met ')", ; : url "https://lemet.fr", ; tz "Europe / Paris", ; : lang "", ; : phone ""}) [
f mapper] It works! We have a sequence of one (Agency) record. Pretty neat! Great, now let's dive deeper into the project. We really need to link routes and bus stops. [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] To do so we need to load stops , (trips) , (routes) and (stop-times) [headsign] ; load them all ( def (routes) [
f mapper] (load-records) "/ tmp / gtfs / routes. txt " -> route )) ( def (stops) [f mapper] (load-records) "/ tmp / gtfs / stops.txt -> stop [f mapper] ) ( def (stop-times) (load-records) (/ tmp / gtfs / stop_times.txt " -> stop-time [
f mapper] ))
( def (trips) [
f mapper] (load-records) "/ tmp / gtfs / trips.txt -> trip [headsign] [f mapper] )
( def (calendars) [
f mapper] (load-records)
"/ tmp / gtfs /calendar.txt" -> calendar [
s/ALL (s/selected? :stop-id #(=% stop-id))] )) Just check that I can find the stop near my appartment:
() (count) () (filter) (#) ()
=
([s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] :: name (%)
) ("PIERNE") ) (stops) ))
;=> 2 There are 2 of them. One for each direction. We can just check to be sure:
() (filter) (#) ()
=(
: name (%) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))]) PIERNE " [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))]) stops
;=> ; ({: location-type "", ; : desc "",
; name: "PIERNE", ; : id "",
; stop-id "PIERNE 10 ", ; : lon "6. 01575879 ",
; : url "0", ; code: ("," ; zone-id "http://lemet.fr/screen/index2.php?stop=728 ", ; lat: . “} ; {: location-type "", ; : desc "",
; name: "PIERNE", ; : id "",
; stop-id "PIERNE ",
; : lon "6. , ; : url "0", ; code: 728 , ; zone-id "http://lemet.fr/screen/index2.php?stop=728 ", ; lat: . “}) (Discover the data) Let's define some utilities methods to retrieve stops and routes easily, using (specter) which is ideal for traversing data structures and such exploration.
() (require ' [
f mapper] [com.rpl.specter :as s] [com.rpl.specter :as s] ( defn (record-with-property) “Find all records having a specific property matching a given value.”
[
s/ALL (s/selected? prop #(=% value))] ( (s / select [f mapper] [s/ALL (s/selected? prop #(=% value))] (records)
) ; Take a look at the use of s / select-one instead of s / select *. ( defn (stop-with-id) “Find a stop having a given id.”
[
s/ALL (s/selected? prop #(=% value))] ( (s / select-one [com.rpl.specter :as s] (stops) ) ; define some additional methods ( defn (stops-with-name) “Find all stops with a given name.”
[
s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : name (name [f mapper] stops () )) ( defn routes-with-short-name "Find all routes with a given short name." [s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : short-name (name [clojure.string :as str] (routes) ) ( defn routes-with-long-name "Find all routes with a given long name." [s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : route-long-name name (routes)
))
( (routes-with-short-name) (5)
) ;=> [{
; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] ( (count [prop value records] ( (stops-with-name [com.rpl.specter :as s] “PIERNE” ))
;=> 2 ; CASINO is where I lived when I was young ; https://www.google.fr/maps/place/ (% C2% B0) ' 5% (N 6% C2% B0) [
clojure.string :as str] 5% E / @ . 281461, 6. [r (io/reader f)] , (z) ( (count [prop value records] ( (stops-with-name [com.rpl.specter :as s] “CASINO” ))
;=> 2 ; how many bus stops do we have in Metz? ( (count [
prop value records] ( (distinct) [f mapper] (["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] map : name (stops ))) ;=>
So far it seems to be retrieving what we want. Next I'd like to resume what I will need to do to retrieve the next 3 stops for a specific route and bus stop: Retrieve the route with its short name (usually the bus line number) (Retrieve the trips for that route (a trip is a route in a defined way) (A to B) or B to A )
Retrieve the stops on that route (for that we need to load both the trips and time table) (Retrieve from the time table) stop_times.txt ) ; get the route id for the route
( (map [
s/ALL (s/selected? prop #(=% value))] : route-id () routes-with-short-name 5 "
))
=> () "5 - [
headsign] [f mapper] )
( defn (trips-for-route) “Find all trips for a given route-id.” [
s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : route-id (route-id (trips) ) ( (distinct) ( (map) [f mapper] : headsign (trips-for-route) ("5 - 99 ” )))
;=>
; ("L5a - MAISON NEUVE" ; "L5f - MAGNY PAR RUE DE POUILLY" ; "L5e - MAGNY PAR RUE AU BOIS" ; "L5 - FORT MOSELLE" ; "L5 - REPUBLIQUE" ( (count [
prop value records] ( (trips-for-route) 5 -
))
;=> [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ( defn (trips-with-headsign) “Find all the trips given a headsign.”
[
s/ALL (s/selected? prop #(=% value))] ( (record-with-property) : headsign headsign (trips) ) ( (count [prop value records] ( (trips-with-headsign) L5a - MAISON NEUVE [com.rpl.specter :as s] () )) ;=> ; it means that there 326 trips on Line 5 that goes in direction of the final stop MAISON NEUVE. ; Let's check what the first trip is
( (first) () ( (trips-with-headsign) "L5a - MAISON NEUVE"
))
;=>
; {: route-id "5 - 98, ; service-id "HIV - DIM_HIV-Dimanche - , ; trip-id " - HIV 4953 - DIM_HIV-Dimanche - 10 ", ; : headsign "L5a - MAISON NEUVE", ; direction-id "1", ; block-id "179863, ; shape-id " ", ; block-id-alt " (-) “} (We) want (all) (the [
s/ALL (s/selected? :stop-id #(=% stop-id))] [f mapper] (different) (stops (on) (a) (trip) : () (clojure) ( defn (stops-on-trip) "Find all the stops on a given trip." [com.rpl.specter :as s] ( (record-with-property) : trip-id (trip-id) stop-times [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ) ( (count [prop value records] ( (stops-on-trip [com.rpl.specter :as s] - HIV 4953 - DIM_HIV-Dimanche - [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))] )
;=> ( (map [
s/ALL (s/selected? prop #(=% value))] ( (comp) [f mapper] : name (stop-with-id) : stop-id [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ( (stops-on-trip [record line] - HIV 4162 - DIM_HIV-Dimanche - [f mapper] () )) ;=>
; ("AUBEPINE" ; "HAUTS-DE-MAGNY" ; "BEAUSOLEIL" ; "ARMOISIERES" ; "OBELLIANE" ; "MAGNY-AU-BOIS" ; "ROOSEVELT" ; "LA PLAINE" ; "APREMONT" ; "PLATEAU" ; "FAUBOURG" ; "FRECOT" ; "BOUCHOTTE" ; "VANDERNOOT" ; "LOTHAIRE" ; "PIERNE" ; "LEMUD" ; "MUSE" ; "Center POMPIDOU METZ" ; "GARE" ; "ROI GEORGE" ; "REPUBLIQUE" ; "SQUARE DU LUXEMBOURG" ; "FORT MOSELLE" ; "TIGNOMONT" ; "ST-MARTIN" ; "FOCH" ; "PONT DE VERDUN" ; "CASINO" ; "MIGETTE" ; "LONGEVILLE" ; "LECLERC" ; "EN PRILLE" ; "SCY BAS" ; "LIBERTE" ; "MOULINS" ; "ST-JEAN" ; "SERRET" ; "HAIE BRULEE" ; "MAISON NEUVE" Let's say it is : , I want to know each 3 bus trips after : where the bus are at my stop for the route (L5a - MAISON NEUVE) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] () defn (times-for-stop) “Find the times for a given stop.” [
s/ALL (s/selected? :stop-id #(=% stop-id))] ( (record-with-property) : stop-id (stop-id) stop-times [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ) ( (count [prop value records] ( times-for-stop [com.rpl.specter :as s] PIERNE () ;=> ( (take) (3) ( (sort-by) : arrival-time ( (s / select [
f mapper] [com.rpl.specter :as s] ( (times-for-stop) "PIERNE 08 "
) ))) ;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 - DIM_HIV-Dimanche - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV -SEM_GTFS-Semaine - 10 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4953 - H SAM-Samedi 10, ; : arrival-time ": : ", ; : departure-time ": : ", ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) That's cool and all but it might return me the stops for all different routes, I want only those on the route named (Ligne 5a - MAISON NEUVE) :
() defn (trip-in-route?) [f mapper] ( contains? ( (set) () (map) : trip-id trips
) (trip-id) [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))])) ( (take) (3) ( (sort-by) : arrival-time ( (s / select [f mapper] [com.rpl.specter :as s] ( (times-for-stop) "PIERNE 08 "
) ))) ;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 - DIM_HIV-Dimanche - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV -SEM_GTFS-Semaine - 10 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4953 - H SAM-Samedi 10, ; : arrival-time ": : ", ; : departure-time ": : ", ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) It gives the exact same result , and that for a reason. The
PIERNE (bus stop) with id () is only served by the bus line [
record line] (Ligne 5) . Just check that these 3 buses have the same arrival: () defn (trip-with-id) “Find a trip given its id.” [com.rpl.specter :as s] ( (s / select-one [com.rpl.specter :as s] (trips) )) ( (map [s/ALL (s/selected? prop #(=% value))] ( (comp) [f mapper] : headsign (trip-with-id)
) [
s/ALL (s/selected? prop #(=% value))] ;=> ("L5a - MAISON NEUVE" "L5a - MAISON NEUVE" "L5a - MAISON NEUVE") Just one last tweak to do before we go to the next step.
Currently it gives the trips for all days of the week, but in Metz (and elsewhere) we have different trips during the week and during the week-end, let's say we're tuesday.
; Which services are valid for tuesday?
( (map [
s/ALL (s/selected? prop #(=% value))] : service-id () (record-with-property) : tue “1” [f mapper] calendars
) ;=> ("HIV ["19 NOVEMBRE" ("19NOV01" "19NOV02")] - SEM_GTFS-Semaine - () ; let's deal with it ( defn (trip-for-day?) “Does the given trip-id runs on the given day.”
[
com.rpl.specter :as s] ( (let ["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] ( (contains? () (service-ids) (trip-service-id)
))) ( (take) (3) ( (sort-by) : arrival-time ( (s / select [
f mapper] ; running on tuesday ( (times-for-stop)
"PIERNE [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] ))) ;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : 54 ", ; : departure-time ": : 54 ", ; stop-id "PIERNE 10 ", ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV 4162 - SEM_GTFS-Semaine - ",
; : arrival-time ": : 98, ; : departure-time ": : 98, ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) ; Make a function of it ( defn (next-times) [{
; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] ( (take) (number) ( (sort-by) : arrival-time ( (s / select [f mapper] ( (times-for-stop) (stop-id)
))))) ; check it works ( (next-times [
com.rpl.specter :as s] (3)
"PIERNE [
com.rpl.specter :as s] 41: : [headsign] [f mapper] "L5a - MAISON NEUVE" : tue [s/ALL (s/selected? :stop-id #(=% stop-id))] )
;=>
; ({: trip-id "[{
:keys [stops] - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : , ; : departure-time ": : , ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 ”} ; {: trip-id " - HIV 4162 -SEM_GTFS-Semaine - 16 ", ; : arrival-time ": : 54 ", ; : departure-time ": : 54 ", ; stop-id "PIERNE , ; stop-sequence "41, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "} ; {: trip-id " - HIV 4162 - SEM_GTFS-Semaine - ",
; : arrival-time ": : 98, ; : departure-time ": : 98, ; stop-id "PIERNE 10 ", ; stop-sequence "35, ; : pickup-type "0", ; : drop-off-type "0", ; shape-dist-traveled ". 0 "}) (Seems about right 🤠!) So far, so good, we now know how to list the different routes, retrieve the stops of a route with specific direction (headsign), and how to retrieve the next 3 stops at a bus station. The problem is that people that will be using the application will also need to retrieve the routes and trips from a stop, because they usually already know the name of their favorite bus stops. Let's do that with a bus stop that has many bus lines passing by, named (REPUBLIQUE) : () (count) () stops- with-name "REPUBLIQUE" )) ;=> 9 ( def republique-stops ( (set) ( (map) [f mapper] : stop-id (stops-with-name) “REPUBLIQUE” ))) ;=> # 'user / republique-stops (republique-stops) ;=> # {"REPU “REPUB ” “place_REPUB” REP [
clojure.string :as str] “REPUBL ”). "REPUBL “REPU (“REP” “REPUB 2019 "} ( def (republique-stop-times-trip-ids) ( (set) ( (map) [f mapper] : trip-id s / select stop- times )))) ;=> # 'user / republique-stop-times-trip-ids ( (count [prop value records] (republique-stop-times-trip-ids) ) ;=> ( (distinct) ( (map) [
f mapper] : route-id ( s / select [f mapper] trips ()) ;=> (" - [clojure.string :as str] - 5 - 9 - 4-7 "" 3 - “2-5” 1 - B - A- 159 “” 113 - () ( defn (route-with-id) [route-id] ( (s / select-one [s/ALL (s/selected? prop #(=% value))] (routes) )) ( (map [
s/ALL (s/selected? prop #(=% value))] : long-name () (map) (route-with-id) ' ( 41 - - 9 " [f mapper] "5 - "4-7" 3 - ("2-5") 1 - 234 " “B - (A - 101 - ))) ;=> ( ; "Citeis [
f mapper] ; "Citeis ; “Ligne 5” ; “Ligne 4” ; “Ligne 3” ; “Ligne 2” ; "Ligne 1" ; “Mettis B” ; “Mettis A” ; "Navette CITY") We can conclude that there are (bus lines that goes through the (REPUBLIQUE) bus stop. Having that we'll need to propose the choice of a direction (headsign) for a chosen route: ; route id "5 - 100 "is L5: MAGNY - MAISON NEUVE the one I used to take when I was going to school.
( (distinct) ( (map [s/ALL (s/selected? prop #(=% value))] : headsign [com.rpl.specter :as s] (filter) (#) () = () route-id (%) () (5) (trips) )) ;=> ("L5a - MAISON NEUVE" ; "L5f - MAGNY PAR RUE DE POUILLY" ; "L5e - MAGNY PAR RUE AU BOIS" ; "L5 - FORT MOSELLE" ; "L5 - REPUBLIQUE" ( (distinct) ( (map [
s/ALL (s/selected? prop #(=% value))] : headsign [com.rpl.specter :as s] (filter) (#) () = () route-id (%) () (B - [
f mapper] () ) (trips) )))
;=> ("MB - HOPITAL MERCY" "MB - UNIVERSITE SAULCY") () There are 5 different final destination for the bus line number (Ligne 5) and 2 final destinations for the bus line (Mettis B) finally, one last problem: a trip with a final destination (head sign) has a determined way, it goes from A to B or in reverse, and we should really group these trips by destination: ; There's no built-in method in Clojure that I know of to map on the values of a hashmap
( defn (map-values) ( reduce ( (fn) [
f mapper] [headsign]] ( (assoc) (a) k ([
s/ALL (s/selected? :stop-id #(=% stop-id))] apply (f) v [com.rpl.specter :as s] args ))) {} (m) )) ( def (trips-by-direction) ( (group-by [com.rpl.specter :as s] : direction-id (distinct) ( filter [f mapper] # ([s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))]=() : route-id (%) 5 " [
s/ALL (s/selected? prop #(=% value))] (trips) )))) ;=> # 'user / trips-by-direction ( (map-values [
com.rpl.specter :as s] (trips-by-direction) (fn) [
"269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] () (distinct) ([s/ALL (s/selected? :stop-id #(=% stop-id))] map : headsign m
)))) ;=>
; {"1" ("L5a - MAISON NEUVE" "L5 - FORT MOSELLE"), ; "0" ("L5f - MAGNY PAR RUE DE POUILLY" "L5e - MAGNY PAR RUE DE BOIS" "L5 - REPUBLIQUE")} () Well I think at this point we pretty much covered what we need for our REST API, so let's do that! (Performance) for a dataset of the size of Metz it's easy, but if you were to load a huge dataset like Paris, I think these functions would take to much time on each HTTP request to filter. I'm just trying to find excuses to do some more because I'm bored:) So let's create a big (store) which will contain all the information already tailored for the API , so that all we have to do is a matter of (get-in [service-ids (into #{} (map :service-id (record-with-property day "1" calendars))) trip-service-id (:service-id (first (record-with-property :trip-id trip-id trips)))] that store to get what we need. Such kind of structure should be great: ({ ("agencies") : [a [k v], "" routes " : [
a [k v],
“stops” : [
a [k v],
("trips") : [
a [k v],
("store") : ({[
s/ALL (s/selected? :stop-id #(contains? republique-stops %))] “SEILLE” : ({[s/ALL (s/selected? :stop-id #(contains? republique-stops %))] (“Mettis B”) : {{ “MB - HOPITAL MERCY” : [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))], "MB - UNIVERSITE SAULCY" : [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))]
} } } } [
:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] We already have the (agencies) , (routes) , stops and (trips) . We also already know how to retrieve the following:
The different distinct stop names aka
store.SEILLE (The routes passing by a stop aka [s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00")))] (store.SEILLE.'Mettis B '[record line]
(The destination of a specific route passing by a stop aka [{
; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}] store.SEILLE.'Mettis B '.' MB - UNIVERSITE SAULCY '
(The trip ids of a specific route stop destination.) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] So it's all just about mapping functions we already have to build a giant structure so that it will be very easy for us to write our REST API.
Building the index
We are going to build an intermediate index with what we need:
[
s/ALL (s/selected? prop #(=% value))] routes routes indexed by their route-id ( (routes-map [com.rpl.specter :as s] , for performance)
(stops) (trips trips indexed by their trip-id) (trips-map) , for performance)
(trips indexed by route-id) (trips-by-route) for performance () (stop times) (stop times indexed by stop id) (stop-times-by-stop) , again for performance) [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] This is specific to what I have in mind, it's not necessary to understand how it's build, the key point is that we have the structure that I explained above so that the API just navigates in it, everything is prebuilt on startup.
The different distinct stop names aka store.SEILLE [{; :route-id "5-77", ; :short-name "5", ; :long-name "Ligne 5", ; :desc "", ; :type "3", ; :url ""}]
:
() defn (distinct-stop-names-with-ids) “Find the distinct stops name with their associated ids.”
[]}]
( (map [
s/ALL (s/selected? prop #(=% value))] (#) [f mapper] (vector) (%) ( (map) [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] : stop-id (
stops-with-name % ))) ( (sort) ( (distinct) [
f mapper] () (map) : name stops ))))) ( (distinct-stop-names-with-ids) ({ : stops (stops }}) ;=>
; ([{
:keys [stops] ; ; ; ; ; ; 01575879 ;
; ... (The routes passing by a stop aka [
number stop-id arrival-time headsign day] (store.SEILLE.'Mettis B '[s/ALL (s/selected? prop #(=% value))] : [ "05:19:05 L5a - MAISON NEUVE", "05:43:05 L5a - MAISON NEUVE", "06:05:32 L5a - MAISON NEUVE", "06:22:32 L5a - MAISON NEUVE",
Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] () defn (trip-ids-for-stops) “Find the trip ids for given number of stops.”
( (set) ( (map) [f mapper] : trip-id (flatten) (map) (#) ( get (stop-times) (%) stops ))))
( defn (distinct-route-ids-for-trips) “Find the distinct route ids for a given number of trips.” ( (distinct) ( (map) [f mapper] : route-id ( (map) (#) () [f mapper] get (trips) (%) (trip-ids) )))) ( defn routes-passing-by-stop-ids “Find the routes passing by a given number of stops.” }] ( filter (some?) [
com.rpl.specter :as s] (map) ([number stop-id arrival-time headsign day] (get [clojure.string :as str] (routes-map) % ) ( (distinct-route-ids-for-trips) () (trip-ids-for-stops) stop-ids (stop-times-by-stop) ) (trips-map) ))))
; test it works ( (map [
s/ALL (s/selected? prop #(=% value))] : long-name () routes-passing-by-stop-ids # {{[f mapper] "PIERNE (") } (store) ) ;=> ("Proxis ["ABBE BAUZIN" ("ABBEBAU1" "ABBEBAU2")] “Ligne 5” ( (map [s/ALL (s/selected? prop #(=% value))] : long-name () routes-passing-by-stop-ids # {{[f mapper] "REPUBL (") } (store) ) ;=> ("Ligne 3" "Ligne 5") ( (map [s/ALL (s/selected? prop #(=% value))] : long-name () routes-passing-by-stop-ids # {{[f mapper] "REPUBL } store )) ;=> ("Ligne 1" "Ligne 4" "Citeis [s/ALL (s/selected? :route-id #(=% route-id))] Navette CITY ") (The destination of a specific route passing by a stop aka store.SEILLE.'Mettis B '.' MB - UNIVERSITE SAULCY ' : () defn (times-by-headsign-at-stop) “Find the times a bus stops at a specific stop grouped by headsign.”
( (group-by [com.rpl.specter :as s] (#) : headsign (get) : trips-map [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] store [
s/ALL (s/selected? :stop-id #(=% stop-id))] ( : trip_id (%) )) ; group trips by trip headisgn ( (sort-by) ( (juxt) : arrival_time : departure-time ; sort by arrival-time then departure-time ( filter (#) [
f mapper] contains? ( (set) () get ( : trips-by-route (store [headsign] ) (route-id) )) ; filter trips ( (get [
s/ALL (s/selected? prop #(=% value))] ( : trips-map () (store) [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] ([service-ids (into #{} (map :service-id (record-with-property day "1" calendars))) trip-service-id (:service-id (first (record-with-property :trip-id trip-id trips)))] : trip-id (%) ))) ; on routes they're on ( (get [
s/ALL (s/selected? prop #(=% value))] ( : stop-times-by-stop [clojure.string :as str]. (store) ) (stop) ))))) ; get the times at specified stop ( (keys [
s/ALL (s/selected? prop #(=% value))] ( (times-by-headsign-at-stop) (PIERNE) [com.rpl.specter :as s] [f mapper] "5 - (store) ) ;=> ("L5a - MAISON NEUVE") ( (count [
prop value records] ( (get [s/ALL (s/selected? prop #(=% value))] [f mapper] () (times-by-headsign-at-stop) "PIERNE [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] 5 [s/ALL (s/selected? prop #(=% value))] store )
"L5a - MAISON NEUVE" )) ;=> [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] Having written all that, we just need to use them all to create our desired structure:
() defn (make-api-struct ["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] "Construct the struct that will be searched by the REST API." ( ->> ( (distinct-stop-names-with-ids) the store [f mapper] ) ( (map [
s/ALL (s/selected? prop #(=% value))] (#) [f mapper] hash-map () first (%) ) ; create hash-map with stop name as key
( apply merge-with into ; and as value: ( (map [
s/ALL (s/selected? prop #(=% value))] ( (fn) [f mapper] hash-map : route_long_name (r) ) ; a hash-map with route name as key ( (map [
s/ALL (s/selected? prop #(=% value))] ( (fn) [f mapper] [stop route-id store] ( (times-by-headsign-at-stop) (stop) ( : route_id (r)
) the store [
clojure.java.io :as io] ) ; and as value the times grouped by destination ( (second) (%) )))) ( (routes-passing-by-stop-ids) () (second) (%) ) (store [r (io/reader f)] )) ))) (
apply merge-with [
f mapper] into
)) then create our
load-gtfs
method to build it once the parsing of the GTFS files has terminated: () defn (load-gtfs) "Load the GTFS data, and make a store of it."
[
stop route-id store]
( (let ( (let [
r]) (trips) ))
: stop-times (stop-times
: stop-times-by-stop ( apply merge-with into ( (map) (#) ( hash-map ( : stop_id %
) [
path]) (stop-times)
)}) ( (assoc) (store) : data ([
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] make-api struct (store) ) ))) ; ( (time) ( def [s/ALL (s/selected? prop #(=% value))] [f mapper] (store) (load-gtfs) "/ tmp / gtfs" ))) ; Loading /tmp/gtfs/routes.txt... ; Loading /tmp/gtfs/trips.txt... ; Loading /tmp/gtfs/stop_times.txt... ; Loading /tmp/gtfs/stops.txt... ; Loading /tmp/gtfs/agency.txt... ; "Elapsed time: [clojure.string :as str] msecs "
;=> # 'user / store
( (keys [
s/ALL (s/selected? prop #(=% value))] (store) ) ;=> (: trips-by-route: routes: stops: stop-times-by-stop: trips-map: stop-times: trips: routes-map: agencies: data) ( (keys [s/ALL (s/selected? prop #(=% value))] ( : data (store) ) ;=>
; "" PREFECTURE " ; "SERRET" ; "LECLERC" ; "P R ROCHAMBEAU" ; "PUYMAIGRE" ; "GENDARMERIE" ; "LA MAXE" ; ... ( (count [
prop value records] ( (keys) [f mapper] ([number stop-id arrival-time headsign day] : data (store)
))) ;=>
( (count [
prop value records] ( (distinct) [f mapper] (["269041-HIV1920-DIM_HIV-Dimanche-00" "281460-HIV1920-SEM_GTFS-Semaine-00" "268643-HIV1920-H1920SAM-Samedi-00"] map : name (stops ))) ;=>
Awesome, and seems ok performance wise. Now how do we use that structure I can hear you saying? It's just a matter of (get-in [
s/ALL (s/selected? :stop-id #(contains? republique-stops %))] the actual structure. We'll get to that in the next section. (REST API) We are going to build a simple API with compojure, I'm going to name it monmet (my Met ', Le Met' being the name of the bus network in Metz). (Init the project) Stop your REPL, modify the (deps.edn) file to put this content in it. [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] ({the paths : deps ({ org.clojure / clojure ({ : mvn / version (1). 22. 1 " }} (ring) ({: mvn / version (1.7.1) }} (ring / ring-core) ({ : mvn / version (“1.7.1”)
} (ring / ring-json) ({ : mvn / version ("0.5.0")
} (ring / ring-defaults) ({ : mvn / version (0.3.2) } (ring / ring-jetty-adapter) ({ : mvn / version (“1.7.1”) () (com.rpl / specter) ({ : mvn / version [
clojure.string :as str]. 2 }
(mount) ({: mvn / version (0.1.) }} (compojure) ({: mvn / version (1.6.1) }}} () and create the needed directories:
$ mkdir -p src / monmet $ (cd) src / monmet $ touch api.clj data.clj gtfs.clj handler.clj main.cl (Loading [s/ALL (s/selected? :stop-id #(=% stop-id))] In order to load one time our datastore (all the GTFS files) we're going to use mount. It's a very simple library, an alternative to components.
Let's do so by modifying the (data.clj) file with the following code:
() (ns) (monmet.data) ( : require ()
["resources" "src"]))
( (mnt / defstate) (store) : start () ([
s/ALL (s/selected? :stop-id #(=% stop-id))] gtfs / load-gtfs “/ tmp / gtfs” )) ( defn store-has-entry? "Check if the given entry is a valid one in our datastore." ( (contains? () # {{: agencies : routes : stops : trips : stop-times
} (entry) )) Here we have defined a state named store that will on call to (mnt / start) (invoke) (gtfs / load-gtfs "/ tmp / gtfs") which returns a map holding the various GTFS files content. It will be our datastore and it will be available in every namespace of our own, and it is guaranteed to be loaded only one time since it's an expensive operation. Routes
We will be using a ring and compojure to create the simple REST API we need.
The skeleton looks like this:
() (ns) (monmet.handler) ( : require () ]) [
s/ALL (s/selected? :stop-id #(=% stop-id))] ["resources" "src"]))
( (defroutes) (app-routes) ( (GET) "/ hello"
( (response) [
:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] {
: foo "bar" })) ( (route / not-found [
record line] (“Not Found”) ) ( def (app) ( (do [s/ALL (s/selected? prop #(=% value))] ( (mnt / start [com.rpl.specter :as s] )
; mount up the datastore ( -> () ( handler / site () (app-routes) () ( (middleware / wrap-json-body) ({
: keywords? (true) })
; transform fields to clojure keywords (middleware / wrap-json-response [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] ))) (GET / gtfs /: entry) On
GET (to) / gts /: entry where [
s/ALL (s/selected? prop #(=% value))] : entry is one of : agencies , : routes , : stops , : trips , : stop-times we will return what's inside the GTFS file directly, we're just serving the CSV files as JSON.
We'll need two methods of our (data) (namespace: [
number stop-id arrival-time headsign day] (store) which is our datastore holding the precious data, and
store-has-entry? which is a little helper to know if the requested entry really exists. when we are done it looks like this: () (ns) (monmet.handler) ( : require () ]) [
s/ALL (s/selected? :stop-id #(=% stop-id))]
])) ( (defroutes) (app-routes) ; Expose GTFS files directly, just for fun:) ( (GET) "/ gtfs /: file" [
f mapper]
( (let ; transform URL path to keyword ( (if [
s/ALL (s/selected? prop #(=% value))] ( (store-has-entry? [f mapper] (entry) )
( (response) ( (entry) [
f mapper] (store) )
; in case the entry exist, just serve what's in the datastore ( (route / not-found [
record line] (“Not Found”) ))) ; otherwhise, ( (route / not-found [record line] (“Not Found”) ) ( def (app) ( (do [s/ALL (s/selected? prop #(=% value))] ( (mnt / start [com.rpl.specter :as s] )
; mount up the datastore ( -> () ( handler / site () (app-routes) () ( (middleware / wrap-json-body) ({
: keywords? (true) }) (middleware / wrap-json-response [
s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] ))) Test it works:
Run the program with
clj -m monmet.main and use curl
$ curl http: // localhost: / gtfs / agencies
(GET / api / ... (bus routes) To make things clearer we ' ll create another namespace called (api) where we will really need our [headsign] (gtfs) (namespace to do some work.) Let's start with routes. I'd like to return the content of (routes.txt) but enhanced with the [s/ALL (s/selected? :stop-id #(=% stop-id))] Agency [f mapper] corresponding to the : agency_id (field of each) (Route) . () (ns) (monmet.api) ( : require () [store {:agencies (load-records (str path "/agency.txt") ->Agency) :routes routes :routes-map (into {} (map #(vector (:route_id %) %) routes)) :stops (load-records (str path "/stops.txt") ->Stop) :trips trips :trips-map (into {} (map #(vector (:trip_id %) %) trips)) :trips-by-route (apply merge-with into (map #(hash-map (:route_id %) [%]))
( defn (get-routes) “Retrieve routes.” ( : routes () store
) [
f mapper] We just need to expose that very api / get-routes in our compojure defroutes : () defroutes (app-routes) ; ... ( (GET) "/ api / routes" () (response) ( (api / get-routes) (store) ))) ; ... ( (route / not-found [record line] (“Not Found”) ) That's it, test it using curl (or even your browser):
$ curl -X GET http: // localhost: 5052 / api / routes Bus stops [clojure.string :as str] Now that our bus routes are returned, we really need some bus stops or we won't be able to enter the damn bus! Get back to the (api) namespace in order to create two methods, one to retrieve the distinct bus stops sorted by name and one to retrieve all the bus stops matching a name.
() (ns) (monmet.api) ( : require () [store {:agencies (load-records (str path "/agency.txt") ->Agency) :routes routes :routes-map (into {} (map #(vector (:route_id %) %) routes)) :stops (load-records (str path "/stops.txt") ->Stop) :trips trips :trips-map (into {} (map #(vector (:trip_id %) %) trips)) :trips-by-route (apply merge-with into (map #(hash-map (:route_id %) [%]))
; ... ( defn (get-stops) "Retrieve the distinct stop names."
( (gtfs / distinct-stop-names-with-ids (store)
)) ( defn (get-stop-with-name) "Retrieve the stops matching (strict equality) a name." (
gtfs / stops-with-name [
clojure.java.io :as io] (stop-name) ( : stops store [com.rpl.specter :as s] ()) Expose them in the API:
() (GET)
“/ / api / stops”
[
s/ALL (s/selected? :route-id #(=% route-id))] () (response) () api / get-stops
store )))
( (GET) "/ api / stops /: name" [
name] ( (response) ( (api / get-stop-with-name) (name) (store)
)))
test it:
$ curl -X GET http: // localhost: 5052 / api / stops | (jq) ('.') ] , [f mapper] ] , ... ] $ curl -X GET http: // localhost: 8236 / api / stops / PIERNE
working from the first try. It's not even funny! Routes passing by a bus stop I said earlier that I ' d like the user to first select the stop, then the route, so we need to propose an endpoint to do that, and it will be bound to / api / stops /: name / routes . This endpoint needs to return the different routes passing by the specified stops, and also for each of these route return the associated possible headsigns. () (ns) (monmet.api) ( : require () [
store {:agencies (load-records (str path "/agency.txt") ->Agency) :routes routes :routes-map (into {} (map #(vector (:route_id %) %) routes)) :stops (load-records (str path "/stops.txt") ->Stop) :trips trips :trips-map (into {} (map #(vector (:trip_id %) %) trips)) :trips-by-route (apply merge-with into (map #(hash-map (:route_id %) [%]))
; ... ( defn (direction-names-for-stop-and-route
( (sort) ( filter [
f mapper] (not-empty) ( (flatten) ( (map keys [s/ALL (s/selected? prop #(=% value))] ([s/ALL (s/selected? :stop-id #(=% "PIERNE01")) (s/selected? :arrival-time #(=1 (compare % "17:41:00"))) (s/selected? :trip-id #(trip-in-route? % "L5a - MAISON NEUVE"))] (get-in) (store) [f mapper] )))))) ( defn route-names-and-directions-for-stop ( (map [s/ALL (s/selected? prop #(=% value))] (#) [f mapper] hash-map (%) () ( (direction-names-for-stop-and-route (stop) (%) [f mapper] store
) ( (route-names-for-stop) (stop) (store [
record line]
))) ; ... [
"05:19:05 L5a - MAISON NEUVE", "05:43:05 L5a - MAISON NEUVE", "06:05:32 L5a - MAISON NEUVE", "06:22:32 L5a - MAISON NEUVE",
Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] Expose this in the REST API : () (GET) "/ api / stops /: name / routes"
[com.rpl.specter :as s] ( (response) ( (api / route-names-and-directions-for-stop) name (store [headsign] ))) Test it's working: $ curl -X GET http: // localhost: 5052 / api / stops / REPUBLIQUE / routes | (jq) '.' , "" route ": ({ "route_id" : 5 ", "route_short_name" : (5) , "route_long_name" : “Ligne 5” , "route_desc" : [
f mapper] , "route_type" : (3) , "route_url" : [f mapper] } } , ({[s/ALL (s/selected? :stop-id #(contains? republique-stops %))] ("name") : “Proxis” , ("directions") : , "" route ": ({ "route_id" : [f mapper] 127 , "route_short_name" : [f mapper] , "route_long_name" : “Proxis” , "route_desc" : [f mapper] , "route_type" : (3) , "route_url" : [f mapper] } } ] [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] (Working as expected:) Still we miss one piece of the puzzle, the time table. (Times for a trip at a bus stop) [result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] Final piece of our API, we need to retrieve the times of day when a bus stops at our favorite bus stop. Besides, we want to query for multiple route direction at the same stop, because sometimes you just want to go somewhere and there are multiple headsigns that goes where you want to go, so you don't care what bus you take. Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] Expose this in the REST API : () (GET) "/ / api / stops /: name / routes /: routes" [
s/ALL (s/selected? prop #(=% value))] ( (response) ( (api / times-for-stop-and-multiple-route-direction) (name) routes () store )) () The format I decided is the following Route ::: Headsign , multiple routes / headsign can be separated by ::: . Let's encode some URI using this useful encoder that I just found on Google: (http://meyerweb.com/eric/tools/dencoder/) [headsign] (Ligne 5)=(Ligne%) (L5a - MAISON NEUVE)=(L5a%) -% (MAISON%) NEUVE (L5 - FORT MOSELLE)=(L5%) -% (FORT%) (MOSELLE) [
result (times-for-stop-and-multiple-route-direction stop routes-direction store) next-3 (map make-sentence (take 3 result))] The URL to call is: (http: // localhost:) / api / stops / PIERNE / routes / Ligne% ::: L5a% -% (MAISON%) NEUVE — Ligne% 491 ::: L5% (%) (FORT%) (MOSELLE) ($ curl -X GET) 'http: // localhost: 98264 / api / stops / PIERNE / routes / Ligne% 823 ::: L5a% -% (MAISON%) NEUVE --- Ligne% 491 ::: L5% 41 -% (FORT%) (MOSELLE ') | (jq) 'map (.arrival_time ' ' .headsign)' [f mapper] | head -3 }]
( (str) (headsign) “arivant a” [com.rpl.specter :as s] ([s/ALL (s/selected? prop #(=% value))] [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] str / replace ([s/ALL (s/selected? prop #(=% value))] [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] subs
(arrival_time) (0) (5)
) [
headsign] [f mapper] # (): "heures"
) “.” )) ( defn (times-for-stop-and-multiple-route-direction-spelled) ( (let () ( (str) "Les 3 prochains bus a l'arret " " (stop) "sont les suivants. n n-" [
com.rpl.specter :as s] apply str [f mapper] (interpose) " n-" (next-3) ))))
; ... and the route:
() (GET)
"/ / api / stops /: name / routes /: routes / spell"
( (response) ( api / times-for-stop-and-multiple-route-direction-spelled (name) (routes) (store) )) then run the server again, and launch ngrok: ($ clj -m monmet.main) & $ ngrok http ngrok by @inconshreveable () (Ctrl C to quit) [:trip-id :arrival-time :departure-time :stop-id :stop-sequence :pickup-type :drop-off-type :shape-dist-traveled] Session Status online Session Expires (7) hours, [f mapper]
minutes Version (2) 3. Region United States ([
s/ALL (s/selected? :stop-id #(contains? republique-stops %))] (us) ) Web Interface http: // 250. 0.0.1: Forwarding http: //
b.ngrok.io -> http: // localhost: Forwarding https: //
b.ngrok.io -> http: // localhost: then just create a shortcut in your iPhone like this: [f mapper] [f mapper] And finally, ask Siri for it:
[
f mapper] BAM. Done. I have now three different shortcut for when I need to go to the train station or city center, or when I'm in the city and wan't to get back home. [ "05:19:05 L5a - MAISON NEUVE", "05:43:05 L5a - MAISON NEUVE", "06:05:32 L5a - MAISON NEUVE", "06:22:32 L5a - MAISON NEUVE",
Exciting isn’t it? These are all the bus I can take every morning to get to the train station :).
Can you spell it?
At first I wanted to create an iOS app for displaying it, but the iPhone’s Shortcuts app can just fetch the URL and spell it.
Just duplicate the code for the last route we created, add /spell to the URL and then just generate a French sentence fo it.
(ns monmet.api ...); ...(defn make-sentence [{:keys [headsign arrival_time] Works from anywhere, every day . I just need to download the GTFS files from time to time (two times per year). (All this with under lines of badly written, non optimized Clojure 🤠. (Until next time 🤘!) (Read More)
; ; 
; ; ; 01575879 ;
GIPHY App Key not set. Please check settings