A MoonBit multi-table CSV dataset integrity checker with declarative profiles, field rules, JSON reports, and a built-in GTFS Schedule profile.
moon update
moon add 2515050242/wayfarer@0.1.0moon update
moon run cmd/csv-check examples/course-catalog-profile.json examples/course-catalogWayfarer: checking CSV profile in examples/course-catalog
PASS: 2 CSV tables, 4 records, 0 findingsmoon run cmd/csv-check examples/course-catalog-profile.json examples/course-catalog-invalidERROR courses.csv:2 capacity="many" [invalid_integer_value] value must be an optionally signed integer
ERROR courses.csv:3 start_date="20230229" [invalid_date_value] value must be a valid Gregorian date in YYYYMMDD format
ERROR courses.csv:3 level="expert" [value_not_allowed] value is not one of the allowed values declared in the profile
ERROR sessions.csv:3 seats="1.5.2" [invalid_number_value] value must be an integer or decimal number
ERROR sessions.csv:3 course_id="C999" [unresolved_reference] referenced value does not exist in courses.csv
FAIL: 2 CSV tables, 4 records, 5 findings| Table rule | Meaning |
|---|---|
| required | the file must be present |
| minimum_rows | the table must contain at least this many data records |
| required_columns | these headers must exist |
| non_empty_columns | these fields must be non-empty in every record |
| unique_keys | each listed column combination is unique within the table |
| field_rules | per-column value rules (below) |
| type | Accepted values |
|---|---|
| integer | an optionally signed run of digits |
| number | an optionally signed integer or decimal number without exponent |
| date | a valid Gregorian date in YYYYMMDD format |
| enum | one of the values listed in allowed |
moon run cmd/csv-check examples/course-catalog-profile.json examples/course-catalog-invalid --jsonmoon run cmd/csv-check examples/course-catalog-profile.json examples/course-catalog-malformed --json| Table | Unconditionally required headers checked |
|---|---|
| agency.txt | agency_name, agency_url, agency_timezone |
| stops.txt | stop_id |
| routes.txt | route_id, route_type |
| trips.txt | route_id, service_id, trip_id |
| stop_times.txt | trip_id, stop_sequence |
| calendar.txt | service_id, weekday flags, start_date, end_date |
| calendar_dates.txt | service_id, date, exception_type |
moon update
moon run cmd/main examples/validWayfarer: checking examples/valid
PASS (supported checks only): 7 parsed files, 9 records, 0 findingsmoon run cmd/main examples/invalidERROR calendar_dates.txt:3 service_id,date="WK,20261031" [duplicate_primary_key] service_id and date pair is already used in this table
ERROR trips.txt:2 route_id="UNKNOWN" [unresolved_reference] referenced ID does not exist in routes.txt
ERROR trips.txt:2 service_id="MISSING" [unresolved_reference] referenced ID does not exist in calendar.txt or calendar_dates.txt
ERROR stop_times.txt:2 arrival_time="" [missing_required_time] arrival_time is required at the first and last stop of a trip
ERROR stop_times.txt:2 departure_time="25:61:00" [invalid_time] time must use H:MM:SS or HH:MM:SS with minutes and seconds from 00 to 59
ERROR stop_times.txt:4 arrival_time="25:00:00" [time_order_violation] time precedes an earlier defined time in this trip's stop_sequence order
FAIL: 6 parsed files, 9 records, 6 findingsmoon run cmd/main examples/header-only-required-tableERROR agency.txt:1 [empty_required_table] required table must contain at least one data record
FAIL: 6 parsed files, 7 records, 1 findingmoon run cmd/main examples/single-stop-tripERROR trips.txt:2 trip_id="T1" [insufficient_stop_times] trip must have at least two stop_times records
FAIL: 6 parsed files, 6 records, 1 findingmoon run cmd/main examples/field-value-errorsERROR stops.txt:2 stop_name="Platform \"East\"\tA" [forbidden_field_character] GTFS field values must not contain tabs, carriage returns, or newlines
ERROR stops.txt:3 stop_name="North\nAnnex" [forbidden_field_character] GTFS field values must not contain tabs, carriage returns, or newlines
ERROR stops.txt:5 stop_name="South\tGate" [forbidden_field_character] GTFS field values must not contain tabs, carriage returns, or newlines
FAIL: 6 parsed files, 9 records, 3 findingsmoon run cmd/main examples/control-character-diagnostic
ERROR trips.txt:2 service_id="WK\u001b[31m" [unresolved_reference] referenced ID does not exist in calendar.txt or calendar_dates.txt
FAIL: 6 parsed files, 8 records, 1 findingmoon run cmd/main examples/no-service-calendarWayfarer: checking examples/no-service-calendar
ERROR calendar.txt/calendar_dates.txt [missing_service_calendar] at least one service calendar file is required
FAIL: 5 parsed files, 7 records, 1 findingmoon run cmd/main examples/stop-time-conditional-errorsERROR stop_times.txt:2 timepoint="2" [invalid_enum_value] timepoint must be empty, 0, or 1
ERROR stop_times.txt:2 arrival_time="09:00:00" [forbidden_with_pickup_drop_off_window] arrival_time must be empty when a pickup/drop-off window is defined
ERROR stop_times.txt:2 departure_time="09:01:00" [forbidden_with_pickup_drop_off_window] departure_time must be empty when a pickup/drop-off window is defined
ERROR stop_times.txt:2 pickup_type="0" [forbidden_with_pickup_drop_off_window] pickup_type must be 1 or 2 when a pickup/drop-off window is defined
ERROR stop_times.txt:2 drop_off_type="0" [forbidden_with_pickup_drop_off_window] drop_off_type must not be empty or 0 when a pickup/drop-off window is defined
ERROR stop_times.txt:2 continuous_pickup="2" [forbidden_with_pickup_drop_off_window] continuous_pickup must be 1 or empty when a pickup/drop-off window is defined
ERROR stop_times.txt:2 continuous_drop_off="0" [forbidden_with_pickup_drop_off_window] continuous_drop_off must be 1 or empty when a pickup/drop-off window is defined
ERROR routes.txt:2 continuous_pickup="2" [forbidden_with_pickup_drop_off_window] continuous_pickup must be 1 or empty when a trip on this route defines a pickup/drop-off window
ERROR routes.txt:2 continuous_drop_off="0" [forbidden_with_pickup_drop_off_window] continuous_drop_off must be 1 or empty when a trip on this route defines a pickup/drop-off window
FAIL: 7 parsed files, 9 records, 9 findingsmoon run cmd/main examples/calendar-semantics-errorsERROR calendar.txt:2 monday="2" [invalid_enum_value] monday must be 0 or 1
ERROR calendar.txt:2 end_date="20260101" [invalid_date_range] end_date must not precede start_date
ERROR calendar_dates.txt:2 date="20230229" [invalid_date] date must be a valid Gregorian date in YYYYMMDD format
ERROR calendar_dates.txt:2 exception_type="3" [invalid_enum_value] exception_type must be 1 or 2
FAIL: 7 parsed files, 9 records, 4 findingsmoon run cmd/main examples/unsupported-location-profileERROR locations.geojson [unsupported_location_profile] location-based on-demand data is outside the supported fixed-stop profile
ERROR stop_times.txt:2 location_id="ZONE-1" [unsupported_location_profile] flexible-location records using location_group_id or location_id are outside the supported fixed-stop profile
FAIL: 6 parsed files, 7 records, 2 findingsmoon run cmd/csv-check examples/inventory-profile.json examples/inventory-profileWayfarer: checking CSV profile in examples/inventory-profile
PASS: 2 CSV tables, 4 records, 0 findingsmoon run cmd/csv-check examples/inventory-profile.json examples/inventory-profile-invalidERROR orders.csv:3 product_id="P9" [unresolved_reference] referenced value does not exist in products.csv
FAIL: 2 CSV tables, 4 records, 1 findingmoon run cmd/profile-demo examples/inventory-profilePASS: 2 CSV tables, 4 records, 0 findingsmoon run cmd/profile-demo examples/inventory-profile-invalidERROR orders.csv:3 product_id [unresolved_reference] referenced value does not exist in products.csv
FAIL: 2 CSV tables, 4 records, 1 findingmoon run cmd/csv-check examples/enrollment-profile.json examples/enrollment-profilePASS: 3 CSV tables, 7 records, 0 findingsmoon run cmd/csv-check examples/enrollment-profile.json examples/enrollment-profile-invalidERROR enrollments.csv:4 learner_id,class_id="L1,C1" [duplicate_key] key value is already used in this table
ERROR enrollments.csv:2 class_id="C9" [unresolved_reference] referenced value does not exist in classes.csv
FAIL: 3 CSV tables, 7 records, 2 findingsmoon fmt
moon info
moon check
moon build
moon testpub(all) enum CsvDatasetFieldKind {
IntegerValue
NumberValue
DateValue
AllowedValues
}pub(all) struct CsvDatasetFieldRule {
column : String
kind : CsvDatasetFieldKind
allowed : Array[String]
}fn CsvDatasetFieldRule::new(column : String, kind : CsvDatasetFieldKind, allowed? : Array[String]) -> CsvDatasetFieldRulepub(all) struct CsvDatasetIssue {
code : String
file : String
line : Int?
column : String
value : String
message : String
}pub(all) struct CsvDatasetProfile {
tables : Array[CsvDatasetTableRule]
references : Array[CsvDatasetReferenceRule]
}fn CsvDatasetProfile::new(tables : Array[CsvDatasetTableRule], references : Array[CsvDatasetReferenceRule]) -> CsvDatasetProfilepub(all) struct CsvDatasetReferenceRule {
source_file : String
source_column : String
target_file : String
target_column : String
required : Bool
}fn CsvDatasetReferenceRule::new(source_file : String, source_column : String, target_file : String, target_column : String, required : Bool) -> CsvDatasetReferenceRulefn CsvDatasetTableRule::new(file : String, required : Bool, minimum_rows : Int, required_columns : Array[String], non_empty_columns : Array[String], unique_keys : Array[Array[String]], field_rules? : Array[CsvDatasetFieldRule]) -> CsvDatasetTableRulepub enum CsvError {
EmptyDocument
EmptyHeader(line~ : Int, column~ : Int)
DuplicateHeader(name~ : String, column~ : Int)
UnexpectedQuote(line~ : Int, column~ : Int)
CharactersAfterQuote(line~ : Int, column~ : Int)
UnterminatedQuote(line~ : Int, column~ : Int)
RowWidthMismatch(line~ : Int, expected~ : Int, actual~ : Int)
InvalidLineEnding(line~ : Int, column~ : Int)
}pub struct FeedIssue {
code : String
file : String
line : Int
column : String
value : String
message : String
}pub struct FieldValueIssue {
code : String
file : String
line : Int
column : String
value : String
message : String
}fn GtfsTables::new(stops : CsvTable?, routes : CsvTable?, trips : CsvTable?, stop_times : CsvTable?) -> GtfsTablesfn GtfsTables::with_services(stops : CsvTable?, routes : CsvTable?, trips : CsvTable?, stop_times : CsvTable?, calendar : CsvTable?, calendar_dates : CsvTable?) -> GtfsTablespub struct SchemaIssue {
code : String
table : String
column : String
message : String
}fn validate_csv_dataset(profile : CsvDatasetProfile, tables : Map[String, CsvTable]) -> Array[CsvDatasetIssue]Install
Download zipA MoonBit multi-table CSV dataset integrity checker with declarative profiles, field rules, JSON reports, and a built-in GTFS Schedule profile.