- Publicity: Public Only All
github-activity-monitor-procs.tcl
GitHub Activity Monitor - main library classes and objects scp packages/xowiki/tcl/github-activity-monitor-*.tcl gustafn@openacs.org:/var/www/openacs.org/packages/xowiki/tcl/ scp www/github-activities.{tcl,adp} gustafn@openacs.org:/var/www/openacs.org/www/
This file defines the following Objects and Classes: ::github::ActivityMonitor
, ::github::am
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
, ::github::ActivityMonitor
- Location:
- packages/xowiki/tcl/github-activity-monitor-procs.tcl
- Created:
- 2026-11-20
- Author:
- Gustaf Neumann
Procedures in this file
- Class ::github::ActivityMonitor (public)
- Object ::github::am (public)
- github::ActivityMonitor instproc backfill_repo_history (public)
- github::ActivityMonitor instproc get (public)
- github::ActivityMonitor instproc status (public)
- github::ActivityMonitor instproc summarize_push_event (public)
- github::ActivityMonitor instproc sync_from_github (public)
Detailed information
Class ::github::ActivityMonitor (public)
::nx::Class ::github::ActivityMonitor
- Testcases:
- No testcase defined.
github::ActivityMonitor method backfill_repo_history (public)
<instance of github::ActivityMonitor> backfill_repo_history args \ [ args... ]
- Parameters:
- args (required)
- Testcases:
- No testcase defined.
github::ActivityMonitor method get (public)
<instance of github::ActivityMonitor> get path [ query_args ]
Issue an API call on GitHub for the configured organization.
- Parameters:
- path (required)
- query_args (optional)
- Testcases:
- No testcase defined.
github::ActivityMonitor method status (public)
<instance of github::ActivityMonitor> status
- Testcases:
- No testcase defined.
github::ActivityMonitor method summarize_push_event (public)
<instance of github::ActivityMonitor> summarize_push_event ev
Fetch commit details and build a summary for PushEvents.
- Parameters:
- ev (required)
- Returns:
- dict with fields of push event
- Testcases:
- No testcase defined.
github::ActivityMonitor method sync_from_github (public)
<instance of github::ActivityMonitor> sync_from_github
An xact-scoped lock stays on the transaction's checked-out DB connection, is shared across threads/processes, and is released automatically on commit, rollback or connection loss.
- Testcases:
- No testcase defined.
Content File Source
::xo::library doc { GitHub Activity Monitor - main library classes and objects scp packages/xowiki/tcl/github-activity-monitor-*.tcl gustafn@openacs.org:/var/www/openacs.org/packages/xowiki/tcl/ scp www/github-activities.{tcl,adp} gustafn@openacs.org:/var/www/openacs.org/www/ @creation-date 2026-11-20 @author Gustaf Neumann } if {$::tcl_version < 8.6} {package require try} namespace eval ::github { # # Data model # ::xo::db::require table github_activity [subst { event_id {bigint primary key} repo {[::xo::dc map_datatype text] not null} branch {[::xo::dc map_datatype text]} sha {[::xo::dc map_datatype text] not null} author_name {[::xo::dc map_datatype text]} author_login {[::xo::dc map_datatype text]} commit_date {timestamptz not null} created_at {timestamptz not null} title {[::xo::dc map_datatype text] not null} url {[::xo::dc map_datatype text] not null} files_changed {integer} additions {integer} deletions {integer} raw_payload {[::xo::dc map_datatype text] not null} }] #::xo::db::require index -table github_activity -col {commit_date desc} ::xo::db::require index -table github_activity -col {commit_date} ::xo::db::require index -table github_activity -col repo,sha -unique true ::xo::db::require table github_activity_state [subst { name {[::xo::dc map_datatype text] primary key} value {[::xo::dc map_datatype text] not null} }] # # Class ActivityMonitor # ::nx::Class create ActivityMonitor { :property {organization OpenACS} :property {api_base "https://api.github.com"} :property {api_token} # Checked by each scheduled sync; no additional scheduler is needed. :property {reconcile_interval 3600} :property {backfill_pages_per_run 2} :property {max_requests_per_run 200} :property {max_seconds_per_run 120} :property {discovery_requests_per_run 20} :property {recent_days 180} :property {repository_metadata_pages 4} :property {fast_repository_checks_per_run 8} :property {hot_repository_days 2} # Exact branch names or Tcl glob patterns; empty means no branches. :property {allowed_branches {main oacs-5-10}} #--- get ------------------------------------------------------------------------- :public method get {path {query_args {}}} { # # Issue an API call on GitHub for the configured # organization. # :check_rate_limit :charge_request set url "${:api_base}${path}" # default query arg dict set query per_page 100 foreach {k v} $query_args { dict set query $k $v } set urlDict [ns_parseurl $url] set encoded {} foreach {k v} $query { lappend encoded "[ns_urlencode -part query $k]=[ns_urlencode -part query $v]" } dict set urlDict query [join $encoded &] set headers [ns_set create headers \ Authorization "Bearer ${:api_token}" \ Accept "application/vnd.github+json" \ User-Agent "openacs-activity-dashboard/1.0" \ ] # Do not log the Authorization header. ns_log notice "GitHub GET [ns_joinurl $urlDict]" set timeout 15 if {[info exists :deadline]} { set timeout [expr {min(15, max(1, ${:deadline} - [clock seconds]))}] } set response_headers "" try { set result [ns_http run -timeout $timeout -method GET -headers $headers [ns_joinurl $urlDict]] set status [dict get $result status] set body [dict get $result body] set rate_headers {} if {[dict exists $result headers]} { set response_headers [dict get $result headers] foreach name {x-ratelimit-remaining x-ratelimit-reset retry-after} { dict set rate_headers $name [ns_set iget $response_headers $name] } } :record_rate_limit $status $rate_headers $body if {$status < 200 || $status >= 300} { ns_log Error "GitHub GET $url failed: status $status, body: $body" return -code error -errorcode [list GITHUB HTTP $status] "GitHub API error $status" } return $body } finally { ns_set free $headers if {$response_headers ne ""} {ns_set free $response_headers} } } :method check_rate_limit {} { set until [:get_state github_not_before 0] if {[string is wideinteger -strict $until] && $until > [clock seconds]} { set when [clock format $until -gmt 1 -format {%Y-%m-%d %H:%M:%S UTC}] return -code error -errorcode {GITHUB RATE_LIMIT} "GitHub requests paused until $when" } } :method record_rate_limit {status headers body} { set now [clock seconds] set remaining "" set reset "" set retry "" foreach {name variable} { x-ratelimit-remaining remaining x-ratelimit-reset reset retry-after retry } { if {[dict exists $headers $name]} { set $variable [string trim [dict get $headers $name]] } } set exhausted [expr {$remaining eq "0"}] set limited [expr {$exhausted || $status == 429 || ($status == 403 && ($retry ne "" || [regexp -nocase {rate limit|secondary limit|abuse detection} $body]))}] if {!$limited} { if {$status >= 200 && $status < 300 && [:get_state github_rate_backoff 0] ne "0"} { :set_state github_rate_backoff 0 } return } set until 0 if {$exhausted && [string is wideinteger -strict $reset] && $reset > $now} { set until [expr {$reset + 2}] } if {[string is integer -strict $retry] && $retry >= 0} { set until [expr {max($until, $now + $retry + 2)}] } elseif {$retry ne "" && ![catch {clock scan $retry -gmt 1} retry_epoch]} { set until [expr {max($until, $retry_epoch + 2)}] } # Missing/invalid headers, or repeated secondary limits: at least # one minute, doubling on repeated failures up to one hour. if {$until <= $now} { set previous [:get_state github_rate_backoff 0] if {![string is integer -strict $previous] || $previous < 0} {set previous 0} set delay [expr {min(3600, max(60, 2 * $previous))}] set until [expr {$now + $delay}] :set_state github_rate_backoff $delay } set previous [:get_state github_not_before 0] if {[string is wideinteger -strict $previous]} {set until [expr {max($until, $previous)}]} :set_state github_not_before $until ns_log Warning "GitHub: requests paused until [clock format $until -gmt 1 -format {%Y-%m-%d %H:%M:%S UTC}]" } #--- accessing information from activity state ----------------------------------- :method get_state {name {default ""}} { return [xo::dc get_value get_github_activity_state { select value from github_activity_state where name = :name } $default] } :method set_state {name value} { # ns_pg_bind maps an empty Tcl value to SQL NULL. Empty queues # and discovery cursors must be stored as empty text, not NULL. xo::dc dml set_github_activity_state { insert into github_activity_state(name, value) values (:name, coalesce(:value, '')) on conflict (name) do update set value = excluded.value } } #--- fetch_new_events ------------------------------------------------------------ :method fetch_new_events {} { # # Fetch available events since the last processed ID. # Repository reconciliation independently covers missing events. # # Last event we processed, if any set last_event_id [:get_state last_org_event_id ""] #set last_event_id [xo::dc get_value last_id { # select max(event_id) from github_activity #}] set events {} set page 1 set done 0 while {!$done} { set body [:get "/orgs/${:organization}/events" [list page $page]] set batch [::util::json2dict $body] if {[llength $batch] == 0} { break } foreach ev $batch { set ev_id [dict get $ev id] # Events are newest-first; once we reach an older or equal ID, stop. if {$last_event_id ne "" && $ev_id eq $last_event_id} { set done 1 break } if {$last_event_id ne "" && $ev_id < $last_event_id} { ns_log Warning "GitHub: reached page with a smaller ev_id than last_event_id ($ev_id < $last_event_id)" } lappend events $ev } incr page if {$page > 3} { ns_log Warning "GitHub: reached page limit while fetching events; stopping at page $page" break } if {$done} { break } } # We collected in newest-first order; reverse to oldest-first for insertion. return [lreverse $events] } #--- summarize_push_event -------------------------------------------------------- :public method summarize_push_event {ev} { # # Fetch commit details and build a summary for PushEvents. # # @return dict with fields of push event set type [dict get $ev type] if {$type ne "PushEvent"} { return "" } set repo_full [dict get $ev repo name] ;# e.g. "openacs/openacs-core" set ref [dict get $ev payload ref] ;# "refs/heads/main" if {![string match "refs/heads/*" $ref]} { return "" } set branch [string range $ref [string length "refs/heads/"] end] set head_sha [dict get $ev payload head] set created_at [dict get $ev created_at] set actor_login [dict get $ev actor login] if {$repo_full eq "" || $head_sha eq ""} { return "" } # Fetch commit details set commit_body [:get "/repos/$repo_full/commits/$head_sha"] set c [::util::json2dict $commit_body] set commit [dict get $c commit] set author_dict [dict get $commit author] set author_name [dict get $author_dict name] set commit_date [dict get $author_dict date] ;# ISO8601 string set message_full [dict get $commit message] set title [lindex [split $message_full "\n"] 0] set stats [dict get $c stats] set additions [dict get $stats additions] set deletions [dict get $stats deletions] set files [dict get $c files] set files_changed [llength $files] return [list \ event_id [dict get $ev id] \ repo $repo_full \ branch $branch \ sha [dict get $c sha] \ author_name $author_name \ author_login $actor_login \ commit_date $commit_date \ created_at $created_at \ title $title \ url [dict get $c html_url] \ files_changed $files_changed \ additions $additions \ deletions $deletions \ raw_payload $commit_body \ ] } #--- reconciliation and durable backfill jobs ------------------------------------ :method charge_request {} { if {[info exists :requests_left]} { if {${:requests_left} <= 0 || [clock seconds] >= ${:deadline}} { return -code error -errorcode {GITHUB BUDGET} "sync budget exhausted" } incr :requests_left -1 } } :method recent_since {} { return [clock format [expr {[clock seconds] - ${:recent_days} * 86400}] \ -gmt 1 -format {%Y-%m-%dT%H:%M:%SZ}] } :method observe_branches {jobsVar repo branches {first 1} {last 1}} { upvar 1 $jobsVar jobs set observed [:get_state observed_branch_heads {}] set anomalies [:get_state unexpected_master_heads {}] set scans [:get_state branch_scan_heads {}] if {$first || ![dict exists $scans $repo]} {dict set scans $repo {}} set scan [dict get $scans $repo] foreach item $branches { set branch [dict get $item name] set revision [dict get $item commit sha] if {$branch in {main master}} {dict set scan $branch $revision} if {[:branch_enabled $branch]} { dict set observed [list $repo $branch] $revision :queue_backfill jobs $repo $branch $revision } } dict set scans $repo $scan if {$last} { if {[dict exists $scan master] && (![dict exists $scan main] || [dict get $scan master] ne [dict get $scan main])} { dict set anomalies $repo [dict get $scan master] ns_log Warning "GitHub: unexpected master branch in $repo at [dict get $scan master]; check CVS conversion" } elseif {[dict exists $anomalies $repo]} { dict unset anomalies $repo } dict unset scans $repo } :set_state branch_scan_heads $scans :set_state observed_branch_heads $observed :set_state unexpected_master_heads $anomalies } :method discover {jobsVar heads} { upvar 1 $jobsVar jobs set allowance [expr {min(${:discovery_requests_per_run}, max(2, ${:max_requests_per_run} / 4))}] set stop [expr {${:requests_left} - $allowance}] set :discovery_deadline [expr {[clock seconds] + max(1, ${:max_seconds_per_run} / 2)}] try { # Always prioritize new pushes, independent of the full cursor. :discover_recent jobs $stop if {[:discovery_available $stop]} {:discover_full jobs $heads $stop} } finally { unset :discovery_deadline } } :method discovery_available {stop} { return [expr {${:requests_left} > $stop && [clock seconds] < ${:discovery_deadline}}] } :method discover_recent {jobsVar stop} { upvar 1 $jobsVar jobs set checks [:get_state recent_repository_checks {}] set seen [:get_state repository_push_fingerprints {}] set checked [:get_state repository_checked_at {}] set hot_since [clock format [expr {[clock seconds] - ${:hot_repository_days} * 86400}] -gmt 1 -format {%Y-%m-%dT%H:%M:%SZ}] set next_page [:get_state repository_metadata_page 1] set priority {} set pages 0 set page 1 # Reserve at least half the discovery allowance for branch checks # and the background sweep, even with very small user budgets. set limit [expr {min(${:repository_metadata_pages}, max(1, (${:requests_left} - $stop) / 2))}] set inventory_deadline [expr {[clock seconds] + max(1, ${:max_seconds_per_run} / 4)}] while {$pages < $limit && [:discovery_available $stop] && [clock seconds] < $inventory_deadline} { set batch [::util::json2dict [:get "/orgs/${:organization}/repos" \ [list sort pushed direction desc page $page]]] incr pages foreach repository $batch { set repo [dict get $repository full_name] set fingerprint {} foreach field {id pushed_at default_branch} { lappend fingerprint [expr {[dict exists $repository $field] ? [dict get $repository $field] : ""}] } lappend fingerprint ${:allowed_branches} set hot 0 if {[dict exists $repository pushed_at]} { set pushed [dict get $repository pushed_at] set hot [expr {[regexp {^[0-9]{4}-[0-9]{2}-[0-9]{2}T} $pushed] && [string compare $pushed $hot_since] >= 0 && (![dict exists $checked $repo] || [clock seconds] - [dict get $checked $repo] >= ${:reconcile_interval})}] } if {$hot || ![dict exists $seen $repo] || [dict get $seen $repo] ne $fingerprint} { if {![dict exists $checks $repo] || [dict get $checks $repo fingerprint] ne $fingerprint} { dict set checks $repo [dict create fingerprint $fingerprint page 1] } if {![dict exists $checks $repo attempted_at]} {lappend priority $repo} } } :set_state recent_repository_checks $checks # Page 1 is fetched on EVERY run. Later pages resume when an # organization outgrows the small per-run inventory allowance. if {[llength $batch] < 100} { set next_page 1 :set_state last_repository_inventory [clock seconds] :set_state repository_metadata_page $next_page break } if {$page == 1} {set page [expr {max(2, $next_page)}]} else {incr page} set next_page $page :set_state repository_metadata_page $next_page } set order $priority foreach repo [dict keys $checks] { if {$repo ni $order} {lappend order $repo} } set count 0 foreach repo $order { if {$count >= ${:fast_repository_checks_per_run} || ![:discovery_available $stop]} {break} if {![dict exists $checks $repo]} {continue} incr count set job [dict get $checks $repo] dict set job attempted_at [clock seconds] if {[catch { set branches [::util::json2dict [:get "/repos/$repo/branches" [list page [dict get $job page]]]] } message options]} { # Keep a failing/deleted/inaccessible repository from # blocking every later repository check in this run. dict unset checks $repo dict set checks $repo $job :set_state recent_repository_checks $checks set errors [:get_state repository_check_errors {}] dict set errors $repo $message :set_state repository_check_errors $errors ns_log Warning "GitHub: recent repository check deferred for $repo: $message" if {[dict get $options -errorcode] in {{GITHUB BUDGET} {GITHUB RATE_LIMIT} {GITHUB HTTP 403} {GITHUB HTTP 429}}} { return -options $options $message } continue } :observe_branches jobs $repo $branches [expr {[dict get $job page] == 1}] [expr {[llength $branches] < 100}] set errors [:get_state repository_check_errors {}] if {[dict exists $errors $repo]} {dict unset errors $repo; :set_state repository_check_errors $errors} :set_state pending_backfills $jobs if {[llength $branches] < 100} { dict set seen $repo [dict get $job fingerprint] :set_state repository_push_fingerprints $seen dict set checked $repo [clock seconds] :set_state repository_checked_at $checked dict unset checks $repo } else { dict incr job page dict unset checks $repo dict set checks $repo $job } :set_state recent_repository_checks $checks } } :method discover_full {jobsVar heads stop} { upvar 1 $jobsVar jobs set cursor [:get_state repository_discovery {}] set now [clock seconds] if {$cursor eq ""} { if {$now - [:get_state last_repository_reconcile 0] < ${:reconcile_interval}} { return } set cursor [dict create repo_page 1 repos {} branch_page 1] } while {[:discovery_available $stop]} { set repos [dict get $cursor repos] if {[llength $repos] == 0} { if {[dict exists $cursor repos_done]} { :set_state repository_discovery {} :set_state last_repository_reconcile $now return } set page [dict get $cursor repo_page] set repos [::util::json2dict [:get "/orgs/${:organization}/repos" [list page $page]]] dict incr cursor repo_page dict set cursor repos $repos if {[llength $repos] < 100} {dict set cursor repos_done 1} :set_state repository_discovery $cursor continue } set repo [dict get [lindex $repos 0] full_name] set page [dict get $cursor branch_page] if {[dict exists [:get_state recent_repository_checks {}] $repo]} { dict set cursor repos [lrange $repos 1 end] dict set cursor branch_page 1 :set_state repository_discovery $cursor continue } set branches [::util::json2dict [:get "/repos/$repo/branches" [list page $page]]] :observe_branches jobs $repo $branches [expr {$page == 1}] [expr {[llength $branches] < 100}] if {[llength $branches] < 100} { dict set cursor repos [lrange $repos 1 end] dict set cursor branch_page 1 } else { dict incr cursor branch_page } :set_state pending_backfills $jobs :set_state repository_discovery $cursor } } :method branch_enabled {branch} { if {![info exists :allowed_branches]} { set :allowed_branches {main oacs-5-10} } foreach pattern ${:allowed_branches} { if {[string match $pattern $branch]} {return 1} } return 0 } :method queue_backfill {jobsVar repo branch revision} { upvar 1 $jobsVar jobs if {![:branch_enabled $branch]} {return} set key [list $repo $branch] set heads [:get_state reconciled_branch_heads {}] if {[dict exists $jobs $key]} { set job [dict get $jobs $key] if {[dict get $job revision] ne $revision} { dict set job followup_revision $revision } elseif {[dict exists $job followup_revision]} { dict unset job followup_revision } dict set jobs $key $job return } if {[dict exists $heads $key] && [dict get $heads $key] eq $revision} {return} set base "" if {[dict exists $heads $key]} {set base [dict get $heads $key]} dict set jobs $key [dict create revision $revision next_page 1 base $base since [:recent_since]] } :public method sync_from_github {} { # An xact-scoped lock stays on the transaction's checked-out DB # connection, is shared across threads/processes, and is released # automatically on commit, rollback or connection loss. ::xo::dc transaction { set locked [::xo::dc get_value github_sync_lock { select case when pg_try_advisory_xact_lock(714230, 1) then 1 else 0 end }] if {$locked} { :sync_locked } else { ns_log Notice "GitHub: sync skipped; another activity writer is running" } } } :method sync_locked {} { if {[catch {:check_rate_limit} message]} { ns_log Notice "GitHub: sync skipped; $message" return } foreach {name default} { reconcile_interval 3600 backfill_pages_per_run 2 max_requests_per_run 200 max_seconds_per_run 120 discovery_requests_per_run 20 recent_days 180 repository_metadata_pages 4 fast_repository_checks_per_run 8 hot_repository_days 2 } { if {![info exists :$name]} {set :$name $default} if {![string is integer -strict [set :$name]] || [set :$name] < 1} { error "$name must be a positive integer" } } if {${:max_requests_per_run} < 8 || ${:discovery_requests_per_run} < 2} { error "max_requests_per_run must be at least 8 and discovery_requests_per_run at least 2" } # Initialize and validate a branch policy on pre-existing objects. :branch_enabled "" llength ${:allowed_branches} set :requests_left ${:max_requests_per_run} set :deadline [expr {[clock seconds] + ${:max_seconds_per_run}}] try { :sync_work } finally { unset :requests_left :deadline } } :method sync_work {} { ns_log Notice "GitHub: refresh_activity start" set jobs [:get_state pending_backfills {}] set heads [:get_state reconciled_branch_heads {}] # Convert old unbounded jobs once. Page numbers of the recent # filtered list differ, so restart jobs without a baseline. set removed 0 dict for {key job} $jobs { lassign $key repo branch if {![:branch_enabled $branch]} { dict unset jobs $key incr removed continue } if {![dict exists $job since]} { dict set jobs $key since [:recent_since] if {![dict exists $job base] || [dict get $job base] eq ""} { dict set jobs $key next_page 1 } } } :set_state pending_backfills $jobs if {$removed > 0} { ns_log Notice "GitHub: removed $removed excluded branch job(s) from pending queue" } if {[catch {:discover jobs $heads} message options]} { :set_state discovery_error $message ns_log Warning "GitHub: discovery deferred: $message" } else { :set_state discovery_error "" } set new_events {} if {[catch {set new_events [:fetch_new_events]} errorMsg]} { ns_log Error "GitHub: fetching events failed: $errorMsg" } foreach ev $new_events { if {[dict get $ev type] ne "PushEvent"} { continue } set ref [dict get $ev payload ref] if {![string match "refs/heads/*" $ref]} { continue } set repo [dict get $ev repo name] set branch [string range $ref [string length "refs/heads/"] end] set revision [dict get $ev payload head] # Deleted branches have a zero head, with no history to fetch. if {$revision eq "" || [regexp {^0+$} $revision]} { continue } set observed [:get_state observed_branch_heads {}] set key [list $repo $branch] if {[dict exists $observed $key]} {set revision [dict get $observed $key]} :queue_backfill jobs $repo $branch $revision } # Checkpointing events is safe only after their work is durable. :set_state pending_backfills $jobs if {[llength $new_events] > 0} { :set_state last_org_event_id [dict get [lindex $new_events end] id] } dict for {key job} $jobs { if {${:requests_left} <= 0 || [clock seconds] >= ${:deadline}} {break} lassign $key repo branch set revision [dict get $job revision] # Missing baselines use the job's fixed recent-history window. set base "" if {[dict exists $job base]} {set base [dict get $job base]} if {[catch { set progress [:backfill_locked \ -repo $repo -branch $branch \ -revision $revision -base $base -since [dict get $job since] \ -start_page [dict get $job next_page] \ -max_pages ${:backfill_pages_per_run}] } errorMsg options]} { # Rotate unfinished/failed jobs so one large branch cannot # monopolize the first requests on every invocation. dict unset jobs $key dict set jobs $key $job :set_state pending_backfills $jobs if {[dict get $options -errorcode] in { {GITHUB BUDGET} {GITHUB RATE_LIMIT} {GITHUB HTTP 403} {GITHUB HTTP 429} }} {break} set errors [:get_state backfill_errors {}] dict set errors $key $errorMsg :set_state backfill_errors $errors ns_log Error "GitHub: backfill failed for $repo $branch; retained for retry: $errorMsg" continue } set errors [:get_state backfill_errors {}] if {[dict exists $errors $key]} {dict unset errors $key; :set_state backfill_errors $errors} if {[dict get $progress complete]} { set attention [:get_state attention_required {}] if {[dict exists $attention $key]} {dict unset attention $key; :set_state attention_required $attention} dict set heads $key $revision # Save the successful head before removing its retry job. :set_state reconciled_branch_heads $heads dict unset jobs $key if {[dict exists $job followup_revision]} { :queue_backfill jobs $repo $branch [dict get $job followup_revision] } } else { dict unset jobs $key dict set job next_page [dict get $progress next_page] dict set job base [dict get $progress base] dict set jobs $key $job } :set_state pending_backfills $jobs } set summary [dict create finished_at [clock seconds] pending_branches [dict size $jobs] \ pending_repository_checks [dict size [:get_state recent_repository_checks {}]] \ requests_used [expr {${:max_requests_per_run} - ${:requests_left}}]] :set_state last_sync_status $summary ns_log Notice "GitHub: refresh_activity done; $summary" } :public method status {} { set result [dict create pending_branches [dict size [:get_state pending_backfills {}]] \ pending_repository_checks [dict size [:get_state recent_repository_checks {}]] \ inventory_page [:get_state repository_metadata_page 1] \ last_inventory [:get_state last_repository_inventory 0] \ last_full_reconcile [:get_state last_repository_reconcile 0] \ github_not_before [:get_state github_not_before 0] \ unexpected_master_heads [:get_state unexpected_master_heads {}] \ attention_required [:get_state attention_required {}] \ discovery_error [:get_state discovery_error ""] \ repository_errors [:get_state repository_check_errors {}] \ backfill_errors [:get_state backfill_errors {}] \ last_sync [:get_state last_sync_status {}]] if {[dict get $result github_not_before] > [clock seconds]} { dict set result state paused } elseif {[dict get $result discovery_error] ne "" || [dict size [dict get $result unexpected_master_heads]] || [dict size [dict get $result attention_required]] || [dict size [dict get $result repository_errors]] || [dict size [dict get $result backfill_errors]]} { dict set result state needs-attention } elseif {[dict get $result pending_branches] || [dict get $result pending_repository_checks] || [dict get $result inventory_page] != 1 || [:get_state repository_discovery {}] ne ""} { dict set result state catching-up } elseif {![dict get $result last_inventory]} { dict set result state not-yet-checked } else { dict set result state idle } return $result } #--- next_backfill_event_id ------------------------------------------------------ :method next_backfill_event_id {} { # # Helper for backfill_repo_history. Returns the next # negative event_id to use. # # smallest existing event_id (could be null, positive, or already negative) set min_id [xo::dc get_value min_event_id { select min(event_id) from github_activity }] if {$min_id eq "" || $min_id >= 0} { # no rows, or only positive IDs so far -> start at -1 return -1 } else { # already have negative IDs -> one step lower return [expr {$min_id - 1}] } } #--- backfill_repo_history ------------------------------------------------------- :public method backfill_repo_history {args} { ::xo::dc transaction { set locked [::xo::dc get_value github_backfill_lock { select case when pg_try_advisory_xact_lock(714230, 1) then 1 else 0 end }] if {!$locked} { error "GitHub activity writer already running; retry later" } # Catch inside the transaction so a cooldown learned from an # HTTP failure commits even when this explicit call fails. set code [catch { :check_rate_limit :backfill_locked {*}$args } result options] } if {$code} {return -options $options $result} return $result } :method backfill_locked {-repo {-branch main} {-revision ""} {-base ""} {-since ""} {-start_page 1} {-max_pages 50}} { # # Backfill commit history for a single repo/branch using # /repos/{repo}/commits. Uses synthetic negative event_id values. # # repo e.g. "openacs/openacs-core" # branch e.g. "main" or "oacs-5-10" # if {![string is integer -strict $start_page] || $start_page < 1 || ![string is integer -strict $max_pages] || $max_pages < 1} { error "start_page and max_pages must be positive integers" } if {$revision eq ""} { set revision $branch } ns_log Notice "GitHub backfill_repo_history: $repo ($branch)" # We’ll assign event_id from next negative downwards. set next_event_id [:next_backfill_event_id] set page $start_page set pages_done 0 set done 0 set count 0 while {!$done} { if {$pages_done >= $max_pages} { ns_log notice "GitHub backfill_repo_history: reached" \ "max_pages=$max_pages for $repo ($branch)" break } # Compare against the successfully reconciled head. Do not # simply break when that SHA appears in a date-ordered commit # list: newly merged side branches may have older timestamps. set batch "" set full_history [expr {$base eq ""}] if {!$full_history} { if {[catch { set comparison [::util::json2dict [:get "/repos/$repo/compare/${base}...${revision}" \ [list page $page per_page 20]]] } message options]} { if {[dict get $options -errorcode] ne {GITHUB HTTP 404}} { return -options $options $message } # An unavailable comparison must not silently advance # the checkpoint or trigger an unbounded history scan. set attention [:get_state attention_required {}] dict set attention [list $repo $branch] [dict create base $base revision $revision reason missing-comparison-base] :set_state attention_required $attention return -code error -errorcode {GITHUB BASE_UNAVAILABLE} "Cannot compare $repo $branch with saved base $base; job retained, no full reload attempted" } elseif {[dict get $comparison status] in {ahead identical diverged behind}} { set batch [dict get $comparison commits] } else { error "Unexpected comparison status: [dict get $comparison status]" } } if {$full_history} { set query [list sha $revision page $page per_page 20] if {$since ne ""} {lappend query since $since} set body [:get "/repos/$repo/commits" $query] set batch [::util::json2dict $body] } if {[llength $batch] == 0} { ns_log notice "GitHub backfill_repo_history: no more commits" \ at page $page for $repo ($branch) set done 1 break } # The public entry point owns the transaction and lock. # Avoid nested transaction abort semantics on budget exhaustion. foreach summary $batch { set sha [dict get $summary sha] # Skip duplicates; a commit shared with another branch # does not prove that all older history is present. set exists [xo::dc 0or1row exists_commit { select 1 from github_activity where repo = :repo and sha = :sha limit 1 }] if {$exists} { continue } incr count # # Fetch full commit details (stats, files, etc.), # similar to summarize_push_event. # set commit_body [:get "/repos/$repo/commits/$sha"] set c [::util::json2dict $commit_body] set commit [dict get $c commit] #ns_log notice "commit keys: [dict keys $commit]" ns_log notice "(page $page, count $count) commit.message($count):" \ [dict get $commit message] set author_dict [dict get $commit author] set author_name [dict get $author_dict name] set commit_date [dict get $author_dict date] set stats [dict get $c stats] set additions [dict get $stats additions] set deletions [dict get $stats deletions] set files [dict get $c files] set files_changed [llength $files] set message_full [dict get $commit message] if {$message_full eq ""} { # Fallback: don’t leave title NULL, DB wants NOT NULL set title "(no commit message)" } else { set title [lindex [split [string trimleft $message_full] \n] 0] } #ns_log notice "commit.title: $title" if {[dict exists $c html_url]} { set url [dict get $c html_url] } else { # construct GitHub web URL as a fallback set url "https://github.com/$repo/commit/$sha" } # Synthetic negative event_id set event_id $next_event_id incr next_event_id -1 set created_at $commit_date ;# for backfill, we just reuse commit time ::xo::dc dml insert_backfill { insert into github_activity ( event_id, repo, branch, sha, author_name, author_login, commit_date, created_at, title, url, files_changed, additions, deletions, raw_payload ) values ( :event_id, :repo, :branch, :sha, :author_name, NULL, :commit_date, :created_at, :title, :url, :files_changed, :additions, :deletions, :commit_body ) on conflict (repo, sha) do nothing } } if {$done} { break } incr page incr pages_done if {[llength $batch] < 20} { set done 1 } } ns_log notice "GitHub backfill_repo_history: done for $repo ($branch)" return [dict create complete $done next_page $page base $base] } } } ::xo::library source_dependent # # Local variables: # mode: tcl # tcl-indent-level: 4 # indent-tabs-mode: nil # End:
![[i]](/resources/acs-subsite/ZoomIn16.gif)