::nx::Object ::nx::Object ::ns::stats::SQLiteHistory ::ns::stats::SQLiteHistory allocatorPoint history allocator history configure history disable history query history save history status history storage historyConnect historyDrop historySchema historyStorage historyWindowCoverage pack quote scalar sql unpack ::ns::stats::SQLiteHistory->::nx::Object

Class ::ns::stats::SQLiteHistory

::ns::stats::SQLiteHistory[i] create ...
Defined in

Class Relations

  • class: ::nx::Class[i]
  • superclass: ::nx::Object[i]
  • mixinof: ::ns_stats[i]
::nx::Class create ::ns::stats::SQLiteHistory \
     -superclass ::nx::Object

Methods (to be applied on instances)

  • history allocator (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history allocator \
    -from from  -to to  [ -limit limit ] [ -cursor cursor ]

    Read compact process-wide allocator history from the configured owner server using one bounded sample SELECT. New samples provide stored projections; old rows without a projection are decoded once in this read. No topology loading or full-window payload rescan is performed. Preserve missing gauges as absent dictionary keys. Timestamps are epoch microseconds; instance and intervalSeconds allow callers to preserve restart boundaries and gaps. Reading does not collect or write samples. Coverage complete and windowExamined remain false: this projection does not certify window boundaries or absence of eviction/write failures. Use history query when full-window coverage assessment is required. Timings distinguish checkout, sample SQL read, decoding and total time.

    Switches:
    -from (required, integer)
    Inclusive lower bound in epoch seconds; must be nonnegative.
    -to (required, integer)
    Exclusive upper bound in epoch seconds; after from, at most 24 hours later and no later than now.
    -limit (optional, integer, defaults to "10000")
    Maximum compact records per page, 1..10000; default 10000.
    -cursor (optional, integer, defaults to "0")
    Nonnegative timestamp cursor in microseconds, default 0; pass nextCursor unchanged for continuation.
    Returns:
    Dictionary containing compact records (timestamp, instance, sampling.intervalSeconds and memory), more, nextCursor, legacyDecoded, timings in milliseconds and conservative coverage metadata.

    Testcases:
    No testcase defined.
    set started [clock microseconds]
if {$from < 0 || $to <= $from || $to-$from > 86400 || $to > [clock seconds]
    || $limit < 1 || $limit > 10000 || $cursor < 0} {
    error "expected an epoch window up to 24 hours, limit 1..10000 and nonnegative cursor"
}
set config [nsv_get ns_stats history]
if {$config eq ""} {error "history is not configured"}
if {[dict get $config owner] ne [ns_info server]} {error "query history from its configured owner server"}
set db [:historyConnect $config]
set checkedOut [clock microseconds]
set transaction false
try {
    :sql $db {BEGIN}
    set transaction true
    set columns [lmap row [:sql $db {SELECT name FROM pragma_table_info('ns_stats_samples')}] {dict get $row name}]
    if {"allocator" in $columns} {
        set projection {coalesce(allocator,'') AS allocator,CASE WHEN allocator IS NULL THEN payload ELSE '' END AS legacy}
    } else {
        set projection {'' AS allocator,payload AS legacy}
    }
    set low [expr {$from*1000000}]; set high [expr {$to*1000000}]
    set beforeRead [clock microseconds]
    set rows [:sql $db "SELECT at,instance,$projection FROM ns_stats_samples WHERE at >= $low AND at < $high AND at > $cursor ORDER BY at LIMIT [expr {$limit+1}]"]
    set afterRead [clock microseconds]
    set more [expr {[llength $rows]>$limit}]
    set records {}; set legacy 0
    foreach row [lrange $rows 0 [expr {$limit-1}]] {
        if {[dict get $row allocator] eq ""} {
            set point [:allocatorPoint [:unpack [dict get $row legacy]]]
            incr legacy
        } else {
            set point [dict get $row allocator]
        }
        lappend records $point
    }
    set decoded [clock microseconds]
    :sql $db {COMMIT}
    set transaction false
    return [dict create schemaVersion 1 source sqlite projection allocator from $from to $to  records $records more $more nextCursor [expr {$more ? [dict get [lindex $rows $limit-1] at] : ""}]  legacyDecoded $legacy coverage [dict create complete false windowExamined false  incompleteReasons [expr {$more ? {allocator-projection-not-certified more-pages} : {allocator-projection-not-certified}}]]  timings [dict create checkoutMs [expr {($checkedOut-$started)/1000.0}]  readMs [expr {($afterRead-$beforeRead)/1000.0}] decodeMs [expr {($decoded-$afterRead)/1000.0}]  totalMs [expr {([clock microseconds]-$started)/1000.0}]]]
} finally {
    if {$transaction} {catch {:sql $db {ROLLBACK}}}
    ns_db releasehandle $db
}

  • history configure (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history configure \
    -pool pool  [ -retention retention ] [ -maxBytes maxBytes ]

    Validate the database pool and retention limits, initialize the schema and start a separate persistence callback. Repeated configuration with identical settings is harmless; changing active settings requires disabling persistence first. Return the resulting history status. Does not start sampling. Normalize retention to fractional seconds and maxBytes to bytes with ns_baseunit before validating the bounds.

    Switches:
    -pool (required)
    Required dedicated persistent nsdbsqlite pool, accessible to the configuring virtual server.
    -retention (optional, defaults to "86400")
    Retention duration accepting ns_baseunit time syntax; 60..604800 seconds, default 86400 seconds.
    -maxBytes (optional, defaults to "67108864")
    Storage budget accepting ns_baseunit size syntax; at least 4 MiB, default 67108864 bytes. Includes database journal headroom.
    Returns:
    History status dictionary after schema initialization and persistence scheduling.

    Testcases:
    No testcase defined.
    set retention [ns_baseunit -time $retention]
set maxBytes [ns_baseunit -size $maxBytes]
if {$retention < 60 || $retention > 604800 || $maxBytes < 4194304} {
    error "retention must be 60..604800 seconds and maxBytes at least 4 MiB"
}
if {[info commands ns_db] eq ""} {
    error "history requires nsdb/nsdbsqlite"
}
set config [dict create  enabled true pool $pool retention $retention maxBytes $maxBytes  owner [ns_info server]  busy false scheduler {}  generation [clock microseconds] lastSaved {}  failures 0 lastFailure {} lastLoggedAt 0 gaps {}  mixins [:info object mixins] topology {} topologyAt 0]
set old [nsv_get ns_stats history]
if {$old ne "" && [dict get $old enabled]} {
    foreach k {pool retention maxBytes owner} {
        if {[dict get $old $k] ne [dict get $config $k]} {
            error "history already configured differently; disable before reconfiguring"
        }
    }
    return [:history status]
}
set db [:historyConnect $config]
try {
    :historySchema $db
} finally {
    ns_db releasehandle $db
}
:locked {
    set old [nsv_get ns_stats history]
    if {$old ne "" && [dict get $old enabled]} {
        error "history was concurrently configured; retry"
    }
    dict set config scheduler [ns_schedule_proc -thread 1 [self] history persist]
    nsv_set ns_stats history $config
}
return [:history status]

  • history disable (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history disable

    Stop future persistence callbacks without deleting stored history or stopping sampling. An outstanding write may finish. Return the resulting history status.

    Returns:
    Resulting history status dictionary; retained database records are not deleted.

    Testcases:
    No testcase defined.
    :locked {
    set config [nsv_get ns_stats history]
    if {$config ne ""} {
        if {[dict get $config enabled] && [dict get $config scheduler] ne ""} {
            ns_unschedule_proc [dict get $config scheduler]
        }
        dict set config enabled false
        dict set config scheduler {}
        nsv_set ns_stats history $config
    }
}
return [:history status]

  • history query (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history query -from from  \
    -to to  [ -server server ] [ -limit limit ] [ -cursor cursor ]

    Return stored samples, associated topology, paging information and coverage for an epoch interval. Queries run in the configuring virtual server and use an operation-local database handle. Historical server selection may include servers no longer active.

    Switches:
    -from (required, integer)
    Inclusive lower bound in epoch seconds; must be nonnegative.
    -to (required, integer)
    Exclusive upper bound in epoch seconds; after from, at most 24 hours later and no later than now.
    -server (optional)
    Optional historical virtual-server name, including one no longer configured; empty returns all stored server data.
    -limit (optional, integer, defaults to "1000")
    Maximum records per page, 1..1000; default 1000.
    -cursor (optional, integer, defaults to "0")
    Nonnegative timestamp cursor in microseconds, default 0; pass nextCursor unchanged for continuation.
    Returns:
    Dictionary containing records, referenced topologies, storage statistics, coverage, more and nextCursor. Final-page coverage examines the full requested window.

    Testcases:
    No testcase defined.
    if {$from < 0
    || $to <= $from
    || $to - $from > 86400
    || $to > [clock seconds]
    || $limit < 1
    || $limit > 1000
    || $cursor < 0
} {
    error "expected an epoch window up to 24 hours, limit 1..1000 and nonnegative cursor"
}
if {[string length $server]>512 || [string first \x00 $server] >= 0} {
    error "invalid server name"
}
set config [nsv_get ns_stats history]
if {$config eq ""} {
    error "history is not configured"
}
if {[dict get $config owner] ne [ns_info server]} {
    error "query history from its configured owner server"
}
set db [:historyConnect $config]
set transaction false
try {
    :sql $db {BEGIN}
    set transaction true
    set low [expr {$from * 1000000}]
    set high [expr {$to * 1000000}]
    set count [expr {$limit + 1}]
    set rows [:sql $db "SELECT at,topology,payload FROM ns_stats_samples WHERE at >= $low AND at < $high AND at > $cursor ORDER BY at LIMIT $count"]
    set more [expr {[llength $rows] > $limit}]
    set rows [lrange $rows 0 [expr {$limit - 1}]]
    set records {}
    set gaps {}
    set restarts {}
    set previous {}
    set topology {}
    set resets 0
    set boundary [expr {max($low,$cursor)}]
    set operator [expr {$cursor >= $low ? "<=" : "<"}]
    set prior [:scalar $db "SELECT payload FROM ns_stats_samples WHERE at $operator $boundary ORDER BY at DESC LIMIT 1"]

    if {$prior ne ""} {
        set previous [:unpack $prior]
    }
    foreach row $rows {
        set record [:unpack [dict get $row payload]]
        if {![dict get $record sampling ready]} {
            incr resets
        }
        if {$previous ne ""} {
            if {[dict get $previous instance] ne [dict get $record instance]} {
                lappend restarts [dict get $record sampling to]
            }
            set delta [expr {([dict get $record timestamp]-[dict get $previous timestamp]) / 1000000.0}]
            if {$delta > 1.5 * max([dict get $previous sampling intervalSeconds],[dict get $record sampling intervalSeconds])} {
                lappend gaps [dict create from [dict get $previous sampling to] to [dict get $record sampling to]]
            }
        }
        set previous $record
        set id [dict get $row topology]
        dict set record topologyId $id
        if {![dict exists $topology $id]} {
            dict set topology $id [:selectServer  [:unpack [:scalar $db "SELECT payload FROM ns_stats_topology WHERE id=[:quote $id]"]]  $server false]
        }
        lappend records [:selectServer $record $server false]
    }

    if {!$more && $previous ne ""} {
        set following [:scalar $db "SELECT payload FROM ns_stats_samples WHERE at >= $high ORDER BY at LIMIT 1"]
        if {$following ne ""} {
            set nextRecord [:unpack $following]
            set delta [expr {([dict get $nextRecord timestamp]-[dict get $previous timestamp]) / 1000000.0}]
            if {$delta > 1.5 * max([dict get $previous sampling intervalSeconds],[dict get $nextRecord sampling intervalSeconds])} {
                lappend gaps [dict create from [dict get $previous sampling to] to [dict get $nextRecord sampling to]]
            }
        }
    }
    set bounds [lindex [:sql $db {SELECT coalesce(min(at),0) AS first,coalesce(max(at),0) AS last FROM ns_stats_samples}] 0]
    set events [:sql $db {SELECT at,first,last,n,reason FROM ns_stats_events ORDER BY id DESC}]
    set first [dict get $bounds first]
    set last [dict get $bounds last]
    set overlap false
    foreach event $events {
        if {[dict get $event first] < $high && [dict get $event last] >= $low} {
            set overlap true
        }
    }
    set through [:scalar $db {SELECT value FROM ns_stats_meta WHERE key='droppedThrough'}]
    if {$through ne "" && $through >= $low} {
        set overlap true
    }
    set next {}
    if {$more} {
        set next [dict get [lindex $rows end] at]
    }
    # Intermediate coverage describes this page. The final response
    # examines all retained intervals in the requested window again,
    # within this read transaction, independently of the paging cursor.
    if {!$more} {
        set window [:historyWindowCoverage $db $low $high]
        set gaps [dict get $window gaps]
        set restarts [dict get $window restartBoundaries]
        set resets [dict get $window resetSamples]
    }
    set complete [expr {$first>0 && $first<=$low && $last>=$high && !$more
                        && !$overlap && $gaps eq "" && $restarts eq "" && $resets==0}]
    set reasons {}
    if {$more} {lappend reasons more-pages}
    if {$first==0 || $first>$low} {lappend reasons before-retained-history}
    if {$last<$high} {lappend reasons after-latest-sample}
    if {$overlap} {lappend reasons eviction-or-write-failure}
    if {$gaps ne ""} {lappend reasons gaps}
    if {$restarts ne ""} {lappend reasons restarts}
    if {$resets>0} {lappend reasons baseline-or-reset}
    set result [dict create  schemaVersion 1 from $from to $to records $records  topologies $topology nextCursor $next more $more  storage [:historyStorage $db $config]  coverage [dict create  complete $complete  windowExamined [expr {!$more}]  scope [expr {$more ? "page" : "window"}]  incompleteReasons $reasons  retainedFrom [expr {$first / 1000000.0}]  retainedTo [expr {$last / 1000000.0}]  gaps $gaps  restartBoundaries $restarts  resetSamples $resets  events $events  evictionOrFailureOverlap $overlap  droppedThrough $through  writeFailuresSinceStart [dict get $config failures]]]
    :sql $db {COMMIT}
    set transaction false
    return $result
} finally {
    if {$transaction} {
        catch {:sql $db {ROLLBACK}}
    }
    ns_db releasehandle $db
}

  • history save (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history save

    Persist the latest published sample if history is enabled, no write is in progress and its timestamp has not already been saved. Claim the write under the shared coordination lock, then perform database operations outside that lock. Store the compressed sample and associated topology in one transaction, deduplicating samples by timestamp. Apply age and storage-budget eviction, retain coverage events and remove topology no longer referenced by retained samples. On failure, roll back the transaction, retain the pending sample for retry and update failure status. Repeated identical warnings are throttled. A subsequent successful write records the failed interval for coverage reporting. Release the database handle and clear the write claim on completion. Does not collect observations or advance the sampling baseline; callers inspect history status for the persistence outcome.

    Testcases:
    No testcase defined.
    set config {}
set generation {}
set stamp {}
:locked {
    set candidate [nsv_get ns_stats history]
    set sample [nsv_get ns_stats pair]
    if {$candidate ne "" && [dict get $candidate enabled] && ![dict get $candidate busy] && $sample ne ""} {
        set stamp [dict get $sample timestamp]
        if {$stamp ne [dict get $candidate lastSaved]} {
            set config $candidate
            set generation [dict get $config generation]
            dict set candidate busy true
            nsv_set ns_stats history $candidate
        }
    }
}
if {$config eq ""} {
    return
}
set db {}
set transaction false
set ok false
set errorMessage {}
try {
    # Collection and SQLite work happen outside the shared control mutex.
    set now [clock seconds]
    set topology [dict get $config topology]
    if {$topology eq "" || $now-[dict get $config topologyAt]>=60} {
        set topology [:describe]
        foreach k {pid bootTime uptimeSeconds} {dict unset topology process $k}
        dict set config topology $topology
        dict set config topologyAt $now
    }
    # Stable Tcl serialization is enough for a private cache key; store
    # the full topology in the table to detect the unlikely CRC collision.
    set topologyId [format %08x [zlib crc32 [encoding convertto utf-8 $topology]]]
    set record $sample
    dict unset record raw
    dict unset record cpu
    dict set record topologyId $topologyId
    set payload [encoding convertto utf-8 $record]
    set rawSize [string length $payload]
    if {$rawSize > 1048576} {
        error "history sample exceeds 1 MiB"
    }
    set packed [:pack $record]
    set packedTopology [:pack $topology]
    set allocator [:allocatorPoint $record]
    set size [expr {[string length $packed]+[string length $packedTopology]+[string length $allocator]}]
    set db [:historyConnect $config]
    :historySchema $db
    :sql $db {BEGIN IMMEDIATE}
    set transaction true
    set cutoff [expr {([clock seconds]-[dict get $config retention])*1000000}]
    :historyDrop $db "at < $cutoff" age
    set page [:scalar $db {PRAGMA page_size}]
    set max [:scalar $db {PRAGMA max_page_count}]
    while {1} {
        set used [expr {([:scalar $db {PRAGMA page_count}] - [:scalar $db {PRAGMA freelist_count}]) * $page}]
        if {$used+3*$size < $max*$page*0.8} {
            break
        }
        if {![:scalar $db {SELECT count(*) FROM ns_stats_samples}]} {
            error "history metadata or sample exceeds storage budget"
        }
        :historyDrop $db {at IN (SELECT at FROM ns_stats_samples ORDER BY at LIMIT 100)} size
    }
    set stored [:scalar $db "SELECT payload FROM ns_stats_topology WHERE id=[:quote $topologyId]"]
    if {$stored ne "" && [:unpack $stored] ne $topology} {
        error "history topology fingerprint collision"
    }
    :sql $db "INSERT OR IGNORE INTO ns_stats_topology VALUES([:quote $topologyId],[:quote $packedTopology])"
    :sql $db "INSERT OR IGNORE INTO ns_stats_samples(at,instance,topology,payload,allocator) VALUES($stamp,[:quote [dict get $record instance]],[:quote $topologyId],[:quote $packed],[:quote $allocator])"
    foreach gap [dict get $config gaps] {
        lassign $gap first last n
        :sql $db "INSERT INTO ns_stats_events(at,first,last,n,reason) VALUES($stamp,$first,$last,$n,'write-failure')"
    }
    :sql $db {DELETE FROM ns_stats_events WHERE id NOT IN (SELECT id FROM ns_stats_events ORDER BY id DESC LIMIT 64)}
    :sql $db {DELETE FROM ns_stats_topology WHERE id NOT IN (SELECT DISTINCT topology FROM ns_stats_samples)}
    :sql $db {COMMIT}
    set transaction false
    set ok true

} on error {message options} {
    set errorMessage $message
    if {$transaction} {
        catch {:sql $db {ROLLBACK}}
    }
    if {$message ne [dict get $config lastFailure] || [clock seconds]-[dict get $config lastLoggedAt] >= 60} {
        ns_log warning "ns_stats: history write failed: $message"
        dict set config lastLoggedAt [clock seconds]
    }
} finally {
    if {$db ne ""} {
        ns_db releasehandle $db
    }
    :locked {
        set current [nsv_get ns_stats history]
        if {$generation eq [dict get $current generation]} {
            dict set current busy false
            dict set current lastLoggedAt [dict get $config lastLoggedAt]
            if {$ok} {
                dict set current lastSaved $stamp
                dict set current lastFailure {}
                dict set current gaps {}
                dict set current topology [dict get $config topology]
                dict set current topologyAt [dict get $config topologyAt]
            } else {
                dict incr current failures
                dict set current lastFailure $errorMessage
                set gaps [dict get $current gaps]
                if {$gaps eq ""} {
                    set gaps [list [list $stamp $stamp 1]]
                } else {
                    lassign [lindex $gaps 0] first last n
                    set gaps [list [list $first $stamp [expr {$n+1}]]]
                }
                dict set current gaps $gaps
            }
            nsv_set ns_stats history $current
        }
    }
}

  • history status (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history status

    Return shared history configuration and persistence status without opening the database. Internal coordination and cached topology fields are omitted.

    Returns:
    Persistence status dictionary including owner, pool, retention, maxBytes, scheduler and write failures; enabled is false when never configured.

    Testcases:
    No testcase defined.
    set config [nsv_get ns_stats history]
if {$config eq ""} {
    return {enabled false}
}
dict unset config busy
dict unset config mixins
dict unset config topology
dict unset config gaps
return $config

  • history storage (scripted, public)

     <instance of ns::stats::SQLiteHistory[i]> history storage

    Return retained bounds/count and SQLite space usage for sizing the dedicated history database. Run in the history owner server. Does not collect samples, write history or change its configuration.

    Returns:
    Dictionary of retained epoch-second bounds, sample count, SQLite page usage, effective byte cap and estimated bytes per sample/second; unavailable estimates are empty.

    Testcases:
    No testcase defined.
    set config [nsv_get ns_stats history]
if {$config eq ""} {
    error "history is not configured"
}
if {[dict get $config owner] ne [ns_info server]} {
    error "query history from its configured owner server"
}
set db [:historyConnect $config]
try {
    :sql $db {BEGIN}
    try {
        set result [:historyStorage $db $config]
    } finally {
        :sql $db {ROLLBACK}
    }
    return $result
} finally {
    ns_db releasehandle $db
}