Hacker News sees over 10 million monthly visitors and serves as a leading indicator for technology trends. When a new framework goes viral, a startup gets acquired, or a security vulnerability surfaces, it often hits HN first. For developer-focused companies, monitoring HN in real-time provides valuable signals:

Developer Relations & Marketing: Track mentions of your product, spot emerging discussions, and engage with your community when it matters most

Competitive Intelligence: Monitor competitor launches, feature announcements, and community sentiment as they happen

Trend Detection: Identify rising technologies, tools, and topics before they go mainstream

Recruiting: Find active contributors discussing relevant technologies and engage them early

But here’s the challenge: HN’s API is pull-based, meaning you need to poll it regularly to catch new content. Building a reliable ingestion pipeline typically requires setting up cron jobs, handling retries, and managing state—all before you can even start analyzing the data.

What if you could do all of this with just SQL?

In this blog, I’ll show you how to use Timeplus Scheduled Tasks to build a complete HN monitoring pipeline, from data ingestion to real-time analysis, without leaving your SQL console.

What is a Timeplus Task?

A Timeplus Scheduled Task runs a SQL query periodically and persists the results to a target stream. Think of it as a cron job, but native to your streaming database.

Tasks complement Materialized Views nicely: while MVs run continuously on streaming data, Tasks run on a schedule for batch-style operations.

Common Use Cases

Some common use cases the user can leverage tasks are:

Pull data from external APIs — Fetch data from REST APIs using Python UDFs and load it into streams

Periodic aggregations — Compute hourly/daily rollups and store them for fast querying

System monitoring — Collect cluster health metrics at regular intervals

Data synchronization — Keep external systems in sync with your Timeplus data

Demo: Real-Time Hacker News Ingestion

Let’s build a data pipeline that pulls new Hacker News posts every 10 seconds and enables real-time analysis.

Step 1: Create a Python UDF to Fetch HN Posts

First, we need a function that calls the Hacker News API (https://hacker-news.firebaseio.com). This Python UDF fetches posts after a given ID with retry logic for reliability:

Step 2: Create the Target Stream

We create a stream to store the HN posts with a 7-day retention policy:

Step 3: Create the Scheduled Task

Now the magic: a task that runs every 10 seconds, checks the latest ingested post ID, and fetches new posts:

How it works:

Query the existing stream to find the highest post ID we’ve already ingested 1.

Call the Python UDF to fetch up to 20 new posts after that ID 1.

Flatten the array and insert each post as a row into hn.hn_post

Step 4: Run Real-Time Analysis

With data flowing in, you can now run analytics. Here are two examples:

Most active users in the last 24 hours:

Post type distribution in the last hour:

You can also create streaming queries (without table()) to get continuous, real-time updates as new posts arrive.

Summary

Timeplus Scheduled Tasks provide a simple yet powerful way to build data pipelines that pull from external sources on a schedule. Combined with Python UDFs, you can integrate virtually any API into your streaming analytics workflow.

In this demo, we built a complete Hacker News monitoring pipeline with just a few SQL statements:

A Python UDF to fetch data from the HN API

A stream to store the posts

A scheduled task to pull new data every 10 seconds

Analytical queries to extract insights

To learn more, check out theTimeplus Task documentation and explore other use cases like alerting, data synchronization, and system monitoring.

Try it yourself: Download Timeplus and start building real-time data pipelines today.