Fetch Tweets Using Twitter API | Step by Step Guide

Now that you've got your feet wet, let's dive deeper into the Recent Search endpoint. This powerful tool is your ticket to finding specific tweets from the last seven days. Here's how to make it work for you:

Essential Access Rule Limits: The Fine Print

Before you start crafting clever queries, it's good to know the guardrails. With Essential access, you can set up to 5 rules for collecting tweets. Each of these rules can be as detailed as you want—just keep in mind that each rule is limited to 512 characters. That means you'll need to prioritize your search logic and make smart use of operators to fit everything in.

If you find yourself bumping up against these rule or character limits, it might be time to consider upgrading your access level. For most beginners and casual projects, though, five well-planned rules should be plenty to get you started!

Basic Query Structure

The Recent Search endpoint is all about the query. Here's a simple structure:

https://api.x.com/2/tweets/search/recent?query=your_search_terms_here

For example, to find tweets about cats:

https://api.x.com/2/tweets/search/recent?query=cats

How Many Tweets Can You Grab Per Request?

Curious how much Twitter goodness you can pull in a single API call? Each request to the Recent Search endpoint will fetch you up to 100 tweets at a time. If you need more than that, no worries—just use the pagination token included in the response to keep going and collect even more tweets from the last seven days.

Modifying Queries for Specific Data

Want to get fancy? Try these query modifications:

1. From a specific user: from:username

2. Containing a hashtag: #hashtag

3. Tweets with media: has:images or has:videos

4. Tweets in a language: lang:en (for English)

The Art of Query Refinement: Precision Matters

Here's the deal—refining your queries isn't just a nice-to-have; it's the secret sauce to gathering high-quality Twitter data without getting buried under a mountain of irrelevant tweets. When you first start searching, your results might be a bit messy or too broad. That's normal! It's all part of the process.

Why refine? Because targeted queries mean you'll collect tweets that actually matter to your project, rather than sifting through endless noise. For example, if you're searching for tweets about the Swift programming language but don’t tweak your search, you might get caught in a tsunami of Taylor Swift fan chatter.

Tips to keep your queries sharp:

• Adjust your keywords to exclude unrelated topics.

• Use advanced search operators (like -taylor if you want Swift without the pop star).

• Explore your initial results and adjust your query terms based on what you see—it's a little like tuning a radio to the perfect station.

• Iterate until your search gives you exactly what you’re after.

This attention to detail is especially crucial in real-time data collection, when you might miss the tweets you actually care about if your net is too wide. So give your queries a little TLC, and you'll be swimming in the right data in no time!

Pro Tips: Crafting Effective Rules for Relevant Tweets

Ready to fine-tune your Twitter data collection game? The secret ingredient is writing smart, laser-focused queries. Here’s how to zero in on exactly the tweets you want—no more, no less:

• Start specific. Begin with a narrow query to target your audience precisely, then broaden if you’re not seeing enough results.

• Use filters to your advantage—combine keywords, hashtags, “from:” usernames, media types, and language codes to exclude the noise.

• Test and tweak. Run a sample search, review the results, and adjust your query to weed out unwanted tweets.

• Watch for ambiguous keywords! For example, searching for “Swift” might snag posts about Taylor Swift when you really want programming chatter. Add context with more keywords (like “Swift language” or “#iOSDev”) to keep things on track.

• Don’t set and forget! As you collect tweets, keep refining your rules to improve quality and relevance. Data collection is an ongoing process.

With each adjustment, you’re getting closer to building a goldmine of targeted, actionable Twitter data.

Combine these for more precise results:

query=cats from:ASPCA has:images lang:en

This would find English tweets about cats from @ASPCA that include images.

Paging Through Results: How to Use the next_token

Let’s say one page of tweets just isn’t enough. Twitter’s API has you covered with easy pagination. Here’s how it works:

• After every API call, check the meta section of the JSON response.

• If there's a next_token field, that means there are more tweets waiting for you.

• Simply take that next_token value and add it as a query parameter—like &next_token=your_token_here—to your next request.

Rinse and repeat: keep using the newest next_token each time, and you’ll page through result after result until eventually the token disappears. When that happens, congrats! You’ve reached the end of the available tweets for your search.

Filtering Tweets by Date Range: The Right Way

Ready to travel back in time (at least as far as the Twitter archives will let you)? If you want to fetch tweets from a specific date range, there's a little secret: you don’t include dates directly in your query string like since: or until:. Instead, recent API versions use special URL parameters to handle time filtering.

Here's how to do it:

• Use start_time to set the earliest date and time for tweets you want to grab.

• Use end_time to set the latest date and time.

Both must be in ISO 8601 format (think: 2024-01-01T00:00:00Z).

So, your URL might look like this:
https://api.x.com/2/tweets/search/recent?query=cats&start_time=2024-06-01T00:00:00Z&end_time=2024-06-03T00:00:00Z

This will fetch tweets containing "cats" from June 1, 2024, up to but not including June 3, 2024.

Pro tip: Popular libraries like twitter-api-v2 (for JavaScript) support these parameters—just pass them in when you call the relevant search method.

Now that you know how to set precise timeframes, you’re that much closer to building your own Twitter time machine!

Pro Tip: Fetching Tweets from Specific Users for a Date Range

So, you want to retrieve tweets from specific user IDs during a custom time window—say, the infamous Covid era? Totally doable! Here’s how to gear up and grab those tweets like a true data wrangler.

First, let’s address the golden rule: when querying by date, the Recent Search endpoint only gets you tweets from the past seven days. If you need tweets from further back (e.g., the entire Covid period), you'll need access to the full archive, which usually requires Academic Research access. Don’t worry, if that’s not an option, there are helpful workarounds below.

Using Python & Tweepy for Simple Fetches (Recent Only):

If your target date is within the last week, Tweepy is your friend. Here’s what you do:

1. Authenticate with your API keys as always.

2. Use the start_time and end_time parameters, not search keywords, when you want to filter by date range.

3. Iterate over your user IDs and make requests like this:

import tweepy
from datetime import datetime

client = tweepy.Client(bearer_token="YOUR_TOKEN")

user_id = "123456789"
start_time = "2020-03-01T00:00:00Z"
end_time = "2020-12-31T23:59:59Z"

tweets = client.get_users_tweets(
    id=user_id,
    start_time=start_time,
    end_time=end_time,
    max_results=100
)

For the Deep Dive: Grab Older Tweets with Command-Line Tools

If you need historical tweets (way more than 7 days back), you’ll want to use tools like Twarc—an academic favorite for serious data dredging:

• Save your user IDs—one per line—in a text file, e.g., twitter_ids.txt.

• Fetch timelines with a specific timeframe:

twarc2 timelines --start-time "2020-03-01" --end-time "2021-12-31"

If you have Academic Research access, you can fetch across the full archive. If not, you’re limited to the most recent ~3200 tweets per user, regardless of date.

• Optional: Flatten the results so you get one tweet per line:

  twarc2 flatten results.jsonl tweets.jsonl

• You can then import tweets.jsonl into your favorite database for analysis.

Troubleshooting Tips:

• Ensure you're using the correct bearer token; Academic endpoints require specific app access.

• If you run into permissions issues, double-check your project type in the Twitter Developer Portal.

• No Academic Access? You’ll be limited to recent tweets, but you can still collect a substantial sample per user.

With these approaches, you’ll be ready to capture tweets from any set of users, for any time period your project demands!

Crafting Advanced Search Rules

Ready to level up your searches? The Recent Search endpoint isn't just about searching plain keywords—you can set up rules to capture exactly the conversations you care about.

Let's say you want to pull tweets about "heat pumps" or "gas boilers," but skip all retweets and focus only on English-language tweets. Twitter API makes this a breeze using query rule syntax. Here's how you can define your search rules in code:

rules = [
    {
        "value": '("heat pump" OR "heat pumps") -is:retweet lang:en',
        "tag": "heat_pump"
    },
    {
        "value": '("gas boiler" OR "gas boilers") -is:retweet lang:en',
        "tag": "gas_boiler"
    }
]

Each rule is a mini search command:

• Use OR to capture different ways people might mention a topic.

• Exclude retweets (so you avoid duplicates) using -is:retweet.

• Set the language, like lang:en for English.

Tags help you label and organize results, making it easy to track which rule caught which tweet. You can define up to five rules with the Essential access level, each up to 512 characters—plenty of room to get creative with your searches.

Using Fields and Expansions

To get more detailed responses, use fields and expansions:

1. Add tweet fields: tweet.fields=created_at,author_id,public_metrics

2. Include user data: expansions=author_id&user.fields=username,verified

Your URL might look like this:

https://api.x.com/2/tweets/search/recent?query=cats&tweet.fields=created_at,author_id,public_metrics&expansions=author_id&user.fields=username,verified

This gives you creation time, author info, and engagement metrics for each tweet.

Heads up: You’ll need to add the parameter (as above) to actually receive user data in your response. When you do, the response JSON will include an extra key called , where you’ll find user-related information—like usernames, whether the author is verified, and more. Check your response object and you’ll see that user details are conveniently separated out in this new section. This makes it much easier to match tweet data with user info, especially if you’re working with multiple authors in a single request.

Building Powerful Queries with Operators

But wait, there’s more! The real magic is in crafting the perfect query using operators—these let you filter tweets with surgical precision. The Recent Search and Filtered Stream endpoints let you build rules using operators that match on tweet text, user bio, location, and more. Each endpoint has its own set of available operators, which may change depending on your API access level.

Let’s say you want tweets mentioning black cat(s), but not dog(s), and you want to skip retweets. Your query would look like this:

Not sure what all that means? Here’s the breakdown:

• – Finds tweets containing either phrase.

• – Excludes tweets mentioning "dog" or "dogs".

• – Excludes retweets for that fresh, original content.

Operator Precedence Pro Tip:
AND has higher precedence than OR, so always use parentheses to control your logic. For example:

• is interpreted as

• becomes When in doubt, add parentheses!

Some Handy Operators to Supercharge Your Searches:

• – Tweets from a specific user

• – Tweets containing a hashtag

• , – Tweets with media

• – Tweets in English

• , – Filter for retweets or replies

For a full list of operators, check out .

Bonus Tools:
If building complex queries feels daunting, try Twitter’s query builder tool to experiment with filters visually. For even more tips, there are plenty of guides on building high-quality filters for Twitter data.

With these query skills in your toolkit, you’re ready to slice and dice Twitter data like a pro.

Taking It Further: Pagination and Rate Limits

But what if you want to scoop up more than just a single page of tweets? Here’s how you can go pro:

• Pagination with next_token: Twitter’s API returns results in pages. Each response may include a next_token value in its meta field. As long as you see this token, grab it and add it to your next request as a query parameter, and you’ll get the next batch of tweets. Repeat until there’s no next_token and you’ve reached the end of the line.

• Respect the Rate Limit: Twitter sets a cap—usually 180 requests per 15 minutes for the Essential access level. That’s roughly one request every five seconds. To play nice and avoid errors, insert a short sleep (about five seconds) between calls if you're looping through lots of pages.
  
  

Example: Looping Through Multiple Rules

If you’re collecting tweets based on several search rules (think: “cats,” “dogs,” “parrots”), you might use a structure like this in Python (pseudocode for clarity):

import time
import pandas as pd

tweets_data = pd.DataFrame()
users_data = pd.DataFrame()

for rule in rules:
    query_parameters["query"] = rule["value"]
    query_tag = rule["tag"]

    json_response = connect_to_endpoint(endpoint_url, headers, query_parameters)
    tweets_data, users_data = process_twitter_data(json_response, query_tag, tweets_data, users_data)

    time.sleep(5)  # Wait to respect rate limits

    while "next_token" in json_response.get("meta", {}):
        query_parameters["next_token"] = json_response["meta"]["next_token"]
        json_response = connect_to_endpoint(endpoint_url, headers, query_parameters)
        tweets_data, users_data = process_twitter_data(json_response, query_tag, tweets_data, users_data)
        time.sleep(5)

What’s happening here?

• Empty pandas DataFrames are set up to store your tweet and user info.

• For each rule (or topic), you update the query and tag, make the API call, process the data, and pause for five seconds.

• If there’s a next_token, you’re not done! Keep paginating until you’ve collected all available tweets for that rule.

• The five-second nap between requests keeps you within the safe zone of Twitter’s rate limits.

Now you’re ready to harvest tweets like a seasoned data wrangler—no more leaving good data behind or running afoul of the API police.

Pro Tips: Beyond the Basics

In the example above, we've used the Recent Search endpoint to retrieve historical data from the past 7 days, but did you know you can use it to get tweets in almost real-time? By leveraging the parameter, you can fetch only the tweets that are newer than a specific tweet ID—perfect for keeping your finger on the pulse as new content rolls in. Check out Twitter’s official documentation for the nitty-gritty on this parameter.

Looking for a true real-time firehose? Consider using the Filtered Stream endpoint. While Recent Search is great for on-demand queries, the Filtered Stream lets you continuously collect tweets as they happen. It’s ideal for live monitoring, dashboards, or when you simply can’t miss a beat.

With these techniques, you’re not just searching the past—you’re tapping into the now.

Troubleshooting Access Issues: When You Can't Search All Tweets

Running into roadblocks with historical tweet searches? You're definitely not alone! If your API credentials or access level aren’t quite cutting it for full archive searches, here’s what you can do next:

• Double-Check Your Access Level: Most beginner or “Essential” Twitter API keys only allow access to the Recent Search endpoint (last 7 days) and won’t support a full historical search. Full-archive magic is reserved for accounts with Academic Research access.

• Look for Academic Access: To unlock /search/all, you’ll need Academic Research access. This is typically labeled as “Academic Research (For non-commercial use only)” in your Twitter Developer dashboard. Without it, you'll be limited to recent tweets.

• Try User Timelines for a Workaround: If you need tweets farther back—up to the last ~3,200 per user—consider pulling from user timelines instead. Many libraries (like twarc or Python Tweepy) let you fetch this data, although you can't specify arbitrary date ranges beyond what fits in the latest tweets.

• Check Your App's Bearer Token: Make sure you’re using the correct set of keys, especially if you have multiple Twitter developer projects or apps connected to your account. Sometimes, it’s just a token mix-up!

So, if the gates to tweet history seem closed, don’t worry. Explore the user timeline endpoints, snag as much data as you can, and always keep an eye on your access tier for future upgrades!

Free and Essential Access: Looking Back Isn't Quite That Simple

Before you start plotting that deep-dive into tweets from yesteryear, there are a few roadblocks you should know about. With most social media APIs, including Twitter, free or essential access comes with a pretty strict time limit: you can usually only retrieve tweets from the past seven days using the standard search endpoint. That means if you're hoping to rewind a few months—or years—you'll hit a wall unless you've secured academic or elevated permissions, which now require jumping through extra hoops (and, in many cases, aren't available at all).

Workarounds and Datasets

If you need older tweets, don’t despair—there are still some clever ways to get your hands on that data:

• Pre-collected Datasets: Organizations like DocNow curate public tweet datasets you can download and analyze. This is a popular option for researchers who need historic data but don't want to deal with access restrictions.

• Hydration Tools: Tools like twarc allow you to "hydrate" (i.e., fetch full tweet objects) using lists of tweet IDs from these public archives. You supply the IDs, and twarc pulls the text and metadata via the API, within the bounds of what your access level allows.
  
  

Command Line Power-Ups

While you won't be able to scour tweets from the distant past via the standard search endpoints, you can still:

• Retrieve up to the last 3,200 tweets from individual user timelines.

• Apply filters like date ranges (where supported by tools), but keep in mind these don't unlock older content—they just help sift through what you can access.
  
  

Heads Up About Access Levels

If you try to reach further back or use the /search/all endpoint without the proper academic credentials, expect to see errors telling you you're not authorized. Only users with approved academic projects have this capability, and that program isn’t accepting many new applicants.

In Short:
Unless you've got academic access, think of API data as more of a rearview mirror than a time machine. For historical deep-dives, public datasets and hydration tools are your best friends. For everything else, set your expectations (and scripts) to recent history only.

You’re now set up to get the most out of the Recent Search endpoint—and know where the boundaries are when your curiosity wanders back in time!

Common Errors When Retrieving Historical Tweets—and How to Fix Them

Just like assembling that Ikea bookshelf with one piece mysteriously leftover, fetching historical tweets can bring its own set of head-scratchers. Here are a few common pitfalls and what you can do about them:

1. Hitting the Seven-Day Search Limit

Without academic access, most APIs (including Twitter’s standard offerings) only let you search tweets from the past seven days. Trying to go further back? You’ll likely hit a “no results” wall—or receive a vague error message. If you need older data, consider using curated datasets from resources like DocNow Catalog and “hydrating” the tweet IDs (that’s just fetching the full tweet info using available tools).

2. Improper Query Syntax

It’s tempting to toss since: or until: right into your search query, but the proper way is to use start_time and end_time as parameters, not in the query string. Some tools expect these as dedicated options—so double-check the documentation if your search isn’t yielding results.

3. Authentication Mix-Ups

Many errors, like “Client Error” or “Unauthorized,” happen because of mismatched or missing Bearer Tokens. Make sure you’re using the exact token associated with the correct access level. For Academic Access endpoints, only the special credentials linked to an Academic Research project will do the trick.

4. API Endpoint & Access Mismatch

If you’re using endpoints locked behind higher access tiers (e.g., /search/all), but only have standard or essential access, you’ll be denied. Verify which endpoints your access covers. With Essential Access, for example, you’re limited to a chunk of recent history (often the latest 3200 tweets per user).

5. Common Pitfalls with Libraries & Tools

If you’re using tools like Twarc or other open-source libraries:

• Double-check that your command-line options match your access level

• For bulk timelines, leave off advanced flags like --use-search unless you’ve got academic credentials

• Use the flatten feature to break multi-tweet responses into single tweets, which can be easily imported elsewhere (think: straight to your MongoDB, for those with serious collection goals)
  
  

Quick Troubleshooting Checklist

• Make sure your authentication keys are correct and valid for the desired endpoint

• Double-check your query parameters for typos or misplacement

• For more data, consider combining public datasets with tools that let you hydrate tweet IDs

• When all else fails, consult the documentation or try sample code from the library maintainers’ tutorials

With these tips, you’ll sidestep the most common snags and keep your data pipeline flowing smoothly.

Digging Into Historical Tweets: Alternative Methods When Access Is Restricted

So, what if you’re on the hunt for tweet archives but your usual endpoints are throwing up roadblocks? No worries—let’s explore your options for gathering historical Twitter data when API permissions aren’t playing nice.

Pre-Collected Datasets: The Shortcut You Need

If you want a quick start, curated datasets are your friend. Websites like DocNow Catalog (https://catalog.docnow.io/) offer collections of tweet IDs on a wide range of topics—from major events to memes and everything in between. While these datasets don’t include the full tweet content, you can use a process called “hydration” (think of it as adding water back to dehydrated soup—except with tweets and metadata) to restore those tweet IDs to their full glory, provided the tweets are still live.

Hydrating Tweets: The Power Tool Approach

To hydrate tweet IDs, you’ll need a third-party tool. Twarc is a community favorite for the command-line crowd. Once installed, simply point it to your list of tweet IDs and let it fetch as much data as your current API access allows. Even if you’re locked out of “academic” endpoints, most hydration tools will still work—just at whatever rate limit is available to you.

Getting Started With Twarc (and Friends)

If you’re new to all this, don’t sweat it. There are plenty of beginner-friendly tutorials to walk you through installing and using tools like Twarc. Video walkthroughs and written guides cover everything from basic setup to advanced filtering. It’s a great way to get hands-on with historical data while sharpening your command-line ninja skills at the same time.

Armed with these strategies, you can keep your Twitter research rolling—even when the usual doors are closed. Just remember: hydrated tweet data will only include tweets that are still public, so you might run into the occasional missing post.

Paging Through Tweets: How Pagination Works

Here's a quick reality check: Twitter isn’t going to send you all the tweets in one giant avalanche. Instead, results arrive in handy, manageable “pages,” with the most recent tweets always coming in first. But what if you want to dig deeper and see more than just that first batch?

Enter pagination tokens—your key to flipping through the rest of the results. After each API call, you'll receive a response that may include a next_token in the "meta" section. This token acts like a bookmark, telling Twitter where you left off.

How does this look in action?

• Make your initial request to the endpoint.

• If the response includes a next_token, add it as a parameter to your next request.

• Repeat: With each new response, keep grabbing the next_token and using it for your next call.

• Stop when the next_token disappears—congratulations, you've hit the end of the available results!

Tip: To be a good API citizen (and not get rate-limited), it's smart to add a brief pause—like a five-second sleep—between requests.

And there you have it: paginated scrolling through tweet history, all with a few tweaks to your request URL and a watchful eye on those tokens.

Pro Tips for Real-Time Tweet Collection

A few words to the wise before you go turbo with real-time tweet fetching: Not all tweets are created equal—or accessible! The Recent Search endpoint only returns publicly available tweets, so don’t expect to channel your inner secret agent and uncover private messages.

To avoid drowning in irrelevant data or missing tweets that matter, keep your query rules as clear and targeted as possible. Here’s a workflow to help you nail it:

• Craft your queries with care—think laser focus over fishing net.

• Run your initial searches and review the results.

• Tweak and fine-tune your queries based on what you find.

• Rinse and repeat until you’re seeing the tweets that matter most.

And a quick pro tip for all you programming fans: If you’re tracking tweets about the Swift programming language, make your queries smart enough to skip over chatter about Taylor Swift. The devil’s in the details—and in the hashtags!

This thoughtful approach means you’ll collect the right tweets without losing gems in a flood of noise.

Unlocking Real-Time Tweets with since_id

Curious about keeping your search results fresh? That’s where the since_id parameter comes in handy. By adding since_id to your request, you tell the Recent Search endpoint: “Only show me tweets newer than this specific tweet ID.” This is perfect for polling Twitter for the latest updates without getting swamped with repeats. Just save the most recent tweet ID from your last batch and use it in your next query—voilà, you’re fetching only brand new content!

Ready to up your game? Check out Twitter’s official documentation for the full scoop on since_id and other advanced parameters.

But Wait—There’s More: The World of Twitter API Endpoints

While the Recent Search endpoint is a fan favorite, the Twitter API is a sprawling metropolis of endpoints, each offering unique ways to collect or act on data. Whether you’re a data scientist, a developer, or just dangerously curious, it pays to know what’s out there.

Some endpoints let you collect data—think tweets, user profiles, or tweet volumes. Others let you take action—post or delete tweets, like and unlike, or follow and unfollow accounts. All these endpoints are represented by different URLs, and each one comes with its own rules around rate limits and access levels.

Head to your Developer Portal and look under Twitter API v2 to get the full rundown. There, you’ll find a buffet of endpoints with handy links to documentation, info on rate limits, and special attributes (like maximum query length). Many endpoints are available across all access levels, but rate limits—how much data you can pull in a given time—will vary depending on your level.

For those keen on data, pay special attention to endpoints like:

• Recent Search: Fetch tweets from the last 7 days.

• Filtered Stream: Monitor tweets in real-time as they post.

• User Tweet Timeline: Grab recent tweets from a specific user.

• User Lookup: Get user profile information in bulk.

You can always check the official API roadmap to see which endpoints are in the works and when you’ll be able to try them out.