We evaluate the context quality of the four major context providers in the LLM market, JinaAI, Tavily, AskNews, and Exa, using Ragas 📐 metrics. We also compare the data structures that each of the services return and how well they contextualize LLMs. Finally, we measure the speed 🏎 and the cost 💸of the response from each provider.
There is a distinct difference in quality and speed observed using the Ragas
answer_correctness ✅ and
context_precision💎 metrics.
AskNews out-performed
Exa and
JinaAI on most query types for context precision by up to 78%. The
JinaAI, and Exa contexts are frequently missing the latest information and context. For example, publication date and entity extractions are not available in
JinaAI or
Tavily, which result in an LLM that has no ability of answering time-sensitive questions, such as
“Who won the Yankees game?”
📌 Note: Check out the Google Colab to see the in-depth analysis and full output. Please contact the author at contact at emergentmethods.ai if you would like to request a change to Colab or to this document.
📌 Edit 1:
Tavily reminded us that for news, we should be using a
topic="news" parameter, therefore — the two `latest news` queries for Tavily should be considered inaccurate in the attached table, until we can re-run the results. This parameter was not used because Tavily does not mention it in their documentation.
The saying originates from the early days of machine learning, training regressors, classifiers, and neural networks. But it has never been more relevant than today, in the age of LLMs with growing context windows and advanced reasoning capabilities.
What you put into the context window matters. Many developers are learning this the hard way - it doesn’t matter how smart their LLMs are, if the context is poorly engineered, the output is going to be useless. Even large corporations are struggling to build context, with Google’s AI search debacles recommending to “eat rocks” and “use Elmer’s glue to keep cheese on pizza.”
Four contenders have emerged, all offering a one-stop-shop for web context. “Put a query in, get the updated web context back!” Great, so they have done the context engineering for us!? But wait — how do they perform? We aim to compare and contrast them here.
answer_correctness metric from
Ragas, an open-source project aimed at evaluating retrieval and LLM output. This metric uses semantic similarity between the generated answer and the ground truth, as well as factual similarity, to determine the correctness of the answer.context_precision metric from Ragas, which evaluates whether all of the ground-truth relevant items present in the
contexts are ranked higher or not.The full notebook is available in the companion Google Colab notebook. However we will walk through the overview here. First, we install the libraries and import them:
pip install asknews tavily-python langchain langchain_openai ipykernel ragas python-rapidjson requests==2.31.0
import requestsimport timeimport osfrom asknews_sdk import AskNewsSDKfrom langchain_openai import ChatOpenAIfrom langchain_core.prompts import ChatPromptTemplateimport rapidjsonfrom tavily import TavilyClientfrom IPython.display import Markdown, displayfrom datasets import Datasetfrom ragas.metrics import context_precision, answer_correctnessfrom ragas import evaluatefrom google.colab import userdatafrom collections import defaultdict
Then we instantiate the clients (and wrap the JinaAI request call):
# Head over to each of the services to get your credentials# - [JinaAI](https://jina.ai/reader/#apiform)# - [Tavily](https://tavily.com/)# - [AskNews](https://my.asknews.app/plans)model = "gpt-3.5-turbo-16k"# Tavilytavily = TavilyClient(api_key=userdata.get('TAVILY_KEY'))# Jina "client"jina_apikey = userdata.get('JINAAI_KEY')def call_jina(query): url = f"https://s.jina.ai/{query}" headers = { "Accept": "application/json", "Authorization": f"Bearer {jina_apikey}" } response = requests.get(url, headers=headers) return response.json()# AskNewsask = AskNewsSDK( client_id=userdata.get('ASKNEWS_CLIENT_ID'), client_secret=userdata.get('ASKNEWS_SECRET'), scopes=["news"])# Shared OpenAI client llm = ChatOpenAI( model=model, temperature=0.0, api_key=userdata.get('OPENAI_KEY'), streaming=False,)# Ragas looks for the key in the envos.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_KEY')
JinaAI and Tavily don’t provide any direct prompt-optimized strings, so we need to massage their responses into a context string for the LLM:
def build_tavily_context(tavily_response): context = "" contexts = [] for i, obj in enumerate(tavily_response["results"]): text = f"<doc>\n[{i+1}]\n{obj['title']}\n{obj['content']}\n</doc>\n\n" context += text contexts.append(text) return context, contextsdef build_jina_context(jina_response): context = "" contexts = [] if "data" not in jina_response or jina_response["data"] == None: return context, contexts for i, article in enumerate(jina_response["data"]): text = f"<doc>\n[{i+1}]\n{article['title']}\n{article['description']}\n</doc>\n\n" context += text contexts.append(text) return context, contextsdef build_asknews_context(input_string): docs_list = input_string.strip().split('</doc>') docs_list = [doc.strip() + '</doc>' for doc in docs_list if doc.strip()] return docs_listdef print_md(string): display(Markdown(string))
Now we can define a prompt that takes a {query} and a {news_context}, and asks the LLM to answer the {query} using the {news_context}:
def get_llm_response(query, news_context): current_time = datetime.now(tz=timezone.utc).strftime("%B %d %Y %H:%M") prompt = ChatPromptTemplate.from_messages( [ ( "system", "A conversation between a helpful Assistant and a User. The User asks the Assistant a question and gives the Assistant a set of documents that might help the Assistant to respond. The Assistant aims to answers the User question as concisely as possible.", ), ("human", f"Given the current time is {current_time} the following documents:\n\n----START DOCUMENT CONTEXT----{{news_context}}-----END DOCUMENT CONTEXT----\n\nPlease answer my question:\n\n{{question}}"), ] ) agent = prompt | llm response = agent.invoke({ "question": query, "news_context": news_context }) return response.content
Almost there, let’s define our queries that we will feed to each of the contenders:
queries = [ { "type": "latest news 1", "query": "What was the score of the Yankees game?", "ground_truth": "The New York Yankees beat Los Angeles Dodgers, with the final score being 6-4 in favor of the Yankees." }, { "type": "latest news 2", "query": "What is the current price of Bitcoin?", "ground_truth": "The current price of bitcoin is $69,354.00." }, { "type": "general web 1", "query": "Who won the SuperBowl this year?", "ground_truth": "The Kansas City Chiefs won the SuperBowl." }, { "type": "general web 2", "query": "Who is Tom Cruise married to right now?", "ground_truth": "Tom Cruise is not currently married." }, { "type": "knowledge search 1", "query": "What are the highlights of the latest Node.js version?", "ground_truth": "Highlights include require()-ing ESM graphs, WebSocket client, updates of the V8 JavaScript engine" }, { "type": "knowledge search 2", "query": "What is the ice sheet area and thickness of Greenland?", "ground_truth": "Greenland's ice sheet has an area of 1.7 million square kilometers and an average thickness of 2.3 kilometers (1.4 miles)." }, { "type": "google-esque 1", "query": "What is the current Fed interest rate?", "ground_truth": "The Federal Reserve's current interest is between 5.25% and 5.50%." }, { "type": "google-esque 2", "query": "What's the cost of living in Amsterdam?", "ground_truth": "A family of four's estimated monthly costs are €5,632, while a single person's estimated monthly costs: €3,317." },]
⚠️
Please note:
these questions are all time-sensitive, and as such, the ground truth will change. In some cases, like
latest_news
it will change by the hour. In other cases, it may take longer (hopefully much longer for the Greenland ice sheet thickness question). If you re-run the google colab, you will need to update some of these ground truths accordingly.
Let’s wrap up the
answer_correctness and
context_precisionfrom Ragas:
def get_answer_correctness(query, contexts, answer, ground_truth): data_sample = { "question": [query], "contexts": [contexts], "answer": [answer], "ground_truth": [ground_truth] } dataset = Dataset.from_dict(data_sample) score = evaluate(dataset,metrics=[answer_correctness, context_precision], llm=llm) return score['answer_correctness'], score['context_precision']
Looking good — all that is left is to feed the queries to the contenders and collect the results:
scores = { 'asknews_scores': {'answer_correctness': defaultdict(list), 'context_precision': defaultdict(list), 'time': defaultdict(list) }, 'jina_scores': {'answer_correctness': defaultdict(list), 'context_precision': defaultdict(list), 'time': defaultdict(list) }, 'tavily_scores': {'answer_correctness': defaultdict(list), 'context_precision': defaultdict(list), 'time': defaultdict(list) },}trials = 3for trial in range(trials): for query in queries: query_str = query["query"] query_type = query["type"] gt = query["ground_truth"] query_hash = str(hash(query_str)) # AskNews start = time.time() asknews_context = ask.news.search_news( query_str, return_type="string", n_articles=5, strategy="latest news" if "latest news" in query_type else "news knowledge" ).as_string scores['asknews_scores']['time'][query_type].append(time.time() - start) contexts = build_asknews_context(asknews_context) asknews_llm = get_llm_response(query_str, asknews_context) asknews_score, precision = get_answer_correctness(query_str, contexts, asknews_llm, gt) scores['asknews_scores']['answer_correctness'][query_type].append(asknews_score) scores['asknews_scores']['context_precision'][query_type].append(precision) # JinaAI start = time.time() jina_response = call_jina(query_str) scores['jina_scores']['time'][query_type].append(time.time() - start) jina_context, contexts = build_jina_context(jina_response) jina_llm = get_llm_response(query_str, jina_context) jina_score, precision = get_answer_correctness(query_str, contexts, jina_llm, gt) scores['jina_scores']['answer_correctness'][query_type].append(jina_score) scores['jina_scores']['context_precision'][query_type].append(precision) # Tavily # For advanced search: start = time.time() response = tavily.search(query=query_str, search_depth="advanced") scores['tavily_scores']['time'][query_type].append(time.time() - start) tavily_context, contexts = build_tavily_context(response) tavily_llm = get_llm_response(query_str, tavily_context) tavily_score, precision = get_answer_correctness(query_str, contexts, tavily_llm, gt) scores['tavily_scores']['answer_correctness'][query_type].append(tavily_score) scores['tavily_scores']['context_precision'][query_type].append(precision) # print the results print_md(f"## {query_str} - {query_type} - TRIAL {trial}") print_md(f"### AskNews") print_md("#### Ragas Report") print_md(f"{asknews_score}") print_md("#### LLM Response") print_md(asknews_llm) print_md(f"#### Context") print(asknews_context) print_md(f"### JinaAI") print_md("#### Ragas Report") print_md(f"{jina_score}") print_md("#### LLM Response") print_md(jina_llm) print_md(f"#### Context") print(jina_context) print_md(f"### Tavily") print_md("#### Ragas Report") print_md(f"{tavily_score}") print_md(f"#### LLM Response") print_md(tavily_llm) print_md(f"#### Context") print(tavily_context)
As shown in the table below, the answer correctness is, on average, 40% better for AskNews than it is for Tavily. Speed is significantly faster in AskNews, averaging 0.43 seconds per retrieval, compared to 8.1 seconds for JinaAI.
Keeping in mind the stochastic nature of LLMs, we ran each query type 3 times and averaged them to get the values shown below.
While it is unknown which methods JinaAI and Tavily use for their scraping/retrieval, AskNews has publicly presented their methods, indicating that their search infrastructure relies on Qdrant. This factor likely contributes greatly to the retrieval speed and accuracy, since every query needs to be fetched by the Qdrant database, using hybrid search on quantized vectors.
Table 1 shows that the ground truth answer is more often contained in the documents for AskNews, compared to JinaAI and Tavily. This is highlighted by the 78% improvement in performance for AskNews compared to JinaAI.
We take a closer look at how the context is retrieved from each service, and what impact that has on the LLM trying to use the context to answer the question.
The context from AskNews includes a broad range of details, including published date, source, sentiment, entity extractions, and full summaries.
Context from AskNews (truncated, see Google colab for full output):
<doc>[1]:Title: Yankees 10-1 Royals (12 Jun, 2024) Final ScoreSummary: Aaron Judge hit his 25th home run of the season, a 436-foot, two-run drive into the fountain behind the center-field seats, as the New York Yankees defeated the Kansas City Royals 10-1 on Tuesday night, securing their 11th win in 14 games.Source: ESPNPublished: June 12 2024 03:46Person: Aaron JudgeNumber: 14, 11th, 25thQuantity: 10-1, two-run, 436-footLocation: center-fieldSports: New York Yankees, Kansas City RoyalsDate: TuesdayClassification: SportsSentiment: 1Reporting voice: Objective</doc><doc>[2]:Title: Yankees vs Royals final score, results: New York wins third straight in 10-1 routSummary: The New York Yankees defeated the Kansas City Royals 10-1, winning their third straight game. The Yankees' offense was led by Aaron Judge and Giancarlo Stanton, who both hit home runs, and Austin Wells, who hit a three-run homer. Marcus Stroman pitched 5.2 innings, allowing no runs and lowering his season ERA to 2.82. The Yankees have now won seven straight games away from Yankee Stadium and have a 26-11 record on the road, the best in MLB.Source: The Sporting NewsPublished: June 12 2024 02:55Sports: MLB, New York Yankees, Yankees, Kansas City RoyalsQuantity: 10-1, 5.2 innings, 26-11, 2.82Person: Marcus Stroman, Austin WellsLocation: Yankee StadiumClassification: SportsSentiment: 1Reporting voice: Objective</doc><doc>[3]:Title: Yankees Launch Three Home Runs In 10-1 Rout Of RoyalsSummary: The Yankees defeated the Royals 10-1 on Tuesday night, with three home runs from Judge, Stanton, and Wells. The team scored four runs in the fourth inning, capped off by Wells' three-run shot. Judge and Stanton added home runs in the seventh, and the Yankees' pitching staff was led by Stroman, who pitched 5.2 scoreless innings. Marinaccio made his return to the team and pitched 2.1 innings, striking out three and allowing one walk. The Yankees will continue their four-game series with the Royals on Wednesday night.Source: WhatsNew2dayPublished: June 12 2024 02:48Sports: Yankees, RoyalsQuantity: 10-1, one walk, 2.1 innings, 5.2Date: Tuesday, WednesdayPerson: Stroman, Stanton, Wells, Judge, MarinaccioTime: fourth inning, seventhNumber: threeClassification: SportsSentiment: 1Reporting voice: Objective</doc><doc>[4]:Title: Yankees launch three home runs in 10-1 rout of RoyalsSummary: The New York Yankees defeated the Kansas City Royals 10-1 on Tuesday night, with a strong performance from their offense and pitching. The Yankees used the long ball to score runs, with Austin Wells hitting a three-run homer and Aaron Judge and Giancarlo Stanton adding solo shots. Marcus Stroman pitched 5.2 scoreless innings, and Ron Marinaccio earned the win in relief. Anthony Rizzo returned to the lineup after missing two games and had a solid game, making a great catch and reaching base on an error. Marinaccio was named the game MVP.Source: YahooPublished: June 12 2024 02:48Sports: Yankees, Kansas City Royals, New York YankeesQuantity: 5.2, 10-1Date: TuesdayPerson: Ron Marinaccio, Giancarlo Stanton, Austin Wells, Marcus Stroman, Anthony RizzoTitle: MVPClassification: SportsSentiment: 1Reporting voice: Objective</doc><doc>[5]:Title: Yankees beat the Kansas City Royals 4-2 at Monday's night gameSummary: The New York Yankees' Juan Soto returned to the lineup after missing a three-game series against the Los Angeles Dodgers due to left forearm inflammation. Soto went 1 for 3 with a walk as the designated hitter in a 4-2 win over the Kansas City Royals. He is batting .318 with 17 home runs, 53 RBIs, and a 1.024 OPS. Manager Aaron Boone said the team is easing Soto back into the outfield and wants to make sure he is fully healthy before returning to the field. The Yankees also gave Aaron Judge a night off, and Anthony Rizzo was left out of the lineup for a second straight game. Boone said the team is providing Rizzo with a mental break and time to work on his hitting mechanics.Source: NBC New YorkPublished: June 11 2024 18:59Sports: Los Angeles Dodgers, New York Yankees, Kansas City Royals, YankeesMedicine: left forearm inflammationQuantity: .318, 53 RBIs, 1.024 OPS, 1 for 3, 4-2, 17 home runsTitle: ManagerPerson: Aaron Boone, Aaron Judge, BooneClassification: SportsSentiment: 1Reporting voice: Objective</doc>
This contains sufficient information for the LLM to answer the question.
LLM Response using AskNews context (
answer_correctness: 0.53):
The score of the Yankees game was 10–1.
The context from JinAI contains a broad range of historical Yankees games, but none of them are the latest Dodgers games. The documents have no publication date, making it even more difficult for the LLM to answer a time-sensitive question.
Join Medium for free to get updates from this writer.
Context from JinaAI:
<doc>[1]Yankees Game Recaps - Pinstripe AlleyA franchise rookie record 14 strikeouts ... quartet of Juan Soto hits led <strong>the</strong> <strong>Yankees</strong> to another win. ... Aaron Judge stayed hot, Giancarlo Stanton continued his excellent season, and <strong>the</strong> <strong>Yankees</strong> kept up their winning ways. ... Clarke Schmidt worked eight innings for the first time in his career to complete the sweep. ... A four-hit <strong>game</strong> for <strong>the</strong> <strong>Yankee</strong> captain and a scoreless Stroman outing ...</doc><doc>[2]Yankees Scores: Scoreboard, Results and Highlights | New York YankeesThe official scoreboard of the New York <strong>Yankees</strong> including Gameday, video, highlights and box <strong>score</strong>.</doc><doc>[3]MLB Scores: Scoreboard, Results and Highlights | MLB.comGet up-to-date MLB <strong>scores</strong> from today’s <strong>games</strong>, as well as <strong>scores</strong> from all the 2023 MLB season <strong>games</strong>.</doc><doc>[4]New York Yankees score, schedule & standingsWhen the match starts, you will be able to follow New York Yankees vs Los Angeles Dodgers scores, updated live as the match progress. Statistics are updated at the end of the game. New York Yankees previous match was against Minnesota Twins in MLB, match ended with result <strong>9 - 5</strong> (New York Yankees ...</doc><doc>[5]New York Yankees | News & Stats | BaseballTrending News & Rumors for Football, Basketball, Baseball, Hockey, Soccer & More</doc>
This results in an LLM that is unable to provide the score. We should also note that on many queries to Jina, we received empty responses and had to run a retry.
LLM response using JinaAI context (
answer_correctness: 0.21):
Based on the given documents, the score of the Yankees game is not explicitly mentioned.
The context from Tavily also contains a broad range of historical Yankees games. One of the documents says something about the Dodgers, but is not relevant to the most recent game. The documents have no publication date, making it even more difficult for the LLM to answer a time-sensitive question.
Context from Tavily:
<doc>[1]New York Yankees Scores, Stats and Highlights - ESPNVisit ESPN for New York Yankees live scores, video highlights, and latest news. Find standings and the full 2024 season schedule. ... and the New York Yankees finished a three-game sweep with a 5 ...</doc><doc>[2]Astros vs. Yankees final score, results: Houston sweeps New York for ...Astros vs. Yankees live updates, highlights from ALCS Game 4 (All times Eastern) 12:08 p.m. -- After taking a ball, Judge grounds out to Pressly to end it.The Yankees win their second straight AL ...</doc><doc>[3]Official New York Yankees Website | MLB.compic.twitter.com/FAUuxi36qCFormer Yankees manager Joe Torre dons the uniform again and walks out to the mound to take Carlos Rodón out of the gameCarlos Rodón speaks on the moment that iconic Yankees manager Joe Torre walked to the mound to pull him from the gameOswaldo Cabrera clubs a long home run to right field to give the Yankees a 1-0 lead in the 3rdCarlos Rodón strikes out five over 5 2/3 no-hit innings and gets congratulated by Hall of Fame manager Joe Torre in his spring start against the PhilliesAaron Boone discusses Aaron Judge preparing to return from an abdominal injury, saying he should be back in the lineup on Wednesday The data behind Oswaldo Cabrera's home runThe data behind Carlos Rodón's outingPregame in Paradise 🌴 pic.twitter.com/HKSQ0BYRaTSpencer Jones, Addison Barger and more take the field as the top prospects from the Blue Jays and the Yankees face off at Spring BreakoutClayton Beeter strikes out two across four innings in his Spring Training outing against the Red SoxWill Warren strikes out five across 2 2/3 innings in his Spring Training outing against the Red Sox RelatedHall of Fame manager Joe Torre leads a roster of guest instructors that includes Yankee legends Ron Guidry, Andy Pettitte and CC Sabathia, Bryan Hoch writes in his Yankees Beat newsletter. After pitching 5 2/3 no-hit innings, Carlos Rodón was taken out by legendary Yankees manager Joe Torre. Watch on MLB.TV or listen on At Bat: Aaron Judge is back in the lineup tonight vs. Pittsburgh.</doc><doc>[4]Yankees Scores: Scoreboard, Results and HighlightsThe official scoreboard of the New York Yankees including Gameday, video, highlights and box score. Tickets. Individual Game Tickets Promotions ... Yankees Game Recaps Yankees Cut4 Yankees Manager Postgame Yankees Reviews Yankees HOPE Week Yankees Curtain Calls Most Popular Yankees Productions Yankees Podcasts MLB Network.</doc><doc>[5]Yankees vs Dodgers final score, results: New York limps to 2-1 loss in ...06-07-2024 • 10 min read. Getty Images. The New York Yankees and Los Angeles Dodgers put on quite the show Friday night in the Bronx, resulting in the latter snapping New York's eight-game ...</doc>
The LLM is unable to answer.
LLM response with Tavily context (
answer_correctness: 0.30):
Based on the provided documents, the score of the Yankees game is not explicitly mentioned.
The context from Exa seems to be updated on the latest Orioles Yankees gam, but it does not contain the score, so the LLM cannot answer.
Context from Exa:
<doc>[1]Title:Orioles vs. Yankees - MLB Game Summary - June 11, 2017 | ESPNHighlights:3 5 4th Holliday singled to shallow center, Hicks and Judge scored. 3 7 6th Judge homered to left center (496 feet). 3 8 6th Castro homered to left (392 feet), Holliday scored. 3 10 7th Hicks doubled to deep right center, Torreyes and Gardner scored. 3 12 7th Judge homered to right center (401 feet), Hicks scored.Published:2017-06-11</doc><doc>[2]Title:Mets vs. Yankees - MLB Box Score - June 11, 2019 | ESPNHighlights:Mets Hitting Batting 2BRosario (11, Tarpley); McNeil (15, Paxton) HRGómez (3, 5th inning off Adams 0 on, 2 Out); Davis (7, 4th inning off Adams 1 on, 0 Out); Alonso (22, 1st inning off Paxton 2 on, 0 Out) RBIGómez (9), Ramos (35), Hechavarria (14), Rosario (35), Conforto (30), Davis 2 (20), Alonso 3 (49) 2Out RBIGómez, Hechavarria, Rosario GIDPRamos Team LOB7 Team RISP6-10 (Davis 1-1, Hechavarria 1-2, Gómez 0-1, Alonso 1-1, Conforto 1-1, Rosario 1-2, Ramos 1-2) Yankees Hitting Batting HRGardner (10, 9th inning off Gagnon 0 on, 0 Out) RBIGardner (25), Urshela (30), Frazier (33), Torres (33) Team LOB8 Team RISP2-8 (Torres 1-1, Sánchez 1-2, Voit 0-1, Frazier 0-3, Urshela 0-1) Fielding DP1 (Urshela-LeMahieu-Voit) EUrshela (9, pop up) Mets Pitching Pitching First-pitch strikes/Batters FacedVargas 16/27; Familia 3/5; Lugo 2/3; Gagnon 1/4 Called strikes-Swinging strikes-Foul balls-In play strikesVargas-12-9-17-22; Familia-4-3-0-2; Lugo-4-4-2-0; Gagnon-4-2-1-3 Ground Balls-Fly BallsVargas 6-9; Familia 1-0; Gagnon 2-1 Game ScoresJ Vargas 47 Yankees Pitching Pitching HBPFrazier (by Tarpley) First-pitch strikes/Batters FacedPaxton 8/17; Tarpley 3/5; Cessa 1/3; Adams 10/19 Called strikes-Swinging strikes-Foul balls-In play strikesPaxton-9-3-13-14; Tarpley-1-3-7-2; Cessa-1-2-6-3; Adams-8-5-25-13 Ground Balls-Fly BallsPaxton 3-4; Tarpley 1-0; Cessa 3-0; Adams 4-6 Game ScoresJ Paxton 19Published:2019-06-11</doc><doc>[3]Title:Didi, arms lead Yanks to victory | 06/12/2018Highlights:© 2023 MLB Advanced Media, LP. All rights reserved.Published:2018-06-12</doc><doc>[4]Title:Aaron Judge Stats, Fantasy & NewsHighlights:100 in 2019 (Gardner, Gregorius, Sánchez)..is the seventh player drafted and signed by the Yankees to hit at least 100HR with the club (Gardner, Posada, Jeter, Mattingly, Munson and Pagliarulo)...Hit a game-tying solo HR with two outs in the eighth inning on 8/31 vs. Oakland..was his seventh career HR to tie the game or give the Yankees the lead in the eighth inning or later and third of the season (also a go-ahead solo HR in the top of the 11th inning on 7/5 at Tampa Bay and a go-ahead two-run HR in the bottom of the eighth on 7/16 vs. Tampa Bay)...Reached base four times in Game 1 of 9/12 doubleheader at Detroit, scoring a career-high-tying 4R and recording a season-high-tying 3BB..hit a two-run HR in Game 2...Started in RF in each of the Yankees' nine postseason games..hit a two-run HR in the fourth inning of ALCS Game 2 at Houston..marked his eighth career postseason HR (in his 23rd career playoff game), becoming the fourth Yankee to hit 8HR within his first 23 postseason games (also Lou Gehrig-9HR, Mickey Mantle-8HR and Bernie Williams-8HR). In 112 games with the Yankees (88 starts in RF, 19 at DH, 1 in CF), hit .278 (115-for-413) with 77R, 22 doubles, 27HR, 67RBI, 76BB and 6SB…the Yankees went 21-6 when he homered in 2018...Was a finalist for the AL Gold Glove Award in RF...Hit .352/.471/.699 (69-for-196) with 47R, 14 doubles, 18HR, 45RBI and 20 multi-hit games in 56 contests at Yankee Stadium…did not go consecutive home games without a hit over a 59-game stretch at Yankee Stadium (9/17/17-7/26/18)…hit safely in 17 consecutive home games from 9/17/17-4/16/18…had at least 1RBI in a franchise-record 14 straight home games from 9/18/17-4/7/18…was tied for the fourth-longest such streak in the Majors since RBI became an official statistic in 1920...Appeared in each of the Yankees' first 56 games of the season (3/29-6/4 G2)...Hit 25HR before the All-Star break, becoming the fourth player to hit at least 25HR before the break in two different seasons as a Yankee and just the second to do so in consecutive years, joining Roger Maris (1960-61), Mickey Mantle (1956 and '61) and Jason Giambi (2003 and '06)...Had a career-long 14-game hitting streak from 9/17/17-3/29/18…had an eight-game streak of drawing at least one walk from 9/24/17-3/29/18, matching the longest streak of his career...Made his first Major League start (and appearance) in CF in 3/31 loss at Toronto...Reached base safely in 23 of his 25 games in April (all except 4/27 and 4/29 at Los Angeles-AL), including each of his first 21 games of the month...Was 3-for-3 with an HR off Red Sox starter Chris Sale on 4/10…entered 0-for-12 with 10K in his career vs. Sale…became the first player to record 3H off Sale in one game since he joined the Red Sox before the 2017 season…prior to Judge, the last player with a three-hit game vs. Sale was Detroit's Miguel Cabrera, who went 4-for-4 with two solo HRs against Sale on 9/5/16 at Chicago-AL...Celebrated his 26th birthday on 4/26…his 63 career homers prior to turning 26 rank 13th in franchise history and are fifth-most in the last 50 years (Mattingly-94, Murcer-77, Jeter-70, Sánchez-71)...Had at least 1RBI in seven straight games from 5/6-13 (14RBI), tying the longest RBI streak of his career (also 7G: 9/24-30/17)...Hit game-winning two-run HR in the 13th on 6/6 at Toronto…was his first career extra-inning HR and was the latest Yankees HR to break a 0-0 tie since Alex Rodriguez's 15th-inning two-run "walk-off" HR on 8/7/09 vs. Boston...Hit game-winning solo HR in the eighth inning on 6/9 at the Mets…was the third straight game the Yankees hit a go-ahead HR in the eighth-inning-or-later, their first time doing so since 8/29-31/77 (three straight - Chambliss, Rivers and Nettles)…Judge hit two of those homers after having just one go-ahead HR in the eighth-inning or-later over his first 239 career games...Started in LF and went 1-for-2 with 1R, 1HR, 1RBI and 1BB in the AL's All-Star Game win on 7/17…his solo HR off Scherzer to lead off the second inning was the first of 10HR hit in the game and the first All-Star homer by a Yankee since Jason Giambi in 2003…became the third Yankee to homer in an All-Star Game at age 26-or-younger, joining Joe DiMaggio (1936) and Mickey Mantle (1955 and '56)…was selected by fans to his second straight AllStar team…is one of five position players drafted by the Yankees to make the All-Star team multiple times as a Yankee (also Thurman Munson, Don Mattingly, Derek Jeter and Jorge Posada)…received a fan-elected starting assignment for the second straight year and is the first Yankee to start at least two straight ASGs since Robinson Canó (2010-13) and first Yankees OF to start two straight since Curtis Granderson (2011-12)...Was 1-for-1 with 1R in 7/26 win vs. Kansas City before being removed from the game in the fourth for PH (Andújar) after being hit by a pitch in the first inning…underwent an MRI at NewYork-Presbyterian Hospital and was diagnosed with a chip fracture of the right wrist (ulnar styloid bone)…was placed on the 10-day disabled list with the injury from 7/27-9/14 (missed 45 team games)…in a simulated game at Yankee Stadium on 9/17, had 11 live at-bats off of RHP A.J. Cole, RHP Chance Adams and minor-league LHP Phillip Diehl...Hit the Yankees' 264th home run of the season with a solo HR in the eighth inning of 9/28 win at Boston, tying the single-season Major League record (1997 Seattle Mariners)…was his first HR since 7/21 vs. the Mets, snapping a 15-game homerless stretch (49AB) and 11-game stretch without a homer since returning from the D.L...Hit .421/.500/.947 (8-for-19) with 6R, 1 double, 3HR, 4RBI and 3BB in five postseason games…homered in each of the first 3G, joining Hank Bauer (1958 World Series Games 1-3) as the only Yankees to homer in each of the team's first three postseason games in a year (note: Johnny Mize homered in 1952 World Series Games 3-5 after missing the first 2G of the Series)…became the third player to begin a postseason with 3G with at least 2H and 1HR, joining St. Louis' Matt Carpenter in 2014 and the Yankees' Hank Bauer in 1958...Hit a two-run HR in the first inning of the Wild Card Game…with an exit velocity of 116.1 mph, was briefly the hardest-hit postseason HR of the Statcast era (since 2015), until Stanton's HR surpassed it later in the game...Set a postseason career high in hits in ALDS Game 1 at Boston, going 3-for-5 with a ninth-inning solo HR. In 155 games with the Yankees (141 starts in RF, 10 at DH), hit .284 (154-for-542) with 128R, 24 doubles, 3 triples, 52HR, 114RBI, 127BB and 9SB... according to MLB Statcast, had a 94.9 mph average exit velocity on balls in play, highest among all Major League hitters…his solo HR on 6/10 vs. Baltimore had an exit velocity of 121.1 mph, setting a record for hardest hit HR in the Statcast-era (since 2015)…received his first career Silver Slugger Award... Placed second in AL Most Valuable Player voting, totaling 279 points (2 first place votes, 27 second place votes, 1 third place vote)…is the second rookie all time to finish second in MVP voting, joining Mike Trout in 2012 (won by Miguel Cabrera)…only two rookies (in either league) have won the award: Boston's Fred Lynn in 1975 and Seattle's Ichiro Suzuki in 2001 both won the AL MVP as rookies... Was unanimously named AL Jackie Robinson "Rookie of the Year" by the BBWAA…was the ninth Yankee to win the award and second by a unanimous vote (also Derek Jeter in 1996)…was the only player listed on every ballot... Was named MLB "Rookie of the Year" by Baseball America, the first Yankee to win the award since Derek Jeter (1996)…was also named 2017 Sporting News AL "Rookie of the Year," as voted on by a panel of AL players…was selected to the Topps MLB All-Star Rookie Team... Led the American League in runs scored (128, 2nd MLB), home runs (52, 2nd MLB), walks (127, 2nd MLB) and and ranked second in RBI (114, 6th MLB), on-base pct.Published:2023-02-26</doc><doc>[5]Title:SportsHighlights:Any commercial use or distribution without the express written consent of STATS PERFORM is strictly prohibited.Published:2021-10-26</doc>
LLM response with Exa context (
answer_correctness: 0.20):
Based on the provided documents, there is no specific information about the score of a Yankees game on June 12, 2024.
AskNews context is filled with contextual details to help the LLM understand time and entities throughout.
Context from AskNews:
<doc>[1]:Title: Biden receives Super Bowl winner Kansas CitySummary: US President Joe Biden received the Kansas City Chiefs, the winners of the Super Bowl, at the White House. The team's star player, Travis Kelce, was also present, who has been making headlines for his relationship with pop star Taylor Swift. The Chiefs have won three Super Bowls in the last five years and are aiming for a fourth title next season, which would be a historic achievement in the NFL. Chiefs boss Clark Hunt expressed his enthusiasm for the visit and the challenge of making it an annual occurrence.Source: Kronen ZeitungPublished: June 01 2024 06:55Title: US PresidentPerson: Joe Biden, Taylor Swift, Clark HuntSports: NFL, Kansas City Chiefs, ChiefsEvent: Super Bowls, Super BowlTime: five yearsClassification: SportsSentiment: 1Reporting voice: Objective</doc><doc>[2]:Title: Biden is hosting the Kansas City Chiefs – minus Taylor Swift – to mark the team's Super Bowl titleSummary: President Joe Biden is hosting the Kansas City Chiefs at the White House to celebrate their third Super Bowl victory in five years. The team will be attending the event on the South Lawn, but Taylor Swift, the girlfriend of tight end Travis Kelce, will not be present. The Chiefs won the Super Bowl in February with a come-from-behind overtime win over the San Francisco 49ers. This is a tradition for major championship sports teams to be invited to the White House. Kicker Harrison Butker, who made headlines for criticizing some of Biden's policies, will also be attending the event.Source: YahooPublished: May 31 2024 17:41Title: PresidentPerson: Biden, Joe Biden, Travis Kelce, Taylor Swift, Harrison ButkerSports: Kansas City Chiefs, Chiefs, San Francisco 49ersEvent: Super BowlTime: five yearsLocation: South LawnDate: FebruaryClassification: PoliticsSentiment: 1Reporting voice: Objective</doc><doc>[3]:Title: Tom Brady: Chiefs face 'big challenge' in attempt to win third consecutive Super BowlSummary: Tom Brady, the most accomplished player in American football history, believes that winning three consecutive Super Bowls is a significant challenge. The Kansas City Chiefs are currently favored to win this year's Super Bowl and would become the first team in 20 years to win three consecutive championships if they succeed. Brady acknowledges the Chiefs' talent, coaching, and continuity, but also notes the variables that come into play, including luck and health. He believes that winning three in a row is nearly impossible, citing the difficulty of winning one Super Bowl in the first place.Source: MSNPublished: May 28 2024 15:26Person: Brady, Tom BradyNationality: AmericanEvent: Super Bowls, health, Super BowlSports: Kansas City ChiefsTime: 20 yearsClassification: SportsSentiment: 0Reporting voice: Objective</doc><doc>[4]:Title: Which NFL teams have never won a Super Bowl in their team's history?Summary: There are 12 NFL teams that have never won a Super Bowl. The article discusses the chances of these teams winning the Super Bowl in the 2024 season. The Minnesota Vikings are rebuilding their quarterback room and may not be thinking Super Bowl this year, but could have a window open in 2025. The Buffalo Bills may not even win the AFC East this year due to the loss of an elite wide receiver and potential regression of their offense. The Cincinnati Bengals, who appeared in the Super Bowl three years ago, are a strong contender with a healthy Joe Burrow, but their quarterback's injury issues are a concern. The article concludes that only the Bengals have a good chance of winning the Super Bowl in 2024.Source: NFL Spin ZonePublished: May 25 2024 20:00Number: 12Sports: Cincinnati Bengals, Buffalo Bills, Minnesota Vikings, NFLEvent: Super BowlDate: 2025, 2024Location: AFC EastPerson: Joe BurrowClassification: SportsSentiment: 0Reporting voice: Conversational</doc><doc>[5]:title: Patrick Mahomes insists Chiefs WILL seal historic Super Bowl three-peat... as he reveals the truth behind this year's game-winning touchdownsummary: Kansas City Chiefs quarterback Patrick Mahomes believes his team will win the Super Bowl again next year, which would be the first three-peat in the Super Bowl era. Mahomes and the Chiefs won their second consecutive championship earlier this year, and he revealed that the game-winning touchdown to Mecole Hardman in overtime was not the original plan. The intended target was Jerick McKinnon, but Mahomes saw Hardman open and threw the ball to him instead. Mahomes has pledged to run the same play, called 'corndog', for a touchdown in a third straight Super Bowl. However, with Hardman being a free agent and Kadarius Toney falling out of favor, it's unclear who will be the targeted receiver for the play.source: Daily Mailpublished: May 02 2024 23:59Organization: Kansas City Chiefs, ChiefsPerson: Patrick Mahomes, Mecole Hardman, Hardman, Kadarius Toney, Jerick McKinnonEvent: Super BowlDate: next year, this yearTime: overtimeclassification: Sportssentiment: 1</doc>
LLM output using AskNews context (
answer_correctness: 0.996):
The Kansas City Chiefs won the Super Bowl this year.
The context has a range of different superbowls discussed, and again, no publication dates to help the LLM ground itself in the time-sensitive question.
Context from JinaAI:
<doc>[1]List of Super Bowl championsThe Cowboys won Super Bowl XXX the following year for three titles in four seasons, and thus were two wins away from four consecutive Super Bowl titles. The <strong>New England Patriots</strong> won Super Bowls XLIX, LI, and LIII for three titles in five seasons. They also appeared in and lost Super Bowl LII ...</doc><doc>[2]Super Bowl Winners and ResultsThe <strong>Pittsburgh Steelers and New England Patriots</strong> are tied at 6 Super Bowl victories each. They are followed by the Dallas Cowboys and San Francisco 49ers at 5 wins each. A total of 12 NFL teams have never won the Super Bowl. This list includes: Cleveland Browns, Detroit Lions, Houston Texans, ...</doc><doc>[3]Who Played in (and Who Won!) the Last 10 Super BowlsHere, see how it shook out, and look back on the previous 10 winners of the Super Bowl. ... Travis Kelce. ... Back-to-back, baby! The <strong>Kansas City Chiefs</strong> won their second consecutive Super Bowl on Feb. 11, 2024, after overtaking the San Francisco 49ers in overtime with a score of 25-22 at Allegiant ...</doc><doc>[4]Super Bowl Winners List - Full NFL winners list from 1967 to 2024 - MARCA in EnglishCheck the updated ranking of all the winners, runners-up, MVP and scores of <strong>the</strong> <strong>Super</strong> <strong>Bowl</strong> winners <strong>year</strong> <strong>by</strong> <strong>year</strong> since the beginning in 1967.</doc><doc>[5]Super Bowl LVIThe <strong>Los Angeles Rams</strong> finished the 2021 season with a 12–5 record under fifth-year head coach Sean McVay. This was their fifth Super Bowl appearance, third as a Los Angeles–based team, and second under McVay. The franchise held a 1–3 Super Bowl record prior to the game, winning Super Bowl ...</doc>
The response from JinaAI does not contain sufficient context to answer the question.
LLM response with JinaAI context (
answer_correctness: 0.20):
Based on the given documents, it is not possible to determine who won the Super Bowl this year.
The context from Tavily appears to be a poorly scraped webpage, filled with different historical superbowls.
Context from Tavily:
<doc>[1]Kansas City Chiefs win Super Bowl LVIII: Patrick Mahomes throws game ...Patrick Mahomes leads the Kansas City Chiefs to a 25-22 victory over the San Francisco 49ers to become Super Bowl champions for the third time in five seasons; Mahomes named Super Bowl MVP for the ...</doc><doc>[2]Kansas City Chiefs win Super Bowl for second year in a row - NBC NewsMinutes later, Mahomes' 3-yard touchdown pass to Mecole Hardman ended in a 25-22 Kansas City victory. Sunday's win had a similar vibe to the 2020 Super Bowl when the Chiefs staged a dramatic ...</doc><doc>[3]2021 Super Bowl score: Tom Brady wins seventh ring as Buccaneers ...While the field goal was a disappointing result, it did cut the deficit to 14-6, and with the Chiefs getting the ball in the second half, they seemed to be in a good position. The frustrations were notable on both sides of the ball for KC, which committed 10 penalties for 100 yards in the game with many miscues coming in key situations that cost the Chiefs on the scoreboard. Just as the Packers' offensive line was dominated by the Bucs' front in the NFC Championship Game, so too was the Chiefs' offensive line in the Super Bowl. Sorry!Breech's picks: 2 thrillers decide who'll be in SB LVIII15 wild facts to know about the title gamesTitle games preview: Odds, picks and moreNew Chiefs wrinkle that turned 'O' aroundPlayoff schedule: Dates, times, TVRaiders find new GM, to hire former Chargers executiveEagles' Johnson out as OC: Top 5 potential replacementsTitle game injuries: Where the game turnedToward the tail end of the first half, it seemed like the Chiefs might be working their way back into the game.</doc><doc>[4]Tom Brady wins Super Bowl No. 7, Buccaneers beat Chiefs 31-9(AP Photo/Ashley Landis)Tampa Bay Buccaneers quarterback Tom Brady (12) celebrates during the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021 in Tampa, Fla. (Ben Liebenberg via AP)Tampa Bay Buccaneers quarterback Tom Brady (12) celebrates during the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021 in Tampa, Fla. (Ben Liebenberg via AP)Tampa Bay Buccaneers quarterback Tom Brady reacts after getting stopped at the goal line against the Kansas City Chiefs during the first half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Ashley Landis)Tampa Bay Buccaneers quarterback Tom Brady reacts after getting stopped at the goal line against the Kansas City Chiefs during the first half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Ashley Landis)Tampa Bay Buccaneers’ Rob Gronkowski (87) celebrates with Mike Evans (13) and quarterback Tom Brady (12) after Gronkowski scored a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Mark LoMoglio)Tampa Bay Buccaneers’ Rob Gronkowski (87) celebrates with Mike Evans (13) and quarterback Tom Brady (12) after Gronkowski scored a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Mark LoMoglio)Tampa Bay Buccaneers tight end Rob Gronkowski celebrates after catching a 17-yard touchdown pass during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Gregory Bull)Tampa Bay Buccaneers tight end Rob Gronkowski celebrates after catching a 17-yard touchdown pass during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Gregory Bull)TAMPA, Fla. (AP) — Tom Brady made the Buccaneers, their fans and their city believe from the moment he arrived in Tampa Bay. (AP Photo/Ashley Landis)Tampa Bay Buccaneers quarterback Tom Brady (12) celebrates during the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021 in Tampa, Fla. (Ben Liebenberg via AP)Tampa Bay Buccaneers quarterback Tom Brady reacts after getting stopped at the goal line against the Kansas City Chiefs during the first half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Ashley Landis)Tampa Bay Buccaneers’ Rob Gronkowski (87) celebrates with Mike Evans (13) and quarterback Tom Brady (12) after Gronkowski scored a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Mark LoMoglio)Tampa Bay Buccaneers tight end Rob Gronkowski celebrates after catching a 17-yard touchdown pass during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Gregory Bull)Tampa Bay Buccaneers’ Rob Gronkowski (87) reacts after scoring a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Charlie Riedel)Tampa Bay Buccaneers’ Rob Gronkowski (87) reacts after scoring a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Charlie Riedel)Tampa Bay Buccaneers’ Rob Gronkowski (87) reacts after scoring a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Charlie Riedel)Tampa Bay Buccaneers’ Rob Gronkowski (87) reacts after scoring a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Charlie Riedel) Security tries to grab a fan on the field during the second half of the NFL Super Bowl 55 football game between the Tampa Bay Buccaneers and the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Mark Humphrey)Kansas City Chiefs quarterback Patrick Mahomes breaks away from Tampa Bay Buccaneers outside linebacker Jason Pierre-Paul during the second half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Ashley Landis)Kansas City Chiefs quarterback Patrick Mahomes breaks away from Tampa Bay Buccaneers outside linebacker Jason Pierre-Paul during the second half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Ashley Landis)Tampa Bay Buccaneers quarterback Tom Brady (12) talks with Kansas City Chiefs strong safety Tyrann Mathieu (32) after throwing a touchdown pass during the first half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Gregory Bull)Tampa Bay Buccaneers quarterback Tom Brady (12) talks with Kansas City Chiefs strong safety Tyrann Mathieu (32) after throwing a touchdown pass during the first half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Gregory Bull)Tampa Bay Buccaneers quarterback Tom Brady celebrates with the Vince Lombardi Trophy after the NFL Super Bowl 55 football game against the Kansas City Chiefs Sunday, Feb. 7, 2021, in Tampa, Fla. Security tries to grab a fan on the field during the second half of the NFL Super Bowl 55 football game between the Tampa Bay Buccaneers and the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Mark Humphrey)Kansas City Chiefs quarterback Patrick Mahomes breaks away from Tampa Bay Buccaneers outside linebacker Jason Pierre-Paul during the second half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Ashley Landis)Tampa Bay Buccaneers quarterback Tom Brady (12) talks with Kansas City Chiefs strong safety Tyrann Mathieu (32) after throwing a touchdown pass during the first half of the NFL Super Bowl 55 football game Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Gregory Bull)Tampa Bay Buccaneers quarterback Tom Brady celebrates with the Vince Lombardi Trophy after the NFL Super Bowl 55 football game against the Kansas City Chiefs Sunday, Feb. 7, 2021, in Tampa, Fla. Tom Brady wins Super Bowl No. 7, Buccaneers beat Chiefs 31-9Tampa Bay Buccaneers’ Rob Gronkowski (87) reacts after scoring a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Charlie Riedel)Tampa Bay Buccaneers’ Rob Gronkowski (87) reacts after scoring a touchdown during the first half of the NFL Super Bowl 55 football game against the Kansas City Chiefs, Sunday, Feb. 7, 2021, in Tampa, Fla. (AP Photo/Charlie Riedel)</doc><doc>[5]Super Bowl 2021: Tom Brady wins seventh title as Tampa Bay ... - BBCThe former player and broadcaster opens up about what motivated him and his very humble originsExposing organised crimeA three-part series following one specialist police unit over an extraordinary two-year operationFeaturedThe political row that lost Arizona a Super BowlThe 1993 Super Bowl was to be a landmark event for Arizona but it disappeared out of the state in a swirl of politics, polemic and division. Top StoriesWatch: UK Snooker Championship - Murphy v Vafaei & Carter v SeltWSL: Shaw scores hat-trick as Man City put seven past Tottenham - reactionVenables the best English coach we've had - LinekerElsewhere on the BBCThe dark side of one of the most beautiful places on earthDI Ruth Calder returns to her native Shetland on the trail of a vulnerable witness... While the Chiefs came into the post-season as the top seed in the AFC Conference, Tampa Bay were the NFC's fifth seed and became the first wildcard team to win a Super Bowl since the Green Bay Packers 10 years ago after a remarkable play-off run. The Chiefs finally got some offence going to reply with a field goal, 61 seconds from half-time, and just as they did in the NFC Championship game against the Packers, Tampa Bay claimed a crucial score right before the half as Brady completed a 71-yard drive by finding Brown in the end zone. The Weeknd was the interval act and, for the first time in the game's 55-year history, he was largely forced to perform from the stands, rather than the pitch, in compliance with strict coronavirus protocols.</doc>
It seems that one of the documents had the right answer hidden inside of it, but the LLM is confused by the poorly engineered context, especially the lack of publication dates for the LLM to ground itself:
LLM response from Tavily context (
answer_correctness:0.22):
The Tampa Bay Buccaneers won the Super Bowl this year.
The context from Exa appears to be a poorly scraped webpage, filled with different historical superbowls.
Context from Exa:
<doc>[1]Title:Official New England Patriots News and AnalysisHighlights:View the Cheerleader roster, appearance schedule, news, photos and more! Supporting charitable and philanthropic agencies throughout New EnglandPublished:2023-02-01</doc><doc>[2]Title:Official Website of the Kansas City Chiefs | Chiefs.comHighlights:A look at some of the community focused events held throughout the week of the 2023 NFL Draft in Kansas City Missouri. Stay up to date on news, access your tickets, stream live games, sign-up for Chiefs Kingdom Rewards & much more Get all your Chiefs gear from the official Chiefs Pro ShopPublished:2012-06-02</doc><doc>[3]Title:New England Patriots Football - Patriots News, Scores, Stats, Rumors & More | ESPNHighlights:Tom Brady Sr., Tom Brady's Father, joins the show to talk about his legendary son's retirement. Greeny and Hembo wrap up the hour talking about some NFC East trash talk and Hembo starts using words from Shakespearean times. 3dPublished:2023-02-06</doc><doc>[4]Title:Philadelphia EaglesHighlights:news Eagles, Nouryon celebrate National Teacher Appreciation Week May 10, 2023 Members of the community, including students, teachers, and school administrators, are encouraged to submit nominations between Wednesday, May 10, and Thursday, October 26.Published:2012-06-02</doc><doc>[5]Title:Tampa Bay BuccaneersHighlights:gallery Bucs' 2023 Schedule Gallery View photos of the Bucs 2023 schedule. MORE PHOTOS Follow Us AdvertisingPublished:2023-02-06</doc>
The context does not contain sufficient context for the LLM to produce the correct answer.
LLM response using Exa context (
answer_correctness: 0.46):
Based on the given documents, there is no information available to determine who won the Super Bowl this year.
Check out the Google Colab output for the other questions:
3.“What are the best features of the latest NodeJS version?” — Knowledge Search
Where we saw the LLM using JinaAI context say “the best features are not explicitly mentioned”
4.“What is the current Fed Reserve interest rate?” — Google-esque
Where we saw all three services provide sufficient context for the current Federal Reserve interest rate. But the context engineering for JinaAI and Tavily are loaded with extraneous (and expensive) tokens, likely due to naive web scrapes.
Each service has their own plans and per-call pricing and so the up-front cost depends on your individual use-case — how many calls do you need per month? At what concurrency? Do you need a commercial license?
Because of this, and the fact that pricing changes over time, we cannot make a full cost comparison, but you can find the pricing pages of each service here: Tavily AskNews JinaAI Exa
But cost doesn’t stop at the price per call — it also includes the number of tokens that need to go into the LLM to get the response. We saw that Tavily produced the most context, causing the most input tokens to the LLM, compared to all other services. Meanwhile, JinaAI produced the smallest amount of context and the smallest number of input tokens, meaning the call to the LLM was cheapest for JinaAI and most expensive for Tavily.
Unfortunately, with Tavily, we see some strange scrapes filled with duplicate lines (see the context example for the Super Bowl question). If you plan to direct that context into your LLM, you will pay for each of those duplicated tokens, again and again. JinaAI provides a “description” as well as “content”. If you dare dumping all the content into your LLM without any pre-processing, you will saturate the context window and pay a hefty sum.
In comparison, AskNews appears to be aiming for delivering “prompt-optimized” tokens, meaning that the context is as dense as possible — with entity extractions and all the other contextual information laid out in a clear concise way for the LLM. It came in 3rd place for number of input tokens, but considering the increase of quality, it is probably worth the extra 15% of input tokens.
AskNews and Exa seems to have wider ranges of desirable filtering options that extend far beyond Tavily and JinaAI. For example, AskNews and Exa let you filter responses on time. Additionally, they have a variety of optimization options such as “auto prompting” and controlling number of return documents. In comparison, Tavily and JinaAI only fetch 5. AskNews takes the filters one step further, allowing you can filter on category (sports, politics, etc.), “reporting voice” and “provocativity”.
📢 Note: This document was written to be as fair of a comparison as possible, as shown in the associated Google Colab notebook. However, it is possible that the author was unable to fully optimize one or more of these services. If you believe your service is misrepresented, and you’d like to request a correction to the document, please contact the author at contact at emergentmethods.ai with your requested change.