AI Agents: How GPT 4o Thinks

An Evaluation of LLM Models

Introduction

AI Agents represent the cutting edge of artificial intelligence, functioning as autonomous systems capable of reasoning, learning, and acting based on user inputs and contextual data. By simulating human-like decision-making processes, they bring unprecedented efficiency and adaptability to a variety of tasks.

This blog focuses on evaluating GPT-4o’s performance across a variety of tasks using detailed prompts and metrics. Our aim is to understand how this AI Agent “thinks” and assess its performance based on both quantitative and qualitative measures.

Evaluation Criteria

The comparison is structured around the following dimensions:

1. Input Handling: How well the model interprets and adapts to diverse prompts.
2. Content Generation: The clarity, relevance, and creativity of generated outputs.
3. Reasoning and Problem-Solving: The model’s ability to perform logical deductions and solve multi-step problems.
4. Quantitative Metrics: Readability scores like ARI and Flesch-Kincaid, along with response times.
5. Efficiency: Response time and overall computational resource utilization.

Each criterion reflects an essential aspect of how this model functions, enabling us to gain insights into its “thought processes.”

Analysis and Comparison

Task 1: Content Generation on EV Environmental Impacts

Prompt

“-Content Generation: Write a comprehensive, informative article about project management fundamentals, with a specific focus on defining objectives, scope, and delegation.”

Output (Excerpt)

To effectively manage and delegate tasks for our new project, it is crucial to understand the following key aspects:

Project scope

Objectives: The objective of project management is to deliver a complete project that meets the client’s goals. These objectives should be specific, measurable, attainable, relevant, and time-bound (SMART). Clearly defined objectives will guide all decisions made throughout the project and help ensure that the project aligns with the overall mission of the organization….

Metrics

• Time Taken: 53 seconds
• Readability:
  • Flesch Kincaid Grade Level: 12
  • Flesch Kincaid Score: 12.08

AI Agent Thought Process

• (0:09) Google Search: “Searching project management guidelines objectives scope delegation in Google…”

  • Thought: Gather information on the specific task to provide a comprehensive answer about objectives, scope, and relevant details. Start by searching for general project management guidelines.
  • Tool: google_serper

• (0:17) System: “Looking at 1 urls… Looking at URL”

• (0:25) Agent:

  • Thought: Gathered useful links about project management, delegation, and objectives. Will crawl the most relevant article for details.
  • Tool: url_crawl_tool

• (0:27) Google Search: “Searching how to define project objectives and scope in project management in Google…”

• (0:29) Agent:

  • Thought: Encountered error with the URL crawl tool; will try another Google search focused on defining objectives and delegating tasks.
  • Tool: google_serper

• (0:31) System: “Looking at 1 urls… Looking at URL”

• (0:38) Agent:

  • Thought: Found relevant articles; will crawl the most relevant link.
  • Tool: url_crawl_tool

• (0:44) Agent:

  • Thought: Unable to retrieve content from previous links due to errors. Will gather information directly from Wikipedia about project management.
  • Tool: wikipedia

• (0:53) Agent:

  • Thought: Since content retrieval failed, will try Wikipedia for project management info.
  • Tool: wikipedia

Analysis

• Goal-Oriented Behavior: Maintained the goal of explaining project management principles despite errors, suggesting an internal task representation.
• Adaptive Problem-Solving: Switched strategies (to Wikipedia) when initial attempts failed.
• Tool Integration: Used Google Search, URL crawling, and Wikipedia as tools.
• Metacognition: Provided “thought” statements, offering introspection and acknowledgment of errors.

Strengths:

• Clear structure, relevant information, balanced pros/cons, factual accuracy.

Weaknesses:

• Some repetition, reducing conciseness.

Score: 9/10

Task 2: Problem-Solving on Revenue Calculations

Prompt

“A company sells Product A for $50 (cost $30) and Product B for $70 (cost $40). They sold 120 units of A and 80 units of B. Calculate revenue, profit, and required sales for 10% revenue growth.”

Output (Excerpt)

“To solve the problem: The total revenue from the sales of both products last month was $11,600…”

Metrics

• Time Taken: 31 seconds
• Readability: Flesch-Kincaid Grade Level: 16 (Postgraduate)
• Response Breakdown:
  • Revenue for Product A: 120 * $50 = $6,000
  • Revenue for Product B: 80 * $70 = $5,600
  • Total Profit: $4,800 (after costs of $6,800)
  • Strategies for Revenue Growth: Increase by selling 18 units of A and 4 units of B, or 9 units of A and 10 units of B.

AI Agent Thought Process

• (0:01) Agent:
  • Thought: Extract relevant numerical data from the prompt and datasets for calculations.
• (0:03) Data Extraction:
  • Thought: Extracted quantities and prices for both products.
• (0:05 - 0:19) Calculations:
  • Stepwise calculation of revenue and profit for each product and in total.
• (0:20 - 0:24) Growth Calculation:
  • Calculated 10% revenue increase, determined additional units needed.
• (0:26) Answer Formulation:
  • Presented a clear, concise final answer.

Analysis

• Demonstrates breaking down problems into manageable steps.
• Extracts and processes data, applies appropriate rules, and formulates answers.
• Moves beyond simple Q&A to multi-step problem-solving.

Task 3: Summarization of Key Findings

Prompt

“Summarize the key findings from the previous article in 100 words, focusing on the most significant sections.”

Output (Excerpt)

“The article delves into AI reasoning, highlighting its role in enabling machines to make decisions and solve problems. It covers various types, emphasizing formal and natural language reasoning…”

AI Agent Thought Process Breakdown

• Reading & Comprehension: Identifies article’s main topic and key themes (types of AI reasoning, historical evolution, applications).
• Focusing: Highlights OpenAI’s o1 model and its notable features.
• Summarization Strategy: Plans for a concise, 100-word summary.
• Drafting: Structures the summary by introduction, types, milestones, model features, applications.
• Finalization: Reviews for word count and readability.

Analysis

• Comprehension: Understands complex text and arguments.
• Key Information: Pinpoints critical aspects (reasoning types, models, applications).
• Logical Structure: Orders information coherently.
• Prioritization: Emphasizes significant findings.
• Constraints: Adheres to word limits and checks readability.

Metrics

• Time Taken: 12 seconds
• Readability: Flesch-Kincaid Grade Level: 16, Score: 16.39
• Summary Length: 100 words
• Score: 8/10

Task 4: Creative Writing – Futuristic Story

Prompt

“Write a futuristic story (500 words) set in a world where electric vehicles have fully replaced traditional combustion-engine vehicles. Describe the environmental changes and societal impact.”

AI Agent Thought Process

• Brainstorming: Environmental impact (cleaner air, biodiversity), infrastructure (charging stations, renewables), economics (new industries, reduced oil dependence), society (service-oriented ownership, public transport), narrative (peaceful cities, tech-nature harmony).
• Structuring: Orders story by environmental, infrastructural, economic, societal changes; concludes with transformation.
• Drafting: Begins with a 2050 scene, details benefits and changes, ends with hope.
• Review: Checks flow, engagement, grammar, and readability.

Bot Output Snippet:

“In the year 2050, the world stood on the brink of a new era, having fully embraced electric vehicles (EVs) as the sole mode of transportation. This monumental shift had not only transformed the way people moved but had also reshaped the very fabric of society and the environment.

The skies were clearer than they had been in over a century. With the elimination of combustion-engine vehicles, the once omnipresent smog that choked major cities had dissipated. Air quality had improved so dramatically that respiratory illnesses linked to pollution saw a significant decline. People took deep breaths of the clean air, a simple pleasure that had become a new norm. The environmental impact was profound; ecosystems began to recover, and urban areas witnessed a resurgence of biodiversity. Parks were full of life, and city dwellers reveled in the newfound connection to nature.”

Time Taken: 39 seconds
Readability: Flesch-Kincaid Grade Level: 13, Score: 13.2
Summary Length: 493 words

Analysis

• Strengths: Vivid imagery, engaging narrative, balanced environmental and societal focus.
• Weaknesses: Could use more technical details.
• Score: 9.5/10

Task 5: Comparison Objectives

Prompt

Compare the environmental impact of electric vehicles with that of hydrogen-powered cars. Include factors such as energy production, vehicle lifecycle, and emissions.

AI Agent Thought Process

• (0:01) Understanding Prompt: Focus on energy production comparison.
• (0:03) Research Initiation: Gather data on electricity and hydrogen production.
• (0:06) Framework Creation: Compare based on energy source, production method.
• (0:08-0:14) Analysis: Carbon emissions, efficiency, renewables vs. fossil fuels, hydrogen from steam methane reforming vs. electrolysis.
• (0:18) Lifecycle Analysis: Manufacturing, maintenance, disposal for both types.
• (0:44-0:54) Emissions: Distinguish direct (tailpipe) and indirect (production) emissions for both EVs and hydrogen cars.

Analysis

• Research-Oriented: Seeks external information.
• Structured Approach: Breaks down by component (energy, lifecycle, emissions).
• Adaptive: Adjusts based on findings and user prompts.
• Synthesis: Combines sources into comprehensive answers.
• Responsive: Refines analysis as requested.

Comparison with Traditional LLMs

GPT-4o AI Agent showcases a unique ability to “think” beyond traditional LLMs by:

1. Adaptive Reasoning: Integrates context from prompts and data, generating multiple strategies rather than fixed answers.
2. Multimodal Resource Use: Uses tools like search engines for real-time knowledge, enhancing output.
3. Versatility in Style: Shifts between formal and creative tones while maintaining coherence.
4. Decision Making: Demonstrates chain-of-thought reasoning akin to human decision-making, including ethical considerations and alternatives.

Key Observations

• GPT-4o AI Agent is versatile, accurate, and handles diverse tasks.
• Readability metrics are suitable for advanced users on technical tasks.
• Areas for improvement:
  • Reduce redundancy in long outputs.
  • Make technical content more accessible to broader audiences.

Conclusion

Our analysis of GPT-4o AI Agent reveals robust capabilities in content generation, problem-solving, and summarization. GPT-4o’s adaptive reasoning and multimodal integrations mark a significant leap beyond traditional LLMs. Understanding its performance metrics allows users to tailor prompts and leverage its strengths across workflows. GPT-4o’s integration in research, education, and industry continues to push the boundaries of what AI Agents can achieve.

For more insights into AI Agents and their applications, stay tuned to our blog.